Why is development hard?
We often see high profile failures, massive budget overruns, and lengthy project delays in the IT sector. In fact, only about 34% of projects meet scope, cost and schedule goals. But most of the challenges we face in IT are not unique to agencies – a lot of these issues are common throughout the software industry.
Young Industry
We've been creating software for just 50 years, making it a very young industry. Consider that against the fact that we've been building houses and roads for thousands of years, giving us plenty of time and opportunities to learn from our mistakes, and you’ll see how far there is to go.
Most industries have prefabricated components. Building in the UK, for instance, requires a chartered surveyor, an architect, planning permission and building control, as well as the need to conform to various regulations. But often in software development, most of the system is created from scratch. Lots of progress has been made, but there is still plenty to make. So, are government regulations, chartered developers, etc. the answer? Probably not. The best way to stifle innovation is too much regulation.
Complexity
Software is one of the most complex things around. It contains thousands of lines of code, and every line is a potential point of failure. Systems can have numerous inputs, outputs, and states to deal with, and each can impact or be impacted by other lines of code. Whether it’s a third party library, compiler or operating system, it is virtually impossible to build a system without dependencies that are just as likely to have bugs and issues.
It’s hard to test every line of code as they rarely exist on their own, instead relying on conditions and data from other pieces of code. There is a diminishing set of returns for testing completely, and the more thorough the testing, the longer it takes – almost exponentially. Software is rarely released defect-free, but often has a number of acceptable or undetected issues.
Stakeholder Input
To add to the complexities of software, there is, more often than not, a lack of stakeholder input. This might be the team that is paying for the system, but frequently it’s the people who will actually be using the software. Software is repeatedly pushed by management to improve or automate processes, but if business users are not consulted this can lead to resistance and problems rolling out the system. Stakeholders can be uninterested during the early stages when a lot of the work involves documentation, only to start querying things after the software has been built.
Change
Analysis is also difficult. Trying to get a full understanding of a system’s requirements is hard, and people often don’t know what they want until they see it. Software suffers from an average 25% change in requirements throughout the course of a project, and even small changes can have a big impact on a system. The later the change, the bigger the impact.
Child’s play
Software development is hard because programming is easy. There are plenty of people who can write code – in fact, it’s a skill that everybody should consider learning. But it’s difficult to write good, maintainable code, and it takes years to learn how to architect and deliver great software. Lots of skills are learnt on the job and through gaining experience while working in teams; however, the best skills are often learnt from making mistakes.
External dependencies
There are nearly always factors outside of the software that can put constraints on projects. Hardware limitations could mean a system can only use a certain amount of memory, while integration with other systems can have requirements on the latency of responses, or regulations like data protection laws can affect how data is encrypted. Some of these may seem trivial, but they all have significant implications on a system.
How long is a piece of string?
They can also be very difficult to test. How do you test integration with another system when that system is still being built? Estimating is more of an art than a science, and since no two pieces of software are the same, it’s very difficult to estimate how long it will take to create software. There can be lots of unknowns, scope creep, and different developer abilities, all affecting timescales. So the earlier in the process that estimations are made, the greater the chance that they will be inaccurate. Likewise, the larger the system the harder it is to be precise.
Resourcing
The management of development teams can be fraught with problems. Think about pregnancy, for example: one woman can have one baby in nine months, but nine women can’t have one baby in one month. It isn't the case that adding more developers increases productivity. In fact, this often leads to increased management overheads and lower productivity due to higher interaction between team members.
What’s more, new developers can’t hit the ground running. It can take months to learn a system. So if you think you should keep adding more developers to a team, how about making the current team do more work instead? But even that solution brings problems since overtime on a project is only useful in short, sharp bursts. Consistent prolonged overtime doesn't guarantee more work will be achieved. In fact, working longer hours reduces cognitive ability, so a 60 hour week will produce approximately the same work as a 40 hour week.
Are digital agencies different?
It might actually be even harder for us since the web is still fairly new and rapidly changing, HTML and CSS constantly evolving, and the need to cater for a multitude of browsers and devices continues to grow. We need to constantly evaluate new technologies and techniques as well as to push innovation in our sector.
There is a massive focus on security at the moment, and barely a week goes by without a high profile company being exposed. We have to perform rigorous security testing to ensure that we don’t risk the reputations of ourselves or our clients.
So what can be done?
A key change in the industry is a move away from the waterfall process, which contains too much documentation and lacks flexibility, to an agile process. Agile methodologies allow teams to concentrate on building great software whilst simultaneously mitigating the traditional pitfalls of waterfall projects.
By following an agile process, user stories can be prioritised based on importance and dependencies. There is a greater focus on delivering working functionality within a sprint, which enables the system to be tested earlier and clients to provide feedback sooner. Change is encouraged rather than frowned upon, and by having short sprints and regular reviews, any new stories can be incorporated into later sprints during planning sessions.
But agile isn't always the golden child. Consideration needs to be taken to ensure that cross-cutting concerns and constraints, like security and logging, are captured and considered during each sprint. Every story needs a well-defined set of acceptance criteria – without this, it becomes difficult to test and easy for developers to go off-track with their interpretations of ‘done’.
I'm not sure this can be stressed enough, but communication is key. A team that doesn't discuss ideas or an agency that doesn't engage with clients is bound to fail. You need to work as a team and the client is part of that team. The more you work together, the more likely issues will be addressed early, requirements will be understood fully, and software will meet stakeholder needs.
To optimise downtime in between projects, teams should be encouraged to constantly improve their skills. Developers can work on creating reusable components and libraries for projects, allowing them to deliver software quicker and provide better value for clients.
Companies should do their best to keep developers happy. It’s far more cost effective to provide beers after work or reward high performers with the occasional pay rise than to recruit and train new members of staff. However, companies shouldn't become dependent on certain members of staff, and when new team members join there should be processes to get them on board quickly and efficiently.
Teams should also be focused. There’s nothing worse than interruptions during the day, and small distractions like a five minute phone call can cause a developer to lose an hour of productivity because it takes so long to get into a rhythm to write code. To tackle this, informal rules can be created – for example, certain times of the day for checking emails, attending meetings, coding. Learning to prioritise work and understanding when to say ‘no’ go a long way to making you more productive.
We need to be aware of what can lead projects off the rails and how we can limit and prevent this from happening. Just remember: there is no silver bullet.