This page may be out of date. Submit any pending changes before refreshing this page.
Hide this message.
Quora uses cookies to improve your experience. Read more
100+ Answers
Dave Baggett
I've contributed code to -- and, in most cases, also managed or co-managed -- software development tasks as small as 7 people (Crash Bandicoot) and as large as several hundred people (ITA Software's multiple person-millennia airline reservation system effort).

As Arun Shroff said, you can find the long answer in Fred Brooks' Mythical Man-Month.

The shorter answer, in my experience, is that most software development projects are much less well understood at their outset than equivalent physical world projects, because they can be.

Software is completely intangible; it doesn't require big machines to dig holes or move heavy objects. It doesn't require complex permitting or regulatory approval. And a single talented programmer can create a very large amount of code very quickly -- to the extent that a very capable single developer can actually create an entire shippable product him or herself. Contrast this with, say, a single worker attempting to build an entire skyscraper alone. The "virtual" aspect of software completely changes the rate at which products can be constructed, and the resources required. But there's a rub...

The other side of the coin is that because software can be developed without extensive planning -- and even without a complete understanding of the problem to be solved -- the market often demands that it be done this way. Put another way: nobody asks a small contractor to build a gigantic building in 6 months, but customers routinely expect software companies to perform the equivalent software task.

A consequence of this is that software development is typically a process of discovery: the developer learns what needs to be developed as the product evolves. And what's worse, customers generally expect they can change their minds during development without dire consequences -- "how hard can it be to change the way that menu works?" -- which leads to frequent "scope creep" and moves the goal posts as the project progresses.

The software development community has reacted to this overarching challenge by experimenting with new, so-called agile development methodologies. The details vary among these methods, but they all share a more iterative approach incorporating regular doses of feedback from customers prior to shipping. Indeed, under many of these methodologies, "shipping" is really only an arbitrary state one enters for political or contractual reasons; the goal is often to have whatever bits of the system are already implemented work reliably and receive thorough, automated testing, continuously.

These agile methods help ameliorate the difficulties of discovering-while-developing, but by no means make them go away. A significant advantage they offer, in my experience is diagnostic: it's easier to tell where "the progress bar" is using an agile methodology, because you are constantly diagnosing your current state and trying to course-correct appropriately.

To (finally) answer your direct question, it's much more likely that your estimate will be off by a factor of 2, 3, or 10x when you have relatively little idea what you're creating, and need to "discover" what that product is via iteration and feedback, than it is when you know exactly what you're making from day 1.

When building something (nearly) completely understood at the outset -- e.g., a typical physical product like a bridge, dam, or skyscraper -- the classic Waterfall and similar methodologies are far more accurate and effective -- and progress can be tracked with familiar tools like Gantt charts, which provide detailed data on progress to date.

Finally, there are some kinds of software that are necessarily developed using Waterfall-like methods. These are projects where the consequences of failure are extremely dire. If an avionics system fails catastrophically, the potential damage is obviously far greater than if, say, Twitter is down.

Projects like these take vastly more resources, planning, and process to accomplish successfully than their less risky peers. As a consequence, they are often extremely expensive. This is why making the guidance system for a missile is substantially more expensive to make than, for example, an email client (which is what I'm doing now).
72.8k Views · 275 Upvotes
Your response is private.
Is this answer still relevant and up to date?
YesNo
Arun Shroff
Fred Brooks who managed the development of IBM's mainframe operating system OS/360 - a mammoth feat of software engineering by any standard - wrote the seminal book on this topic The Mythical Man Month.  It describes his experience and lessons learnt about what causes most software projects to be delayed.  Listed below are some of these reasons

(based on various excerpts from Wikipedia that have been compiled and edited for relavance: The Mythical Man-Month)


1. The Mythical Man-Month and Brook's Law:


Brooks discusses several causes of scheduling failures. The most enduring is his discussion of Brooks's law: Adding manpower to a late software project makes it later. A man-month is a concept of a unit of work proportional to the number of people working multiplied by the time that they work; Brooks's law says that this relation is a myth, and is hence the centerpiece of the book.

Complex programming projects cannot be perfectly partitioned into discrete tasks that can be worked on without communication between the workers and without establishing a set of complex interrelationships between tasks and the workers performing them.

Therefore, assigning more programmers to a project running behind schedule will make it even later. This is because the time required for the new programmers to learn about the project and the increased communication overhead will consume an ever increasing quantity of the calendar time available.

When n people have to communicate among themselves, as n increases, their output decreases and when it becomes negative the project is delayed further with every person added.

Group intercommunication formula: n(n − 1) / 2
Example: 50 developers give 50 · (50 – 1) / 2 = 1225 channels of communication.


2. The tendency towards irreducible number of errors:

In a suitably complex system there is a certain irreducible number of errors. Any attempt to fix observed errors tends to result in the introduction of other errors.
This is very difficult to anticipate and causes unpredictable delays in debugging the system leading to delays.

3. Feature creep, creeping featurism or featuritis:  is the ongoing expansion or addition of new features in a product, such as in computer software. Extra features go beyond the basic function of the product and so can result in over-complication rather than simple design. Viewed over a longer time period, extra or unnecessary features seem to creep into the system, beyond the initial goals.



Occasionally, uncontrolled feature creep can lead to products far beyond the scope of what was originally intended. For example:  Microsoft's Windows Vista was planned to be a minor release between Windows XP and then the codenamed Windows "Blackcomb" (Windows 7), but it turned out to become a major release which took 5 years of development. And was still a disaster!

4. Accidental complexity:

This is complexity that arises in computer programs or their development process which is non-essential to the problem to be solved. While essential complexity is inherent and unavoidable, accidental complexity is caused by the approach chosen to solve the problem.

While sometimes accidental complexity can be due to mistakes such as ineffective planning, or low priority placed on a project, some accidental complexity always occurs as the side effect of solving any problem. For example, the complexity caused by out of memory errors is an accidental complexity to most programs that occurs because one decided to use a computer to solve the problem.

[ Accidental complexity ]
68.6k Views · 243 Upvotes
Your response is private.
Is this answer still relevant and up to date?
YesNo
Steve Cheung
It's not just large software projects that overrun, many other large projects overrun as well.  The reason for this is multifaceted and it is generally different from project to project.  However, after working on and running projects of different sizes for the last 15 years, the best way I have found understand these failure is to consider projects as Complex Systems.  More specifically, Chaotic Systems.  (See: http://en.wikipedia.org/wiki/Com...). 

From wiki link above, we see that for a system to be considered chaotic, it must satisfy the following properties:
  1. It must be sensitive to initial conditions,
  2. Its periodic orbits must be dense, and
  3. It must be topologically mixing,
Let's take a look at these first to make sure that software projects are indeed Chaotic systems.

Software projects are sensitive to initial conditions
This one is easy to understand.  Initial conditions for a project like resources available, time lines, scope of requirements, talent level, geographical and time zone distribution of teams,  the number of people and teams involved, etc. can all dramatically impact how the project turns out.  Anyone who has worked on a software project will probably agree that they are sensitive to initial conditions.

Software project have have periodic orbits and they are dense
Orbit is not really the right word to use, in software development, we call them processes, methodologies, and lifecycles. Projects are similar from one to the next.  They are often self similar as well in that large projects are divided into similar projects of smaller size. However, if you look at the actual tasks and their execution, there are always many differences.  So software projects have dense periodic obits, or in our case, similar repeating processes.

Software projects are topologically mixing
This one is harder to imagine, but I think of it this way - different projects can often get into similar states event if they have very different initial conditions and there are many differences in execution paths.  Hence, software projects are topologically mixing, i.e. you can get to similar state even though you started vastly differently.

Based on the above, we can establish that software projects are chaotic systems.  Why is this important? Because this lets us examine why projects fail using the features of Complex Systems.  (I selected the relevant features from Wiki, please visit wiki for the original descriptions.)

  • Cascading failures
Software projects are often tightly coupled.  Features depends on each other, teams depends on outputs of other teams, resources gets moved from one team to another, etc.  When a single team or single component is behind schedule, or has lots of issues, this often cause problems for other teams and components amplifying the issue.

  • Projects are open systems
As much as developers like to have well defined specs, and closed scopes, the larger the software projects the more they act like open systems.  People come and go, they get sick, go on vacation, gets impacted by personal lives.  Project scope and requirements change overtime as market demands change, or, quite often, as the understanding of the domain increases.  All these reduce predictability of projects, the larger the size, the less predictable they become because there are more moving parts that can change.

  • Projects have memory
You can interpret this in a number of ways. A project that behind by a week today, will not magically be on time next week.  Bugs introduced this week will not magically disappear next week, i.e. technical debt accumulates through out the project.  C'est la vie.

  • Complex systems are nested
Large projects are often broken into smaller projects, smaller projects are taken on by teams, teams are made up of people.  From projects to teams to people to the actual software being produced, these are all complex systems in and of themselves. When all these complex systems are put together into one huge complex system, results are often impossible to predict.  Mind you, stock markets are the same way, and that doesn't stop people from trying.

  • Dynamic network of multiplicity
This feature manifest itself in many levels of a software project.  From interacting sub teams or team members, to interactions and dependencies of features and components.  There are many of pieces of moving parts, all causing each other to change.  Understanding and adapting to all these parts and interactions in real time is impossible.

  • Projects produce emergent phenomena
An example of this is often seen in large projects that requires integration of a number of components.   Each component can be designed to spec, well tested, but somehow, when everything is put together you get unexpected issues.  (Remember Blackberry outage last year?)  No many how many root causes you do, how many additional process you introduce, this will almost always happen if the project is large enough. 

  • Project relationships are non-linear
In software, a one line code change can cause massive issues.  Requirements with a single misused words can have huge implications.  Remove a single person from a project can had drastic result.  At the same time, all these can also have no effect on the project at all.  It is impossible to know, what impact certain seeming trivial actions can have on a large scale project.

  • Project contains feedback loops
This one is a given for anyone who has ever worked on a software project.  A day delay on one team can cause delay on multiple teams which eventually cause more delays for all teams, a difficult bug take focus away from new development causing more bugs down the road, a bug in requirements causes multiple bugs in software and delays in project, etc.  A minor anomaly can amplify and cause many major issues in later stages of a project.

So what do these all mean?Well it means that Software Projects are inherently difficult to predict. Like trying to predict the weather, even though we know how we can predict the weather to perfect accuracy, calculation that gives us arbitrary accuracy will take longer to calculate than for the event to occur itself.  Project planning is effectively a modelling exercise, so the resulting project plan cannot be taken as the gospel cause they will almost always be wrong.  Trying to do tradition risk management is also pointless, because there are simply too many unknown unknowns.   A better approach is to use statistical data to predict project overruns, and instead of using hard timelines, use probabilistic completion dates.  However, life does not always work the way it should.

Good methodologies, excellent talents, well defined requirements, etc. are all initial conditions.  As such, they can have huge impact on the result of the project, so it is important to get them as right as possible.  Unfortunately, this is often where a lot of compromises are made.   Bidding and estimation processes are often biased towards the most optimistic estimates, setting the projects up for failure. Quality and availabilities of resources are compromised because of priorities and budget.  Etc. etc.  

At the same time though, even if you have all the best initial conditions to start with, your project can still fail simply because of the inherent uncertainty and the amplifying and reinforcing effects in complex systems. Therefore, failed project does not necessarily equal bad plans or mediocre talents or bad process/methodologies.   We live in an uncertain world, and sometimes, shit happens.

One is also reminded of Gall's Law (http://en.wikipedia.org/wiki/Gal...), which extreme programming and agile methodologies are based on.  My personal view though, is that just using methodologies like these are not enough.   Companies that build software also need to make sure that they are structured in such a way teams that make up software teams are complex systems that are known to work.  This means keeping teams tight and consistent instead of moving people from project to project, having teams what work well together continue to work together etc.

Anyway, this is already too long a response, so I'll shut up now.
3.6k Views · 12 Upvotes
Your response is private.
Is this answer still relevant and up to date?
YesNo
Because the plan is a work of fiction. Like all works of fiction, the most important criterion is how the audience likes it.

The purpose of a budget and plan for your average large project is to get the project approved. If accuracy gets in the way of that, then accuracy will fall by the wayside.

From what I've seen, this is rarely intentional. But everybody wants the project to be approved, and everybody knows that low numbers are better. That means they will challenge any number that seems high, while ignoring any number that seems low. The only thing pushing back against systemic low-balling is when the numbers get so low that they seem implausible. The result is a schedule where the due date is the first day that finishing isn't impossible.

It's also important to note the systemic bias in approval. If you only approve projects with relatively low prices, then you've eliminated the projects most likely to come in at or under budget. So naturally, it will seem like everything is over budget.

But wait, it gets worse! Once the project is underway, people eventually realize that their schedule is unrealistic. But people never say, "Boy, this plan sucks, let's make a better one." Instead they say, "OMG we're behind schedule," as if the schedule were something other than a work of fiction. Then they start rushing, cutting corners, and working enough extra that they make themselves stupid, frazzled, and short-sighted.

One thing wrong with that is that in software hurrying to catch up often makes you later. To understand why, read Michael Wolfe's majestic answer on software estimation.

But equally bad is that all that corner-cutting never gets cleaned up after the release; people push the project across the goal line and limp home to see if their children still recognize them. So the next big project to use that code base and generated data has been handed a time bomb. That means yet another unrealistic schedule. And so the software project cycle of life begins anew.
3.7k Views · 19 Upvotes
Michael Wolfe
Let's take a hike on the coast from San Francisco to Los Angeles to visit our friends in Newport Beach. I'll whip out my map and draw our route down the coast:


The line is about 400 miles long; we can walk 4 miles per hour for 10 hours per day, so we'll be there in 10 days. We call our friends and book dinner for next Sunday night, when we will roll in triumphantly at 6 p.m. They can't wait!

We get up early the next day giddy with the excitement of fresh adventure.  We strap on our backpacks, whip out our map, and plan our first day. We look at the map. Uh oh:


Wow, there are a million little twists and turns on this coast. A 40-mile day will barely get us past Half Moon Bay. This trip is at least 500, not 400 miles.  We call our friends and push back dinner til Tuesday. It is best to be realistic. They are disappointed, but they are looking forward to seeing us. And 12 days from SF to LA still is not bad.

With that unpleasantness out of the way, we head off. Two hours later, we are barely past the zoo. What gives? We look down the trail:


Man, this is slow going! Sand, water, stairs, creeks, angry sea lions! We are walking at most 2 miles per hour, half as fast as we wanted. We can either start walking 20 hours per day, or we can push our friends out another week.  OK, let's split the difference: we'll walk 12 hours per day and push our friends out til the following weekend. We call them and delay dinner until the following Sunday. They are a little peeved but say OK, we'll see you then.

We pitch camp in Moss Beach after a tough 12 hour day. Shit, it takes forever to get these tents up in the wind. We don't go to bed until midnight. Not a big deal: we'll iron things out and increase velocity tomorrow.

We oversleep and wake up sore and exhausted at 10 a.m. Fuck! No way we are getting our 12 hours in. We'll aim for 10, then we can do 14 tomorrow. We grab our stuff and go.

After a slow slog for a couple of hours, I notice my friend limping. Oh shit, blisters. We need to fix this now... we are the kind of team who nips problems in the bud before they slow our velocity. I jog 45 minutes, 3 miles inland to Pescadero, grab some band-aids, and race back to patch up my friend. I'm exhausted, and the sun is going down, so we bail for the day. We go to bed after only covering 6 miles for the day. But we do have fresh supplies. We'll be fine. We'll make up the difference tomorrow.

We get up the next morning, bandage up our feet and get going. We turn a corner. Shit! What's this?


Goddamn map doesn't show this shit!  We have to walk 3 miles inland, around some fenced-off, federally-protected land, get lost twice, then make it back to the coast around noon. Most of the day gone for one mile of progress. OK, we are *not* calling our friends to push back again. We walk until midnight to try to catch up and get back on schedule.

After a fitful night of sleep in the fog, my friend wakes up in the morning with a raging headache and fever. I ask him if he can rally. "What do you think, asshole, I've been walking in freezing fog for 3 days without a break!" OK, today is a loss. Let's hunker down and recover. Tomorrow we'll ramp up to 14 hours per day since we'll be rested and trained... it is only a few more days, so we can do it!

We wake up the next morning groggy. I look at our map:


Holy shit! We are starting day 5 of a 10 day trip and haven't even left the Bay Area! This is ludicrous! Let's do the work to make an accurate estimate, call our friends, probably get yelled at, but get a realistic target once and for all.

My friend says, well, we've gone 40 miles in 4 days, it is at least a 600 mile trip, so that's 60 days, probably 70 to be safe. I say, "no f--ing way... yes, I've never done this walk before, but I *know* it does not take 70 days to walk from San Francisco to Los Angeles. Our friends are going to laugh at us if we call and tell them we won't see them until Easter!

I continue, "if you can commit to walking 16 hours a day, we can make up the difference! It will be hard, but this is crunch time. Suck it up!" My friend yells back, "I'm not the one who told our friends we'd make it by Sunday in the first place! You're killing me because you made a mistake!"

A tense silence falls between us. The phone call goes unmade. I'll call tomorrow once my comrade regains his senses and is willing to commit to something reasonable.

The next morning, we stay in our tents till a rainstorm blows over. We pack our stuff and shuffle off at 10 a.m. nursing sore muscles and new blisters. The previous night's fight goes unmentioned, although I snap at my idiot friend when he leaves his water bottle behind, and we have to waste 30 minutes going back to get it.

I make a mental note that we are out of toilet paper and need to stock up when we hit the next town. We turn the corner: a raging river is blocking our path. I feel a massive bout of diarrhea coming on...
Jim Stone
Michael Wolf's answer is brilliant. I wish I'd written it.

That said, . . . I'll make a similar point in a different way.

Here is a way to visualize how complex projects often take longer to complete than originally planned -- using a time lapse video of a to do list.

I set out to complete a very simple programming task, that I originally thought had only 3 steps. You can see that, as I worked, I discovered the task was more complicated than I originally realized,

This first video shows how my to do list expands (as I add new tasks) and contracts (as I complete tasks). 

[NOTE: there is no audio.]


And here's a video that shows how much the overall plan expanded as I worked on it. In this case I do not remove tasks as I complete them.


The main point here is that when we plan, we generally plan at a high level of abstraction that leaves out many of the details and internal conflicts lurking in our project.

As we work on the project we uncover these, and must plan solutions to them on the fly.

This takes time, and generally adds to the amount of time originally projected for finishing the project.

BTW, the planner in the video is an older version of the Fractal Planner.
6.7k Views · 18 Upvotes

Related Questions

Related Questions