Engineering Management: Why are software development task estimations regularly off by a factor of 2-3?

Let's take a hike on the coast from San Francisco to Los Angeles to visit our friends in Newport Beach. I'll whip out my map and draw our route down the coast:


The line is about 400 miles long; we can walk 4 miles per hour for 10 hours per day, so we'll be there in 10 days. We call our friends and book dinner for next Sunday night, when we will roll in triumphantly at 6 p.m. They can't wait!

We get up early the next day giddy with the excitement of fresh adventure.  We strap on our backpacks, whip out our map, and plan our first day. We look at the map. Uh oh:


Wow, there are a million little twists and turns on this coast. A 40-mile day will barely get us past Half Moon Bay. This trip is at least 500, not 400 miles.  We call our friends and push back dinner til Tuesday. It is best to be realistic. They are disappointed, but they are looking forward to seeing us. And 12 days from SF to LA still is not bad. 

With that unpleasantness out of the way, we head off. Two hours later, we are barely past the zoo. What gives? We look down the trail:


Man, this is slow going! Sand, water, stairs, creeks, angry sea lions! We are walking at most 2 miles per hour, half as fast as we wanted. We can either start walking 20 hours per day, or we can push our friends out another week.  OK, let's split the difference: we'll walk 12 hours per day and push our friends out til the following weekend. We call them and delay dinner until the following Sunday. They are a little peeved but say OK, we'll see you then.

We pitch camp in Moss Beach after a tough 12 hour day. Shit, it takes forever to get these tents up in the wind. We don't go to bed until midnight. Not a big deal: we'll iron things out and increase velocity tomorrow.

We oversleep and wake up sore and exhausted at 10 a.m. Fuck! No way we are getting our 12 hours in. We'll aim for 10, then we can do 14 tomorrow. We grab our stuff and go.

After a slow slog for a couple of hours, I notice my friend limping. Oh shit, blisters. We need to fix this now... we are the kind of team who nips problems in the bud before they slow our velocity. I jog 45 minutes, 3 miles inland to Pescadero, grab some band-aids, and race back to patch up my friend. I'm exhausted, and the sun is going down, so we bail for the day. We go to bed after only covering 6 miles for the day. But we do have fresh supplies. We'll be fine. We'll make up the difference tomorrow.

We get up the next morning, bandage up our feet and get going. We turn a corner. Shit! What's this?


Goddamn map doesn't show this shit!  We have to walk 3 miles inland, around some fenced-off, federally-protected land, get lost twice, then make it back to the coast around noon. Most of the day gone for one mile of progress. OK, we are *not* calling our friends to push back again. We walk until midnight to try to catch up and get back on schedule.

After a fitful night of sleep in the fog, my friend wakes up in the morning with a raging headache and fever. I ask him if he can rally. "What do you think, asshole, I've been walking in freezing fog for 3 days without a break!" OK, today is a loss. Let's hunker down and recover. Tomorrow we'll ramp up to 14 hours per day since we'll be rested and trained... it is only a few more days, so we can do it!

We wake up the next morning groggy. I look at our map:


Holy shit! We are starting day 5 of a 10 day trip and haven't even left the Bay Area! This is ludicrous! Let's do the work to make an accurate estimate, call our friends, probably get yelled at, but get a realistic target once and for all.

My friend says, well, we've gone 40 miles in 4 days, it is at least a 600 mile trip, so that's 60 days, probably 70 to be safe. I say, "no f--ing way... yes, I've never done this walk before, but I *know* it does not take 70 days to walk from San Francisco to Los Angeles. Our friends are going to laugh at us if we call and tell them we won't see them until Easter! 

I continue, "if you can commit to walking 16 hours a day, we can make up the difference! It will be hard, but this is crunch time. Suck it up!" My friend yells back, "I'm not the one who told our friends we'd make it by Sunday in the first place! You're killing me because you made a mistake!"

A tense silence falls between us. The phone call goes unmade. I'll call tomorrow once my comrade regains his senses and is willing to commit to something reasonable.

The next morning, we stay in our tents til a rainstorm blows over. We pack our stuff and shuffle off at 10 a.m. nursing sore muscles and new blisters. The previous night's fight goes unmentioned, although I snap at my idiot friend when he leaves his water bottle behind, and we have to waste 30 minutes going back to get it.

I make a mental note that we are out of toilet paper and need to stock up when we hit the next town. We turn the corner: a raging river is blocking our path. I feel a massive bout of diarrhea coming on...