A friend of mine just tipped me off to this 2001 IEEE article called Frequently Forgotten Facts About Software Engineering that compelled me to write about it here. Although the author (Robert Glass who also wrote a book called Software Runaways) introduces this article by expecting criticism and disagreement, I totally see this guy as spot on.
You can imagine what a software runaway is. Basically, the project gets out of hand. Key decision makers get disappointed as results miss expectations and the project is perceived as a failure. According to this author, the two biggest causes for runaway projects are unstable requirements and optimistic estimations.
Software requirements churn for many reasons. Subject Matter Experts lack clear understanding of the target market's needs. Also, a lot of requirements get discovered in the development phase as the details get unfolded. This leads us to the second cause for runaway projects, optimistic estimates. Estimates are normally done at the beginning of a project; however, a lot of requirements get revealed and folded into the project during development. So, the original estimates are no longer accurate. Optimistic estimates lead to unrealistic expectations which foster the perception that the project is a failure. Also, optimistic estimates lead to premature deadlines. It's amazing to see what bone-headed decisions people make when faced with a looming deadline that they cannot make. It is important to re-evaluate and re-estimate during all phases of a software project and not just at the beginning. Also, wherever possible, consider scheduling requirements for later release.
In the article, Glass presents four important keys to successful software development.*Maintenance enhancements are the most important life cycle phase of your project so choose an architecture and process that optimizes for maintenance enhancements.
Another inconvenient truth that Glass mentions is that there are trade-offs in software engineering. You cannot optimize for both memory and speed. You cannot optimize for both ease of use and capability. There is more to quality assurance than bug fixing and you cannot optimize for all aspects of quality.
I encourage you to read the article for yourself but, if there is only one thing you get out of this article, then I believe it should be this. Learning new tools and techniques are important because it increases your toolbox which increases your chances of finding the right tool for the job; however, understand that promised productivity for these tools and techniques is most likely to be overrated. Also, there may be higher productivity eventually but don't expect higher productivity initially until all the players are up to speed with the new tools.