Date: 2008
I've been a software developer for 10 years and since I can remember, I've heard this term being passed around as if it were the bane of any software project: over-engineering. In the early years, I was a novice in a shop full of seasoned developers, so I absorbed every bit of sage advice like a sponge. This idea that you shouldn't over-think things because you'll end up with a system that is far more complex than it needs to be made sense. Unfortunately, it seems as though the original intent of this phrase has been lost to the current generation of programmers. From my experience, it seems that far too many developers have interpreted this phrase to mean "do no engineering at all". Complex systems are created one line at a time by trial and error. Let me point out right away that this article is by no means an attack on people developing software this way. This is my attempt to reach out and help improve the state of the industry. Hopefully, it will make all of our lives a little easier.
When I re-examine the phrase now, the entire concept of over-engineering seems laughable. The purpose of engineering is to design an optimal solution to a practical problem. How can you make a system worse by thinking more about how it can be improved? I think this oxymoron of a concept has stemmed from two possibilities. This first is that these developers don't understand the role of engineering and the second is that they don't understand the project requirements.
I've experienced the first condition firsthand. When I was a pup, I thought software engineering was creating software from scratch. I created every bit of it, so I must have been the engineer! Unfortunately, it turns out that those projects simply had no engineer at all. There weren't even specific requirements of the system before I started writing code. I had a rough idea of what it needed to do, so I created it the best way I knew how: by trial and error. This was a gross misunderstanding of the role engineering on my part. I've since learned that software engineering is much like any other type of engineering.
One major qualification required of an engineer is a deep understanding of all of the available materials. If an electrical engineer doesn't understand the difference between paper and plastic capacitors, their circuit may not hold up under certain conditions. If a mechanical engineer doesn't understand the tensile strength of iron vs aluminum, they may not make the best decision when deciding which to use. In either case, it's possible that the person in question could design a solution to the problem; it may even turn out to be a good one. But a true engineer would understand all of the costs, strengths, and weaknesses of all of the available materials. They would be able to make a scientific decision about which should be implemented. This same principle is applied to software engineering. If you don't understand the costs, strengths, and weaknesses of the available technologies, how can you make an informed decision about which to implement?
Another major qualification of an engineer is to understand the problem in its entirety. If an architect is asked to design a home but does not consider whether it will be built on an island where little electricity is available, in a desert, or in the middle of a metropolitan city, it's likely that something won't work out. This is an extremely obvious example, but the point I want to illustrate is that the person who wants the house built may not understand all of the issues that need to be considered. It is the job of the engineer to be able to see these issues in the design stage and raise the appropriate questions when necessary. Otherwise, you may end up with a very expensive sculpture. In the realm of software, a client may ask for a custom CRM package but doesn't mention that they want it to be usable offline. A software engineer needs to be able to foresee this type of issue and raise questions. I've seen too many systems built where nobody stepped back to look at the big picture. There would be an immense focus on getting a small task done and when it was finished, it turned out that it couldn't be implemented as needed or someone on the next team had already solved the problem.
The final qualification I'll highlight is the ability to create documentation. You could engineer the perfect system but if it's not spelled out specifically, details get lost. By the time a project enters the development stage, all of the questions need to be answered in writing. Nobody would expect a contractor to build a structure without a blueprint. Neither would you build a complex circuit without a schematic. The same should go for software. Every detail should be spelled out on paper before anyone even starts dreaming of code. There will be plenty of revisions, for sure. But the problems found at this stage will take far less time to correct than in the development stage.
The qualifications I've mentioned are things that we need to start seeing more of in software development. It's a cop-out to say, "It's my code, nobody else will ever touch it." or, "I don't have time to create documentation.". I've inherited too many extremely complex projects left behind by people who said these very things. I've even been guilty of doing those things in the past. It's time to evolve. To me, the fact that the National Vulnerability Database posts 20+ new vulnerabilities a day illustrates that we're doing something wrong.
Lets make a change.