The Fallacy of Perfect Knowledge

We all know in our heart of hearts that we don't know everything. Every day we hopefully learn a bit more about our profession, our society and ourselves. But we simply can't know it all. If we stop learning we fall behind rapidly, especially in the software industry. The idea that one can apprentice to a trade and practice that trade the rest of one's life has gone the way of the dodo. Remember the dodo? No? That's the point.

Technology, techniques and the ideas upon which they are built change far too rapidly in our era for any practitioner to know all they need to know at any point in time. We must constantly learn and we must equally adjust to a state of ignorance, which requires us to spend some portion of every project researching the knowledge we need. Why, then, do we persist in pretending that we must, or even can, know everything about a software project during its development phase?

The history of software engineering is replete with attempts to control software projects, through carefully bound development and maintenance activities to prevent buggy, failed software. Most such methodologies, such as the classic "waterfall" methodology, presume that with sufficient time and up-front diligence a software project can be completely understood. Many demand that requirements be set in stone before a line of code is written. What nonsense!

Giving up on knowing it all during development, we might think that we can know it all later. Several software development methodologies presume this, such as the spiral or agile methodologies. Iterative development is seen as the key to delivery of a software project encoding "final" requirements. Unfortunately for adherents of those methodologies, delivery of a software project is just a comma in development, not a period.

Requirements, even when "agreed" upon in detailed up-front design, will change during development. It is impossible to know them all in advance. Multiple requirements often result in inconsistencies, even when they are gathered from a single source. Requirements may even mean different things to different people. Differing interpretations may be due to perception, goals. or language. In order to create a successful software project, we must accept and even embrace these ideas. We do not know it all and we never will.

The Fallacy of Perfect Knowledge is the delusion that it is possible to capture complete, non-conflicting requirements for a software project. The reality is that requirements will never be fully known at any time during a software project's lifecycle; not during analysis, development, maintenance nor even (or especially) when the system becomes legacy.

Continuous use of the agile techniques of iteration and refactoring into the maintenance phase of the software lifecycle begin to address some of these concerns. A fuller understanding of the ways that software evolves may be the next step. Until we have those conceptual tools, use them daily, and accept our ignorances big and small, we will continue to fall victim to the Fallacy of Perfect Knowledge.