Software Matters! - The Difficulty of Software Engineering

26 July 2012 Mark Perry

Software engineering is a difficult, complex and intellectually challenging discipline. To discuss the challenges of software engineering and why it is hard we must first have a definition of software engineering as a basis for discussion.

Definition

Software engineering is that form of engineering that applies:

a systematic, disciplined, quantifiable approach
the principles of computer science, design, engineering, management, mathematics, psychology, sociology and other disciplines
innovation and inventiveness

to creating, developing, operating, and maintaining cost-effective, reliably correct, high-quality software solutions.

Subfields

The field of software engineering can then be decomposed as:

Theory: mathematical theories, program verification, construction, derivation, generation, and transformation
Technology: the effectiveness of software tools
Methodology: the effectiveness of systematic procedures
Management: the effectiveness of managerial techniques to help people and groups of people produce quality software
Production of software artifacts: the actual development of a particular instances of software

All of the subfields are concerned with the production of quality software. All those involving tools, methods and techniques are concerned with effectiveness; it is illegitimate to claim they work unless demonstrated that they do so. The usage of opinion in substitute of demonstrated effectiveness is still a pervasive industry problem today.

bq. "The required techniques of effective reasoning are pretty formal, but as long as programming is done by people that don’t master them, the software crisis will remain with us and will be considered an incurable disease. And you know what incurable diseases do: they invite the quacks and charlatans in, who in this case take the form of Software Engineering gurus." - Dijkstra

Expert Opinion

Software engineering … is the part of computer science that is too difficult for the computer scientists.

— F.L. Bauer

Donald Knuth (most famous for his landmark three volume encyclopedia of algorithms) explained at a 1989 keynote talk that one of the lessons learned from the ten year development of his TeX typesetting system is that "software is hard".

What were the lessons I learned from so many years of intensive work on the practical problem of setting type by computer? One of the most important lessons, perhaps, is the fact that SOFTWARE IS HARD…. From now on I shall have significantly greater respect for every successful software tool that I encounter. During the past decade I was surprised to learn that the writing of programs for TEX and for METAFONT proved to be much more difficult than all the other things I had done (like proving theorems or writing books). The creation of good software demands a significantly higher standard of accuracy than those other things do, and it requires a longer attention span than other intellectual tasks.

— Knuth

Figure 1. Donald Knuth

Fred Brooks, famous for his "No Silver Bullet" paper in 1986 argued the complexity of programming can be divided into two groups:

accidental: arising from the production of software
essential: inherent in the nature of the problem

Figure 2. Fred Brooks

Accidental complexity can be addressed and resolved, e.g. the details of progamming in a lanuage. However, essential complexity is caused by the specification, design and testing of the software abstraction. The construction of the program and verification of the program to it’s specification is accidental complexity. Getting the abstraction right will always be hard with no silver bullets.

The components of essential complexity are:

Complexity
Conformity
Changeability
Invisibility

Complexity

Software is more complex than any other entity constructed by humans "because no two parts are alike".

bq. "Good software practice mitigates against duplication of parts since two parts that are similar are collapsed into a single callable routine in which the differences have been parameterized. In other more physical, human-built entities (bridges, buildings, cars, computers, etc.) the most obvious characteristic is the repetition of parts." - Brooks

Advances in mathematics are due to ignoring complexity not relevant to the problem at hand. In software, ignoring complexities yields an incomplete program that does not solve the original problem. Yes, abstraction is essential for the decomposition of software. However, ultimately these ignored details must be considered. The complexity of software cannot be escaped.

Conformity

Whenever a software system is built, if anything needs to be bent to get the hardware, software, firmware, and peopleware to co-exist the software is that which changes. Software is far more malleable than are hardware, firmware, and people. Hardware and firmware come with predetermined behavior, and the behavior of human users is limited. Ultimately, the software is adapted to conform to the rest of the system.

Changeability

Software is subject to change far more than other technology. Once delivered, computer hardware and buildings change infrequently. The cost of changes is too large to do so. In practice, the cost of a change in software is high. The cost of maintaining a program over its lifetime is more than the cost of developing it and due to ripple effects and coupling, the cost of making a minor change can be major. However, a common perception is that the cost to make small changes is small.

Invisibility

Brooks observes that "software is invisible and unvisualisable". For objects with a geometric reality, such as buildings, diagrams are faithful, complete, and useful representations from which inconsistencies and omissions can be gleaned. Software has no physical reality. Apart from the source code itself, software has no complete representation. Diagrams only capture particular aspects of a system. No set of diagrams captures all aspects of the system, their mutual inconsistency almost assured. Our ability to detect inconsistencies and omissions in them is limited.

There cannot be as dramatic an improvement in software simply because the human brain cannot be improved by the orders of magnitude that are required. Scientists know we cannot get smarter. Progression is made by standing on the shoulders of giants and moving technology progressively forward. Human brains can tame accidental complexity, but will always struggle with essential complexity.

Theorem Proving

It has been demonstrated by the Howard-Curry isomorphism that programming is theorem proving and vice versa. It is known that the existence of a program to satisfy a given specification is undecidable. Also undecidable is whether a program satisfies a given specification. An algorithm cannot, in general, generate a program to satisfy a given specification. The existence of a program that satisfies a given specification can be demonstrated only by a special case proof. Proving a theorem is exactly writing a program that meets a given specification. Both are of equal formal difficulty.

Wicked Problems

Many problems addressed by software engineering are considered by many to be wicked problems with the following characteristics:

The problem’s definition and solution must be carried out concurrently.
There is no unique definition or unique solution for the problem.
There is always room for improvement in any problem definition and solution.
The problem is complex because it is composed of many interrelated subproblems.
The problem has not been solved before and is unlike any other that has been solved before. It thus requires new approaches, and the resulting solution is not likely to be applicable elsewhere.
Many parties with differing priorities, values, and goals have a stake in the prob-lem and its definition and solution.

Wicked problems defy formalization, if a formalization is available, it can be improved. It is unlikely that the formalization can build on the existing body of theory, and thus the formalization is built from the ground up for each new problem. These properties make the work harder than that usually performed by mathematicians. In many senses programming is harder than proving theorems, mainly because it often deals with problems that have yet to be formalized.

Summary

Software engineering, the use of mathematical ideas and abstractions, is different from more classical kinds of engineering, which work with physical substances and objects. Those differences are what make software so complex and software engineering so deep, interesting and intellectually challenging.

Bibliography

Boehm, Software Engineering Economics.
Berry, Academic Legitimacy of the Software Engineering Discipline.
Brooks, No Silver Bullet.