Yves here. Object oriented programming was supposed to alleviate some of the problems Goodwin describes. I wonder why it failed to live up to its potential.
By Bob Goodwin, an investor and medical device entrepreneur who lives in Mercer Island, Washington
Inside the software engineering field there are several huge paradigms with cult-like followers that continue to evolve in parallel with each other. Each seeks to master the process of developing software, as words like agile have been promoted in the English language from an adjective to a value system. But are any of these approaches adequate?
The goal of each of the processes is to make software production more predictable. Predictable has slightly different connotations in Computer Science, in that we are trying to extract useful information early and adapt to incomplete information elegantly. But from an outsiders point of view predictability means that the software had more of the correct behavior (features) and less of the incorrect behavior (bugs) and more consistency between how people who are collaborating measure good and bad behavior. Methods that are in vogue today seek fast adaptability and fast feedback on any divergence from expectations. Some of the practices that emerge might seem counterintuitive. Putting two programmers in front of the same screen decreases productivity by two, but causes such a massive increase in detecting divergence that it pays for itself. Writing all tests before writing the code that is tested is also massively inefficient, and its primary benefit is that is avoids the common case where a programmer believes the first thing he creates must be correct. Tests are less likely to hide bugs when they are written first, and when a test fails, the programmer is more likely to fix the bug than the test if the test is written first. Inefficiency is again trumped by improved feedback.
Predictability, to computer types, speaks nothing of schedule or on budget. Over 30 years we have gone full circle on how to talk about time and budget. As wild eyed hippies we confidently asserted our true belief that a completion time was unknowable, but was certain to get later each time you asked. Now as seasoned and practical professionals, backed up by scholarly data we say exactly the same thing. Our paymasters then were less technical, older and openly frustrated. Today they are the same, except younger and silent in their frustration. It is highly unusual to hear the word dollar come out of the mouth of a software engineer unless he was referring to a unit for a value of an item within a program.
In fairness, software is complex. Even simple software is complex. Most of the obstacles that are overcome are similar to the obstacles that a tradesman finds when he discovers that something won’t fit where it was supposed to and starts to improvise, or back off. But the software engineer is often working in a massive shanty town, and improvisations can take him up and down alleyways until he has a set of a dozen fixes that resolve a conflict without inducing excess risk.
It really is impossible to know how many obstacles will be overcome. Even in excruciating well thought out modules, there are unintentional and unstated assumptions. What if you had designed a building, and after construction began, were told it had to withstand the impact of a large airplane? Independencies that required reanalysis, would overwhelm you, before considering how disorientating it is to re-conceptualize something massive after it is half built. Even though software engineers feel like we get hit with that magnitude of spec change all too often, we usually pull it together given enough time. It’s just that nobody has learned how to simultaneously create software, and also know when it will be ready for customers. Big tech rarely manages to dates anymore, and when it does, it is usually with the explicit understanding that features will be sacrificed as necessary whenever the date is given primacy. Gone are the days when the boss asks for it all.
Again, the question comes down to root causes of the crisis. And this is where I am going to deviate from my peers and professors, and start speaking to people who buy or use the software rather than restating conventional opinions from our trade. What is wrong with software? What is the root cause, and what is the solution?
There are three problems with software. Please visualize a medium sized software project as being the approximate complexity of all the buildings, roads and Infrastructure of major city. The ‘typical’ unit in that city is probably a single family residence with a small lawn.
ISSUE #1. Need for building codes. Any building inspector or contractor could use standard tools to open up a wall in any house and will find 2x4s and sheetrock and copper pipes, electrical wire and 110V outlets. Different countries may have different conventions. There might even be some regional differences. But the convention of using 2x4s with a fixed spacing was settled at exactly the moment in history when housing became a commodity and the commodity became affordable and predictable.
Software engineering is currently organized like a massive shantytown in Bangalore, and I doubt there is a city planner in Bangalore who would even attempt to do a planned infrastructure update within a shantytown. Most everything in an American city is conventional across that city, and largely consistent across the country. There are lots of roads and apartments and shops. A few malls and factory, and the odd infrastructure items like sewer plants and flood control. The conventions get more specific as the infrastructure becomes more specialized, for example sewer plant, but convention still exists. Planners with expertise on single family homes are transportable across the country, and can make plans from their offices without ever consulting a craftsmen.
Software needs building codes, because we are building infrastructure that must be controlled by people who are entirely outside the field of software engineering. Software engineers can still build things that do not require building codes, just as people can build treehouses and experimental aircraft. But you probably can’t rent out a treehouse, and won’t find many takers for transportation via a blimp.
Building codes are needed to enhance the predictability of commerce. The very best module of software that is produced in the very best tech company today probably has building codes that its authors are very proud of, and they have certainty were a big part of the excellence of the package. But agreements amongst artisans on how to lathe consistent components is microscopically less interesting than conventions that unify all software. And the irony is that in the conventions of these excellent projects conflict with projects right down the hall, and a big chunk of the effort of our very best engineers is the conflict that arises from these small variances in convention. True reuse of code and having fungible talent are essential to fill the gap between the supply and demand for talent. This is only possible by creating building codes.
The benefit of standards also accrues back to the programmer, and this is where the biggest win occurs. Complexity quickly cascades, and consistency in patterns creates consistency in expectations and consistently in how pieces work together. There are fewer surprises to think about, fewer scenarios to analyze, fewer options to consider, and less opportunities for hidden problems to cascade to other pieces of software. There are a limited combination of ways to connect Lego blocks. While these codes are being developed, great effort is spent to analyze the consequences of each combination. If the choices and outcomes are enumerable, they can be remembered and consistent across the trade. This vastly simplifies process of examining options, and creating consistency of outcomes. There probably are 50 or 100 types of Lego blocks commonly used for single family homes, and maybe a different set for roads. A craftsman will still need to master his tools, but now has the confidence that following convention will reduce unexpected outcomes.
ISSUE #2. Standardize transparency. In software engineering there are three things happening in complete isolation from each other that are often dependent on each other. The fact that a software engineer has learned to work within these abstraction, can think about each independently, and make each work together is a testament to the skill. There is the code, or the software, which in our city is the buildings. There is the data that the software is using at any time, which in our city is the people and goods moving around. Then then there is the user interface, which has no parallel in our city, because the city is not serving customers through an interface. But perhaps if our city was Las Vegas, we could visualize our user interface as the Strip and its casinos.
To reduce the load on the engineer, and hopefully even allow non-engineers to regain control of their city, we need to stop making people work in the three abstractions one at a time. I cannot drive while looking at a map and also giving a presentation over a cell phone. I cannot do this even if all three are the same topic. Transparency is the ability to see everything in your city at any time. Transparency is also having all three abstractions working together in a way that gives non engineers visibility and control within their city.
Things go wrong all the time in every city. How does a city adapt to small problems? A problem with food can escalate to a health evacuation. A problem with transportation can disrupt a major show. The answer is that a city adapts because it watches for problems and escalates solutions. More precisely a cities people do all of this. A person can look in a kitchen and see what is missing, go down to a store and see what they have, and fill the gap. Nowhere in this corrective action was there a requirement for a craftsman to modify a building to fix a problem. Transparency allows a person responsible for a process to buy milk, and give it to the kitchen. People are the real problem solvers, and their absence in large software systems are a large reason why they are so brittle and prickly.
Organizing software for transparency is done explicitly to allow normal people to view and control the movement of data through consistent patterns of buildings and roads across an entire city. It is especially important that this visibility and control allows regular people to directly impact the user experience.
Issue #3. Isolation of complexity. There are always specialized and intricate pieces in any city and in any project. An auditorium and stage may some complex special effect. A tall building may require special elevators. The sewer system may have unique technology. Any person in a city would naturally be able to distinguish between the conventional and the special. That same person would also assume that it was special because it could not be conventional, but was important anyway. Lastly that same person would automatically defer to an expert on all things special, while happily make assumptions about all things conventional. The same must occur in software engineering.
Complexity should be explicitly made impossible when using Lego blocks according to building codes. Anything that is not built with Lego blocks should be considered a very dangerous black box, which could not easily breached accidentally.
All three recommendations I have given above are certain to happen in my lifetime, but I am pessimistic they will be driven from within. External policy makers are slow by design, wanting to build consensus on both the problem and solution, and to ensure that policy can be implemented. At least now we have a name for the problem. “Code is Law.”
In the next section I am going to discuss government policy and how it is impacted by the challenges facing software engineering.
96 comments
Comments are closed.
Yves,
Object oriented programming was one of the mid 90s consensus item and was awesome. The objects have been adapted to cross the cloud, but only as extensions of their original purpose. OOP does not solve the newer problems in their current form.
There is an immense amount of non-OOP software written every year, much to my frustration, and this is an example of the larger problem of shanty-towns. But there were valid reasons made for not using OOP in some cases, just as their might be a case for substandard but safe housing for the homeless, if we did not insist on building codes.
Yves
Object Oriented programming helped solve a certain type of problem. It was incredibly successful in doing that. It obviously didn’t solve ALL problems.
Civil engineering has been around close to 2500 years. There’s lot of discipline in building codes now. Though it took several centuries for everyone to agree on those. Software engineering only about 40. You can’t expect a 5 year old to know and solve problems they are going to face when they are 50. That’s clearly the state of software engineering.
Clearly, new engineering problems are being uncovered and getting solved. Agile (you are right, it’s a sort of a cult now) is the latest in the discipline to answer problems of lack of quality and costly budget and time overruns.
thanks
I knew an Assembler programmer once whom always suggested that we write everything in Assembler (it’s really fast!) because he wanted to extend the length of his programming career. But we always told him that we have OOP languages now like Visual Studio and Java and these development environments come with massive OPP libraries we can use and re-use that are already tested and de-bugged, and the idea is we can then focus on writing modules of code that are doing higher levels of functionality – hopefully all the way up the hierarchy to something the user thinks a computer should do.
But invariably someone else would point out that the user[management] hasn’t adequately specified what they want the system to do. That was always the rub.
oops. OPP s/b OOP. My Visual Studio compiler editor woulda caught that too.
Reminds me of a Linux developer making fun of a BSD developer. In an example of obsoleted methods. This BSD developer was writing assembly code for a certain method because it was fast.
The Linux developer showed that C code was better on all levels, 1) being more readable and maintainable and 2) just as fast -perhaps faster- because of improvements in GCC (the GNU C Compiler) 3) and the C code was portable across architechures (x86, MIPS, ARM, …).
Often, once software tools improve (GCC in this example) other knowledge becomes lost to the tool (Assembly programming) – the ability becomes commoditized, but the knowledge is now centralized.
Hmm – Lots of comments from yesterday are gone. NSA-Edit in action?
— or not enough caffeine yet at my end!
Pretty simple actually. Get some ace programmers who can actually code. Stop outsourcing. In most engineering products the number of ace programmers needed to do the job is of the order of a few 100.
Possibly. It’s true that the most productive programmers are orders of magnitude more productive than the least, a power curve steeper than in any other craft or profession.
Except, perhaps, poetry.
Regarding your comment about ace programmers. It is a very insightful point. I always thought of myself as one of those wunderkind, and too often used it to assert power.
I think I became a grown up the day when the problems got large enough where 10 wunderkind were not enough, and I had to learn skills that that might be otherwise considered common.
But what I also realized, which is the key to your point, is that the only advantage that a wunderkind actually has is the ability to hold a larger amount of complexity fluidly in their mind. Maybe a 2 or 3 times greater capacity can yield a 10 or 20 time increase in efficiency.
The problem comes at scale, which is what this entire article was meant to highlight. Increasingly, and rapidly, the problems can no longer fit in a single brain.
I have a million tricks and conventions I use subconsciously. Collaboration requires conventions be explicit. Mass efficiency requires conventions be universal.
I’m not into the “Ace programmer” canard. You don’t want morons, but organized competant programmers can beat ace programmers.
And complexity is a problem that has been made much easier with FOSS programming – one projects reach critical mass (enough eyes looking at it). Although this doesn’t solve the problem pointed to in your article (unless Healthcare.gov was a project started by a municipality 10 years ago and grew from there.
Unfortunately, I don’t have time to rebut the sheer quantity of wrong in this article. I hope someone else who has experience delivering large software projects will come along to do so.
I will simply state that there is no “crisis” in software engineering. Software engineering practices have improved remarkably since the famous 1994 Standish Group Chaos study.
What has not improved at all in that time is the procurement and specification process. Software continues to be specified and purchased by people who are utterly unqualified to do so, and I have never seen any attempt by a procuring organization to impose any sort of accountability on those responsible for making critical decisions.
I’ve seen many projects that were doomed to failure from the moment they were greenlighted.
To use a civil engineering analogy, just because you want to build a bridge to Hawaii, and can find a salesman who will promise to build one for you, doesn’t mean it is going to happen, no matter how much you beat up on the engineers.
Michael, I think the difference between our perceptions derives from the definition of the actors and the definition of the problem. What is written about here is ‘a’ crisis, not ‘the whole thing is rotten’ crisis, but there clearly is a segment of the industry where gaps between expectations and outcomes is large enough to merit the word crisis. But I agree is far from universal.
Engineers want to blame the government or contractors, and thus state that the problem is not with the trade. This is where I disagree. It is the interplay between institutions that defines the craft, and I believe it is the engineers who are in the strongest position to solve the crisis, even if ‘they are not to blame.’ It is absolutely silly to expect the contractors to fix it because most of what they are paid for is the cargo cult, and to be the scapegoats.
Had I read the comments up front, I would have been more careful in my use of the term software engineering, which I had meant to mean something more akin to how the term civil engineering is used today. It is not just the programmers and the architects. It is not just the code. It is all of our systems, biases and actions that we use to translate this technology into value. There are many people who are working with software technology who do not see the issue I am trying to highlight, but there are many more that do, and most of what is written derives from a strong stream of thinking already within the profession.
I am sorry that I did not make this distinction clearer in the post.
While I agree that there are often fundamental issues at the specification/design stage of software, I’m less inclined to place the blame on the buyer/procurer. Often software folks have brains that are logic and data driven, at the expense of communication abilities. Even with strong communication skills however, it is an art in itself to possess both the technical (software) knowledge and the ability to discern the features required to draw up specifications that will meet the end user’s needs. It often isn’t realistic to expect users to know enough to know exactly what they need or want. Putting some extra time and talent in on specs on the front end and throughout the process can be repaid many times over in terms of hours and frustration saved.
The Clay Shirky piece from yesterday’s Links post essentially says the healthcare.gov site went wrong because it used the software management techniques of the 1990’s and earlier (the shorthand is “waterfall”), instead of the methods of this millenium (“agile”). Shirky says “By putting the most serious planning at the beginning, with subsequent work derived from the plan, the waterfall method amounts to a pledge by all parties not to learn anything while doing the actual work.”
So Shirky’s saying that there’s a way to do better, and that in fact may people are doing better — that the “software crisis” is over in many places, and could be over anywhere that adopted modern methods. I don’t know that I agree with the second part, but I agree that the crisis is over in many places.
Bob clearly disagrees — but I can’t really tell why, and I can’t reverse-engineer his extended metaphor back into concrete methods for building software.
Building codes: Require everyone to use the same programming language? Software libraries? programming methodologies? Or what?
Standardize transparency: Require a model-view-controller organization, and forbid any one programmer from working on more than one of those? Or since “[n]owhere in this corrective action was there a requirement for a craftsman to modify a building to fix a problem”, Bob seems to be calling for constructing software that (a) has no mistakes, and (b) will never need to do more than was originally contemplated. Waterfall on steroids.
Isolation of complexity: Bob says “Complexity should be explicitly made impossible when using Lego blocks according to building codes”. Software construction is in one sense a continual struggle to do complex things in the simplest possible way — reducing the internal complexity of our code makes it possible to build software that achieves complicated ends. So yes — let’s find more and better ways to reduce complexity. But saying “make complexity explicitly impossible” is on a par with saying “failure is not an option” — as Shirky says, “a fantasy version of how non-engineers should motivate engineers.”
Agile is not a silver bullet. Shocker, I know.
And what software methodolgy advocates and technoocrats never consider is whether the systems should be built at all. The Pentagon would still be the Pentagon even if its procurement methodologies were revised. Who cares about building a better self-licking ice cream cone?
People have tried that at various points. But as you know, software is enormously flexible, and people have prized flexibility over almost all other attributes.
More standardized software, languages and frameworks are abandoned almost as fast as they’re created in favor of things that are initially simpler and more flexible.
It’s like trying to move people out of shanty town into a planned city (maybe like Brasilia) and people keep preferring the shanty town. Worse, you don’t even know why because they keep changing the shantytown they prefer. But the time you’ve beaten shantytown 1, they’ve moved to shantytown 2.
So why doesn’t this happen in the real world too? I think physical reality is too limiting and so after a while, people just can’t live like this anymore. Meanwhile, cyberspace, and software, is just flexible enough, and community of programmers and users just smart enough, to accommodate the chaos.
It’s not just programmers. Users are very demanding and insist on following the latest styles. Even if the latest style is just a Potemkin village where nothing works behind the facade. Then programmers have to adapt on the fly with their shantytown tools.
You wouldn’t do this with your house because its too expensive and physical safety may be impaired. Software is cheap and nobody ever dies because facebook is down. You rarely hear of mission critical software failing, like airplane control systems. Software bugs are inconvenient but society as a whole would rather adapt to it than fix it.
You wouldn’t do this with your house because its too expensive and physical safety may be impaired
But the contractor building your house would! There are very clear and visible differences in the specifications for condos intended to be sold to “investors” and rental apartments, intended to be owned by a business and hired out the next 30-40 years.
“For-Sale” apartments are cheap shoe-boxes with up-market surfaces, kitchen, bathroom and fittings for maximum wife approval factor. In 10 years time, all that shiny will be worn and have to be replaced.
“For-Hire” apartments are solid materials, robust finishes and conservative construction that will not give the owner maintenance expenses over a long period. In 10 years time, not much, besides painting every 5 years, will be done.
You know, you could say the fetish for flexibility may be the downfall of our current society.
In a way, the banking crisis was driven by this as well. A magical world of extremely flexible money that was a shanty town behind the facade.
And in real life, things continue to be decontented to have lesser and lesser substance behind them.
I agree Jonas. To joke that gets recycled amazingly frequently reflects this. We had three ways to communicate, and it was confusing, expensive and caused a lot of bugs. The we all the developers from those three teams got together and collaborated on a unified design. Now we have four ways to communicate.
The ugly point here is that after years of bottom up, standardization only happens top down. The sum of all correct local decisions of methodologies is a macro environment that is badly inefficient. Even if Google internally could overly constrain all of its practitioners into a narrow set of methodologies, it would immediately become a priority of its competitors to undermine these standards in the industry. This is why I see policymakers needing to start making specific demands on software engineering when failing to do so has bad outcomes on the implementation of policy.
Coding for airplanes is interesting because the code is actually mathematically proven not to fail (the core parts that cannot fail). No “whoops, back to the drawing board” here.
And people that can do this are quite rare and expensive.
Well it’s also very time consuming. If that was a requirement for software, we’d have a lot less of it.
In the UK the (shockingly incompetent in my opinion in its execution of… pretty much everything…) government has attempted something that parallels Obamacare but for social security payments. It doesn’t of course have the “Marketplace”, but in its desire to knit together many different, discrete systems it shares many of the same risks and issues.
You will of course be stunned, just stunned, to learn the project is in the process of becoming a monumental failure.
As Bob rightly says, we’ll have to wait for the full autopsy on the Obamacare Marketplace roll out. But in the UK, our National Audit Office has done a fair job on the woes of the social security system project. It’s here http://www.nao.org.uk/report/universal-credit-early-progress/ and if you’re interested in this subject, it’s worth reading the Full Report if you have time.
Of course, those of us who work in the industry could have practically written it out ahead of time, containing as it does the usual litany of all too familiar failings.
Big hat tip to Bob though for brining out two concepts which have lurked in the back of my mind (I would almost say, my subconscious) and brought them to life. In short-form:
1) IT / software has moved past the point of it being a “solution” and is now just another part of whatever “problem” is trying to be fixed.
2) Governments — or even an enterprises with a rigid operating model — attempting to effect change through software development (and the UK’s social security project / the US’s Obamacare are excellent examples of this) face an uphill struggle because they are trying to use the wrong mechanism.
I find it in poor taste for an outsider to offer an opinion that is way better expressed by insiders.
Software complexity stems mainly from two sources: Firstly, software systems contain humongous amounts of details. Secondly, many of the details are interrelated. All software engineering methods, e.g. Agile or OOP, attempt to lower both counts. None of them succeeds although each method has a religious following that puts regular religions to shame.
All software engineering methods are based on the principle of divide and conquer. Agile builds small increments of the system with the participation of the customer. OOP divided concerns and isolates and hides details. Both methods succeed beautifully on classroom systems and hardly matter when one develops huge systems.
There simply is no crisis. We say that we have a crisis of obesity, but we don’t say that we have a crisis of aging. Software is complex and will stay complex. We get old and software gets complex. That’s life! There are a myriad ways to reduce complexity. For instance, reuse existing and well-functioning software or use “industrial” software component you buy from components manufacturers. These techniques help but there is always enough unique elements in software the leave way too much complexity to handle.
Companies such as Google and Microsoft have an interesting way to handle complexity; they hire only the best and brightest. They pay them handsomely and shower them with benefit and luxury. It’s an ingenious approach. Bright software engineers handle complexity way better than the average software engineer. In the 1970, the psychologist Gerald Weinberg showed that the best programmers are 30 times more productive than the ones on the other side of the scale. (Number and statement simplified; look for Weibergm G. 1971 for precise and full details). By and large this approach works for the giants of industry; the approach is not flawless.
Software is difficult to produce and will stay this way unless a revolution that eluded a lot of our best people for more than 50 years will take place. Don’t hold your breath.
I have to agree with you as far as good programmers being 30x better than those at the other end of the spectrum with many (!!!!). Programming, in my view, should taught through an apprenticeship program not through classroom instruction.
But I think you are wrong in dismissing his points I think the items he listed make sense–however, if we are relating it to the Obamacare mess, then no those points are largely irrelevant. The issue there is/was management not coding. After many years doing contracting work for the gov’t that was 90% of the issue in terms of whether an IT project worked or stumbled.
Read tomorrows point where I will try to knit the logic back from Obamacare to software.
Even if the contractors are conspicuously bad, and an obvious scapegoat, if we want to improve our world, we should consider that the engineers are in the best position to solve the problem. I know I have been ambitious to try to tie together such a wide array of actors and opinions, but I think the insights add to the discussion.
Bob,
Valuable discussion and many good points. I’d like to add several more considerations I have not seen mentioned in the dialog so far:
1. Capers Jones has published a number of excellent books that contain an enormous amount of data on software project successes and failures and the factors that contribute to outcomes. Among the most salient as it relates to healthcare.gov (or any other large project) is the fact that 40-60% of projects in this size range fail to meet schedule, cost, quality expectations, and over 20% are cancelled. Success is possible with large projects, but only if known (but infrequently applied) best practices are used.
2. Well proven software cost estimating models developed by Jones, Putnam, Galorath, and others clearly demonstrate there is predictable minimum time and cost for a project of a given size (potentially measured in function points) – we cannot put 5 pounds of sand in a 3 pound bag – we cannot run a 3-minute mile. Clearly the time available to complete healthcare.gov was inconsistent with the size (scope) envisioned – that project likely never had a chance to succeed. Probably some few of the contractors knew it could not be done, but were co-opted by the profit motive. Based on the little information available it is likely healthcare.gov is somewhere north or 30,000 function points – for comparison, an airline reservation system is in this general size range. Imposed schedules are the kiss of death. Unreasonable unattainable management demands cause many failures – even in the rare cases software leadership understands reality and pushes back they are typically over-ruled by the powers that be.
3. As I demonstrate in my book (Managing the Black Hole – The Executive’s Guide to Software Project Risk) success is most likely when projects are managed for Quality, not Schedule. Finding and fixing defects early, beginning in requirements, leads to minimum achievable cost and schedule.
Your point about Google, Facebook, etc. is a valuable insight. First, they have a huge surplus of those very bright people (only a percentage of them turn out to be valuable in producing production code). Second, they are rarely on tight, published timetables or under any budgetary constraints. These operational freedoms give them the flexibility to do the experimentation and learn from the seemingly unavoidable dead-ends of this kind of work.
Is there any way to adpt not being on tight deadlines or budget constraints to govt IT projects?
I can’t see it. That would require a complete reimagination of how govts do things.
In addition, would it actually be an improvement?
Without it, I don’t see govt IT projects as ever being successful (at least to the level people expect).
isnt that the norm for sny software project, whether in government, or business? all IT projects have deadlines, and alll have budgets. most aren’t publicized in any real way. and most seem to fail too. just that most dont get recognized as failing though
Goodwin is not an outsider. He’s been involved in software development since he was a kid in the 1970s and before he went the medical device route, had a senior post at Amazon.
In that case it’s really peculiar that what he’s written has been so poor.
The most charitable explanation I have is that he hasn’t worked in the industry for a very long time.
That factoid gets trotted out a lot. There’s really no evidence for it. The study that is cited was of very poor quality, and not long enough to provide any evidence either way. It’s almost certainly not true.
Microsoft is notorious for the poor quality of much of their code, and also for the awful internal bureaucracy. So this may not be the best example. Google perhaps, but their software is of mixed quality.
There’s no software engineering crisis and, I’m not sure there ever was.
There’s a management crisis. They can not properly manage either software construction nor recruitment and retention of good geeks.
I made a comment above without reading yours. Usually, as I said it is 90% of the problem.
THIS. In my 20 years of working on different sized projects I have never seen one fail because of the technology. Its always management whether project management or upper management, there always someone incompetent involved making decisions.
There aren’t that many great engineering problems. It would be nice to suppose that we were all working on the edge of what science allows but that is just not the case. Most software projects are cookie cutter types that could be implemented by a teenager. Thats why we have terms now like “code monkey”. Its actually a good thing as it shows software has progressed like all other engineering disciplines.
What makes software hard is that it is always changing. If you want improvements, you have to live with change. Software is unique among engineering disciplines in that the pace of change is limited by peoples imagination. Any other type of endevor would require time and resources. You’re a Chemical Engineer and you have a great idea for a new type of distillation column? Get ready to spend months or years trying to find out if it might be successful (on a small scale). Not to mention wiping out your retirement savings.
Software is relatively much easier to change, so people change it. The good changes get picked up by others and survive. It is a constant effort to keep up, but with the ease of access that other disciplines don’t have. There are dumb programmers, just like there are dumb pipe fitters. Both will derail your projects if you decide cheaper is better.
In the end though, its the MBA in charge of the project who decided that the lowest bidder gets the bid who sinks the project. Everything after that is white noise.
Your response feels a lot like someone say I have never seen a house burn down because it was made of wood. It always burned down because it was on fire.
Ok, we don’t blame the wood for being flammable. But can’t we discuss non-flammable alternatives to wood, rather than redirect to a discussion to fire, for which we have less control?
Bingo.
It’s a problem in a lot of technical fields. The skills that make a great programmer aren’t transferable to leadership / management.
The people who love to program resist becoming management because they will lose their abilities to program. The ones who don’t like to program (or are only so-so) jump at the opportunity to manage.
The successful companies have to actively work to build management (and people) skills in their programmers.
NO. To “advance” good programmers into positions requiring social competence they don’t happen to have is a lose-lose situation. Companies that don’t want to suck have to build advancement tracks for their skilled employees that have nothing to do with glad-handing and bureaucracy.
At the same time, they also have to ensure that management knows the technology, as they did two centuries ago before the elites became too effete and arrogant to feel the need to know their businesses on a hands-on basis. Even knowing the tech just well enough to manage upward would be a good step in the right direction. It is a tough balance to strike, especially when some of your most productive people are auties/aspies and generally loath to play complex social games any more than absolutely necessary.
As I said in commenting on the other Godwin screed, the truth of politics and the truth of technology are antagonistic to one another. You will find very few individuals who can do both successfully. And apparently, it is very difficult to inculcate these skills where substantial natural talent is lacking; neither can be done by rote or are affected by credentials. As software projects grow in size and criticality, they will have to be made dumber so more people can work on them and understand them — the KISS principle.
war an politics cohabitated quite comfortably through much of history. I think the assertion is wrong.
You seem to be assuming war is a species of technology. Of course there is military technology, but until rather recently it was fairly stable and well understood by many of its practitioners. I was thinking of contemporary high technology, for example, the implementation of the ACA web site, which had been the subject of our previous discussion. Hence I assumed that reference to this particular kind of technology would be understood.
Your description of modern managers and their lack of grasp on the technical reminds me of the decadent period when European elites completely lost touch with the technology that put them in power in the first place, namely, warfare. There came a point where battles were not even fought anymore. Instead aristocrats lined up their armies across from one another, took a head count and negotiated some sort of resolution based on the relative strengths of each side. It got so ridiculous that much of the decorative panoply of armor, weaponry, and peacockery displayed by the elites were next to useless in actual battle, and they were practically defenseless when they had to fight against real armies. The current decadent trend growing among our elites will lead to similar disasters when their shanty town monopolies come crashing down around the poor bastards that are stuck living in them (us), except our elites will take the money and run.
As mentioned above, the political power and skills necessary to create and maintain a monopoly are so completely opposed to the development of meaningful standards that we will never see standards or anything like a building code take hold until the monopolistic tendencies of the biggest and best in the industry are restrained. so what remains is a political project.
I still maintain, as I said even a year ago, that we should INSIST on a “building code” and permitting process for any software that has anything to do with handling money or critical systems (utilities, oil refineries, emergency communications, aircraft, etc.). I understand that some of that probably exists already (yay for that), but I’m sure improvements could be made if basic standards were developed and enforced across industries and especially if the industry-specific regulators had to answer to a higher authority on code so that there would be some protection against regulatory capture. Yeah, I’m a dreamer…
I agree, management is where the article goes wrong. The problem is not so much that management is bad (though much of it is): it is that one of the primary goals of management – the separation of planning and labor – is bad for software production. Henry Ford’s factories applied the principles of Frederick Taylor, reducing the costs of labor by precisely measuring, controlling, and deskilling it. Software is seen as an industry like automobile manufacturing, where interchangeable parts and repeatable practices are appropriate. But it isn’t like manufacturing (nor like home construction). Nearly half a century of attempts to make it so have failed.
“Software production today appears in the scale of industrialization somewhere below the more backward construction industries.” —- M. D. McIllroy, at the 1968 NATO conference that popularized the term software engineering
“a software factory should be a programming environment residing upon and controlled by a computer. Program construction, checkout and usage should be done entirely within this environment. Ideally it should be impossible to produce programs exterior to this environment.” — R. W. Bemer, 1968
Object oriented development will inaugurate “a software industrial revolution based on reusable and interchangeable parts that will alter the software universe as surely as the industrial revolution changed manufacturing.” — Brad Cox, 1990
“The essence of the software-engineering movement was control: control over complexity, control over budgets and scheduling, and, perhaps most significantly, control over a recalcitrant workforce.” — Nathan Ensmenger, 2010
Software is critical to many large organizations, but the specialized expertise required leaves managers feeling that they are not really in control. Since the 1960s they have stuggled to reassert their authority. If software can be reduced to nothing more than a technical concern, then software workers can be controlled.
It is telling that the development methodologies preferred by management (e.g. OO) emphasize repeatable processes, standardized components and interchangeable workers, while those preferred by developers (e.g. agile methodologies) emphasize agility and greater skills. This article proposes more standardization and control.
The right question to ask, then, is not how to make building software more like building houses or cars. Rather, why have such efforts failed in software when they succeded so dramatically in other industries?
The answer is that software is fundamentally not like houses or cars. Those things are mass-produced. Software is not. Cars are not made of cars: but software is made of software. When development is most efficient, every piece of software is an innovation.
This difference has preserved a certain amount of independence for workers. From that independence has flowed free and open source software; from their ranks have come the likes of Aaron Swartz and Edward Snowden. I shudder to imagine the outcome if Bemer’s vision should come to pass. I do not think software craft workers possess some special character that shields them from the fate of, say, the educated and cultured silk weavers of 19th century Spitalfields, England. Computers were invented to calculate numerically what repeated application of algebra could not solve. The mathematical and managerial intractability of software is not, if you ask me, a bug: it’s a feature.
If you are interested in the social dimension of the software crisis, I highly recommend Nathan Ensmenger’s book The Computer Boys Take Over. Sean Micheal Mahone’s work on the history of computing is also worthwhile (though lacking, I think, a critical edge). As for Taylorism, I believe Henry Braverman’s Labor and Monopoly Capital is a classic.
Management failures are common. And I think management is part of software engineering. If software cannot be managed then it has less value. Perhaps then the fault comes back to the technology, in that it is not yet mature enough to be efficiently deployed. Feels like people like me, software engineers, should shoulder some of that responsibility.
Also, by advocacy for controls was only intended for a certain class of problems, not for all problems. I would not want a nuclear power plant simulation to be built on Lego blocks. I also practiced and taught Agile, and derived many of the benefits it promised. My post is about ACA class of software development which are actually very complex on many levels, but are not intrinsically mathematical or technical. But I do think engineering intensive problems amount to 1% of what we need to build in the next 50 years, and the rest are complex only because of cascading interdependencies of actors, information, politics, and consumers. Regulation of software should only occur where unregulated software harms the ability to regulate.
Interesting points Geoff, but I think there is a distinction to be made between the sort of top-down control over production processes favored and benefiting monopolists and regulatory control that enforces a standard of performance that doesn’t harm the general welfare of the population relying on the resulting technology. That said, I think it’s crucial to understand the former if we are to sufficiently inoculate ourselves against their attacks when we attempt to raise up the political juice to accomplish the latter. In any event, I think I have my holiday reading all mapped out now. Thanks for that.
There is basically one uber-crisis that affects everything in our world today;
At every juncture, management has accepted the challenge to squeeze a penny until it drips dollars.
The PTB have fostered a managerial class that has accepted job one is turning every worker, no matter how skillful or productive, into a day laborer.
Our economic culture has become a religion where cheap is worshipped and grace has been supplanted by greed.
We don’t have the best solutions because we are unwilling to pay for them.
Add to that the fact that any business tree you shake rains buddy-contracts, and you have an explanation for most of our current situation.
It’s too bad that explanations don’t equal solutions.
True, but the sad thing is this approach hasn’t even really saved any money.
Any money that was supposed to be saved instead got turned to bezzle or manager’s bonus or whatever else.
It’s some version of work expands to fill the time available.
I think following part 1 “healthcare.gov issues” with part 2 “why does all software suck” is a non-sequitur, though I very much like this article. Anybody with large-scale software dev experience could see that this project was doomed.
That’s not just a result of incompetence: there is also bad incentivization, and magical thinking about the complexity and consequent delivery time. People got paid to build a crappy website, and as far as I can tell, will be paid again to fix it. Horrific cost and time overruns seem to be a given in most large-scale public procurements, at least in part because the trough never empties.
A lot of software sucks simply because it’s very hard to do well. Trends towards lowering the bar for entry with ‘simpler’ languages just result in a generation of programmers who don’t know how a computer works.
Problems are exacerbated when bean counters treat the work as a commodity, whereby N offshore warm bodies can do the job as well as N/2 experienced onshore developers.
not only applies to public (i.e. government projects). it applies to private ones too. no difference in what type of organization created the project. they all usually fail because some one fails to recognize what is needed to do the project. writing software is more like art than engineering, in that while you can make things easier to do (like code control) the rest still requires skills. its almost 99% never the technology (many claim it is, but they are sold on their technology as the fix for all. about the only thing that might be relevant, is at least size the solution to what you need to do. I.E. dont use pickups to move 1000s of tons of rock. it breaks the trucks and takes to long to do). and software is driven by the users, not the developers, if they dont know what they need (or change their minds in the middle of the project) then it will fail.
dw said “art than engineering”
Actuall it is both art and engineering!
Really good software people tend to have both good right brain skills (ability to see the whole, ability to understand requirements/purpose, ability to relate system “things” to the real world well, good command of language and words) and good left brain skills (logical, ability to abstract/modularize, see the proper sequence of steps, properly categorization).
Ian McGilchrist has discussed this in his book:
The master and his emissary.
mansoor h. khan
Thanks, Watt4Bob. That’s the long and short of it.
And it’s so disheartening for all workers to be just day laborers and yet there’s nothing else we are to management. They’ve made that clear and then they pat themselves on the back for doing so.
And we all live with the consequences, at work and at home.
At every job people are shorthanded and there’s no slack in the system. With no slack in the system, bad things happen. And that includes you and me if we someday wind up in a hospital with overworked nurses and aides.
I remember a time when there was enough slack in the system that there were periods with little work where people would take pride in examining things and improving them.
Wouldn’t happen today.
I believe a certain amount of slack in the system was an institution’s “seed corn” and guarantor of quality, but we ate the seed corn and started cracking the whip daily over workers and we kid ourselves we are the better for it.
they want life, and all of its details, to be run at the pace of a machine. they want us to be machines.
just like that article someone linked to yesterday re: Big Data. he expressed that to the minds of these types (the same workers as this article discusses) there is NO extraneous variable. all variables affect each other, they just haven’t figured out how yet.
this is more machine-like thing. and it’s glamourized.
you want to go back to being human? put the machine back into its place as a tool. much like stuffing djinns back into bottles, probably impossible. can we un-invent the knife? the pencil?
I found the article guy’s metaphor to be strained. yes, some managerial overlord WANTS it to be this way, and there would be certain advantages to having it this way, but from a total outsider’s point of view, software is nothing like a building.
you can use some (not all and perhaps not even many) techniques to build a house as to build a football stadium. they are vaguely similar in that way–at least there are codes that govern both of them. but would you use the same codes to build a racecar? a rocketship? the mars rover? a bridge? an airplane? a sailboat? yes, there’s overlap, but can you make one ‘thing’ (which sounds to me like some kind of programming AutoCAD) which can do them all and use the same batch of codes to do it with? much less, unknown project of the future that is so particular to one specific situation.
gov’t DOES need the kind of software he’s talking about. and that’s because they need all of their stuff to talk to each other, eventually.
*disclaimer–I couldn’t program you a blank screen on the internet.
You express derision at the common perception of big data, and as an insider I believe the perception is actually wrong. Data is not an unlimited pool of useful and actionable information that is just waiting for the correct algorithm, that if collected long enough can be exploited ad-infinitum.
In truth the value of data has what is called a sigmoid function, which essentially means that the value of some data is extremely easy to harvest (I went to work yesterday, so I probably will tomorrow). But at some point the cost of extracting additional information falls off a cliff. We will never have a big data application that knows every word spoken by man, and then can predict the next word you will say. Extracting predictive value from data becomes hard, and then quickly impossible.
A wise man once told me;
“If you ever have some information that you think is valuable, act on it immediately because it might well be worthless tomorrow.”
I am curious why the Systems Engineer is left out of the article. One of his functions is to be the voice of the User of the software. Making and executing test cases to prove that the User requirements are first discovered, then tested, verified, validated, and documented. It is well accepted in industry that the cheapest place to correct mistakes is at the earliest point of detection. That may not be a politically happy point. …
Tomorrows post will knit in government policy and Obamacare. So I purposely lumped (but I now see confusingly) many groups of practitioners into a single bucket. Management, System Engineers, Program Managers, Product Managers, Architects, Systems Engineers, Applications Engineers. I agree there is a wide range of roles, skills, pay and influence in these group. But all have significant impact on the engineering of software, and in fact almost all of them are engineers, and earn my respect every day with their analytical skills, knowledge of the trade, and passion for their work. I do however separate the institution of big tech (Google, Microsoft, Facebook, Apple, Amazon, etc.) from software engineering. Big Tech has nearly monopolized the top talent. Their goals are different, but interdependent on the institution of software engineering.
I’ve been a software engineer for 11 years, and most of my projects have been successful with defect rates far lower than the norm. You could write books about the benefits and disadvantages of approaches like Object Orientation, Test Drive Development, Functional Programming, etc.., but these things are just tools. In my experience the core cause of most software failures is that people lose sight of the fundamental thing they are trying to do.
The bottom line is that all software is a human configuring a binary state machine to produce a given set of outputs from a given set of inputs. Also that there is no configuration for this machine that does not involve trade-offs between space, processing power, and latency, so you need to tailor the solution (and the trade-offs it implies) to the problem you are trying to solve.
The best software engineers are the ones who accept this fundamental truth about the thing they are trying to do and then use it as the lens through which they evaluate all the tools, techniques, and development processes that other people present as the single solution for everything. Subsequently, the best ideas I’ve seen recently in the techno-sphere for software have been those around mechanical sympathy.
http://www.infoq.com/interviews/martin-thompson-Low-Latency
“In my experience the core cause of most software failures is that people lose sight of the fundamental thing they are trying to do.”
What is the point of the ACA website? For Obama, the website is meant to provide a political win whether through the illusion of a hip experience which will appeal to kids or a way to count people as success stories.
Then of course, the natural progression is what is the point of ACA.
“In my experience the core cause of most software failures is that people lose sight of the fundamental thing they are trying to do.”
Exactly!
What we’ve lost track of in this discussion, and every discussion surrounding the ACA, is the fact that the main issue is one of economic justice, not software development.
What is badly needed is a healthcare delivery system that is affordable and easy to use for every person in this country.
What we’ve got is a government captured by a rapacious insurance industry that takes more and more of our money and doesn’t deliver the promised healthcare.
The ACA is a complex and expensive scheme to both allow the continued looting by health insurance companies, and provide the powerful illusion that people in government and the health insurance business are trying to ‘fix’ what’s wrong with the American healthcare system.
We have to accept the reality we are facing here;
Nobody in either government or industry is trying to ‘fix’ anything, they are preserving, and providing cover for an on-going economic injustice.
To expand on your Mechanical Sympathy analogy, I’d say we the people have been driven hard and with no maintainance for decades now, and it’s no longer possible to hide the wear and tear.
As I commented above, this discussion is not about the inability to develope quality software, it’s about the inability to sympathize with our fellow man.
….’there is no configuration for this machine that does not involve trade-offs between space, processing power, and latency, so you need to tailor the solution (and the trade-offs it implies) to the problem you are trying to solve. …’
Ultimately, however, all problems of social production are political, because politics is the theory and practice of whose will shall prevail in a community. If all or most of your projects have been successful, and you did not do a lot of politics, then someone or something was protecting you by doing the politics for you. Otherwise, you would have been confronted with problems which cannot possibly be solved technologically, like changing, uncertain, and contradictory goals, inadequate resources, external interference, sabotage, and so on. Programming and other parts of ‘software engineering’ and ‘computer science’ do not take place in an ideal vacuum: they are not distinct from other kinds of human social life, but are affected by the same political (and psychological) problems.
Anarcissie said:
“Ultimately, however, all problems of social production are political”
yes. that is correct. but the real question is:
Is it possible to manage our human nature better?
For past 40 years I have seen the Western culture become extremely relativistic and almost nihilistic. The question is never:
Can utopia be achieved? through practicing whichever “ism”.
But can we signaficantly improve our situation via a better culture/values/system/government? Is it possible? or are we foreever just as screwed as we ever were?
Mansoor H. Khan
The human genome can’t be changed much or quickly, but human culture changes all the time, so I have some hope a less destructive one can be encouraged. I would not call this hope utopian, however.
The reason technologies like OO do not prevent software project disasters is because disasters are not a result of technology but a result of bad management. To build large scale systems you need highly talented and motivated developers. But let’s face it, nobody like that would want to work at large consulting firms where promotion is based more on self promotion and brown-nosing than technical ability. In fact, highly skilled technical people are usually regarded as ‘difficult’ and ‘not team players’ in that kind of environment, which is focused not on building a quality product, but on maintaining and increasing monthly billable hours.
I have been developing software for 40 years. I started with assembly language and now use object oriented languages like Java.
No matter what development platform you use and what QA rules you follow, the end result will be poor without high quality project management and high quality personnel working on the system. Complex projects have to be run with regular design meetings to make sure all the bases are covered. I suggest anyone that is interested in understanding what happened here, read the Mythical Man Month by Fred Brooks. It was written in the 1960s by the project leader of IBMs then new operating system, MVS, and is still relevant today.
I don’t have details on exactly how this project was run, but the people that are visible have no experience building software, they used many companies to build parts of the system and didn’t appear to have proper specifications.
Most software complexity comes from business clients seeking to have the software fix what are irrational business processes with strong internal political defenders. When a corporation seeking cross-division customer relationship management systems cannot agree on a definition of what a Product is, you introduce complexity in the the IT system, for example. When the defense of each division is that that is the way we have always done it and we are not going to change, you have a political issue that must be resolved outside IT.
The other issue is down-level management trying to avoid scrutiny through roll-up, drill-down capabilities provided for reporting to up-level executives. The fog that can be introduced into reporting requirements is amazing. And since most limited IT projects deal with down-level clients, the possibilities of volatility of requirements increase.
Looking at the issue purely through the lens of project management of IT personnel has only made the situation worse and provided lots of opportunities for methodological fads and the consultants who peddle them.
Hey THD, good to see you commenting here.
Please read my reply to sbranda just above.
“No matter what the problem is, it’s always a people problem.” – ‘Secrets of Consulting’ Gerald Weinberg, 1985.
Jerry Weinberg was one of the first people hired by IBM to do what would later be called ‘computer programming’, and in pretty short order he led development of the Project Mercury telemetry system. He spent a career programming, teaching programming, consulting and teaching consulting. He wrote a multitude of valuable books on the technical and non-technical aspects of software development, and the advice he offers when analyzing these kinds of challenges is “No matter what the problem is, it’s always a people problem.” For example, Object-Oriented Programming (OOP) is a great paradigm for solving certain kinds of problems. However, how OOP gets implemented in a particular organization depends heavily on the experiences and preferences of the executives, managers and technical leads within the organization. OOP is a tool, like a chef’s knife. It’s not the knife that makes the restaurant, though bad, poorly sharpened knives can certainly hurt a restaurant and its employees.
“No matter what the problem is, it’s always a people problem.” – ‘Secrets of Consulting’ Gerald Weinberg, 1985.
Jerry Weinberg was one of the first people hired by IBM to do what would later be called ‘computer programming’, and in pretty short order he led development of the Project Mercury telemetry system. He spent a career programming, teaching programming, consulting and teaching consulting. He wrote a multitude of valuable books on the technical and non-technical aspects of software development, and the advice he offers when analyzing these kinds of challenges is “No matter what the problem is, it’s always a people problem.” For example, Object-Oriented Programming (OOP) is a great paradigm for solving certain kinds of problems. However, how OOP gets implemented in a particular organization depends heavily on the experiences and preferences of the executives, managers and technical leads within the organization. OOP is a tool, like a chef’s knife. It’s not the knife that makes the restaurant, though bad, poorly sharpened knives can certainly hurt a restaurant and its employees.
As a media artist with a strong desire to take complete control of my “instrument” (the computer as audio/visual display controller), I took it upon myself to learn programming, and progressed to the point that I could “role my own” displays quite easily and effectively, first in BASIC and then in the much more powerful C.
When object oriented programming emerged, notably in the C “upgrade” called C++, I was eager to master that as well. I got nowhere, however, and eventually gave up programming altogether. While “object oriented” programming was supposed to make things easier by enabling one to construct “objects” that could easily be reused, it was burdened by so many fussy protocols and other obscure requirements that what originally was both challenging and fun became simply a series of tiresome and frustrating burdens.
My approach to programming, not unlike that of many others, was to build on already existing software that dealt with all the complicated system stuff, like communicating with the display screen, so I could concentrate on the more creative stuff, like outputting and manipulating images and sounds.
But when I tried to apply my usual “bricolage” approach to C++ I was simply unable to get a handle on how it worked or how I was expected to put all the enigmatic little “objects” together. I had the disturbing feeling C++ had been concocted by people with more of an interest in the theory of programming than in the practical application of same in order to get meaningful results in a reasonable amount of time, without having to beat your head against a wall from dawn to dusk and beyond.
While I am far from being a software professional, I do have the suspicion that object oriented programming may in itself have become a big part of the problem we now face.
Another difficulty, which may well be related, is the emergence of programming environments such as Visual C, which are supposed to make things easier, but in fact introduce all sorts of confusing protocols that actually make things harder. I was a lot happier back in the days of plain old C, and never felt any need for objects or “helpful” environments. Just give me a book or CD containing reusable code to get the system stuff out of the way and I was happy as a clam.
Personally, I think what you did is wonderful and only wish more people would try it. One of the strongest concerns I have regarding the direction computing is taking is that computer literacy in general and programming specifically is becoming so far removed from the general public and its needs and so tailor made for resolving problems on behalf of big business and big data only. Problems that Big Business uses invariably to extract more RENT from and more intrusive monitoring of the public and for little else.
In the early 90’s, it looked for a while as if Microsoft was truly going to provide a foundation for the democratization of computer literacy and software development. But then they took a sharp turn in the opposite direction. Soon after, they became lost, like everyone else, in the race to generate web based services for RENT EXTRACTION and to set up an environment – the cloud – where people increasingly HAVE to use them. Most people didn’t even notice this trend and couldn’t care less, they are utterly seduced by convenience, but it comes at a huge hidden cost and partly explains why violations of our privacy create such a collective yawn – people simply are NOT aware of what is going on, of how deep and intrusive the invasion of privacy is.
What the industry now calls easy programing languages such as Java and C# are not necessarily easy for the general public; they are easy for sweat shops full of poorly paid programmers that are hired to crank it out more and more quickly with less and less skill and definitely less and less craftsmanship, but they are not easy for the average person to just pick up. The software industry needs its own Charlie Chaplin to poke fun at the way the software industry is becoming more and more drone like and less and less human in its goals.
Ah, but there are software building codes, so to speak. There’s a html validator that will check if your page meets the W3C standards. There are plenty of static code analyzers, free and $$ versions, that will look through your code and tell you where there might be problems, based on well known design patterns. There are runtime code analyzers, which will tell you where you might have memory leaks or other inefficiencies.
My experience is it’s pretty rare when a software engineering organization actively embraces and uses these tools.
Again, this reads as a distraction from the real issues. “Building codes” are of minor importance compared with professional codes, certification, liability. Software engineering is affected by the same issues as any other engineering discipline – bad management, insufficient funding, lack of standards and regulation – and the broken incentive structure translates directly into lack of liability by the *businesses* distributing software.
“Transparency” cannot reasonably be discussed with a specific aspect of “open source” – the right to audit. In the extreme, no software used to implement policy should be permitted to be proprietary.
The off-hand reference to “Code is Law” is misappropriation of the phrase.
To adopt a concept, as we are beginning to build a more and more “programmable society”, the issues that this article and its predecessors distract from are absolutely essential. But like many commenters above appear do indicate, this post is proposing unconvincing remedies for strawman issues, and while our society suffers severe infrastructure and management issues, they are in no way specific to software engineering.
On “code is law,” ahem:
http://www.nakedcapitalism.com/2012/04/code-is-law-literally.html
http://www.nakedcapitalism.com/2012/09/code-is-law-once-more.html
“unconvincing remedies for strawman”
I think you hit on the words that made me most reluctant to do this post. I could be cynical and say in software engineering every debate devolves into strawman arguments and divergent remedies. But I agree I actually went one step further than that, and did so because I thought the points were important enough to put myself out there.
Software engineering has always told the world that people don’t use software correctly. Software engineering has always told people that reality is different then expectation, and somehow the software engineer ended up with the ability to create reality. Managers fault. Users fault. Bad Spec. Government can’t conform to software. I get it, software is harder than we like to think.
A class of problems is now emerging that are large enough not to fit in a single brain, are extremely subject to failure based on cascading problems across unstable interfaces, are complex enough to defy planning, and are mission critical enough to matter. Of these projects, the only ones that are not seeing high rates of failure are the ones being built at astronomical costs by big tech.
I was trying to tie dots together here. The dots are many, far apart, and not well discussed in public forums. So it was necessary to write in vignettes and glib assertions. But this was the best means at my disposal on this forum to express the issue.
As for object oriented programming, I’d go with Dijkstra: an idea so bad, it could only have originated in California. At its core, I see OOP as a completely broken approach to defining a contract (semantics) i.e. editing a vtable (implementation). There are decade-old alternatives (e.g. signatures) that scale better, and do not suffer from combinatorial issues, and lend themselves to sharing implementation by composition instead of inheritance.
Because there is so much non OO code printed today, I have to agree that OO is not a convention that reached its aspirations. I think the two reasons people find it not worth its benefits are 1) It is harder to learn and use, so introduces some complexity even while promising to remove some later, and 2) By increasing the programming surface area per unit of problem solved, there were a lot of hidden conventions, which created friction during collaboration.
I have been programming for a long time and have just been handed early retirement. So while I understand the game, I don’t have an oar in the water anymore.
The best code I have seen is when the programmer worked with the user. The more layers between production and consumption the worse the result. This is because the original misunderstandings are embedded in the architecture by the time they are discovered. Adding more testing and verification layers makes the product worse because of the additional misunderstandings introducted at each layer.
Then, of course, as the delivery date approaches and the misunderstandings are revealed, instead of re-archetecting the product, it’s patched instead.
I don’t know for sure what the solution is, but I can be sure to find lots of them for sale. Perhaps software failures are a form of market failure? Perhaps software is too immature an industry to have scaled-up as fast as it has?
ACA is a case where “management” would never be able to say what the goals for the website and user interface would be because the essence of ACA is simply to protect Big Pharma, HMOS, and Insurance while getting people to stop demanding reform.
Did the Obama Administration want a website which would help people make healthcare purchases* or did they want a website which would make a political splash by spitting out high enrollment numbers and appearing to be something which would get the kids off their facebook? Obama ultimately wanted a result which wouldn’t offend anyone important and would underscore his view of himself. This is the primary goal of ACA which is part of the reason the website is complete FUBAR. I suspect there was very little direction beyond “get a website for the young people and their computers.” Even the importance of the website for signing up underscores the lack of effort and care when still so many people don’t have access to reliable internet or internet they can use for several hours especially in rural and poor urban areas.
The primary purpose of the ACA website was to project an image of Barack Obama as a hip 21st century technocrat worthy of adulation. Besides hiring questionable companies to do this, I really doubt the programmers had much time or resources to accomplish much of anything once the ACA status quo was settled.
*I’m ignoring the absurdity of this for my purpose.
Software Development is to Engineering as Economics is to Science. We aspire to be software “engineers”, and have come a long way in terms of standardization, building codes, etc., but there is still a long way to go before it is truly an engineering discipline.
Thank you!
I said this in the first post on this subject.
First up- Define “software engineer”
When there are fewer than 1,000,000 different answers, it may be a beginning. Right now it’s marketing speak for “get one of those egg-heads in here”, which in the press is “we have our engineers working on the problem”.
Hardware is engineered. If it bursts into flames, someone has his name and stamp on the design docs.
Why has OO not lived up to its promise? Object oriented design can be very difficult, for one thing. To make an elegant framework that publishes its functionality in an inheritable object oriented manner requires unusual skill and considerable practice.
Also, it doesn’t always lend itself well to groups of Engineers where every one of them is to some degree the designer of his/her own solutions to their part of a project but who are not given time or don’t have the inclination to develop a common set of objects between them.
You can have an extremely smart engineer who might do quite poorly when trying to think in OO. That doesn’t mean his or her solutions to problems don’t represent excellent solutions.
Then, OO’s emphasis on hierarchy doesn’t always adapt readily to modeling every problem. OO code can be wonderful at making tool kits and frameworks, such as a graphical interfaces; things that are specifically designed for subsequent use by other developers. But if your job is to design and implement some one-off that hooks up two protocols, by yesterday, OO design in its formal sense may be more than you want.
Frequently there is confusion about OO languages such as Java and C#. It is assumed that because they are object oriented by nature, their power derives from that aspect. But frequently developers get around this aspect when for what ever reason they don’t want to design actual OO solutions. Make every procedure into an object and voila, you have OO code used in a procedural manner. Each object has little or no relation, and absolutely no hierarchical relation, to the objects surrounding it. They are called from a managing routine with the usual and traditional control flow directives.
But often the greatest power of these “interpreted” languages (run inside a virtual machine – often written in straight C), is in managing memory and other aspects of the code process that languages such as C++ left up to each developer and/or team with extremely diverse results running the gamut from true shanty town to absolutely sparkling and breathtaking Taj Mahals.
As to the three points Mr. Goodwin made, Standards, Transparency and Modularity/Simplicity, I always have problems with software evangelists no matter how brilliant or well presented their arguments are or how good their intentions.
A lot of these suggestions such as enforced standardization, and modularity would benefit business as rent extractor at the expense of innovation (and thus ultimately at the expense of business). It would direct the purpose of software to a narrow band of leggo applications and the result would tend towards an increasingly narrowly skilled set of programmers – particularly with the overtones of enforcement Mr. Goodwin requires of putting his views into practice. How many times have I heard the phrase, “Let’s not re-invent the wheel”, only to find that someone with no special license and no special authorization makes a MUCH better widget than the one that was supposed to be the “standard”. It’s called evolution and yet has the terminal flaw of timidity when forced into a world of GMO’s where even the inventors must be authorized to invent.
Code requirements in the building industry are an excellent example of the dangers involved in this approach. Ten years ago there used to be essentially one book of code requirements for building residential homes. Now there are seven or eight. The average contractor simply doesn’t have time to keep up with it all. Naturally, one can argue that as we develop more and more restrictions on what one can do, and more and more requirements on what they must do, we are making the buildings safer and more energy efficient. And that is true as far as it goes. But we are also making them expensive to the point no one except big business can afford to build them and no one except big business can afford to buy them and they come with so many restrictions about what even the home-owner can do that no one would want to live in them. Indeed, they DO satisfy all the requirements Mr. Goodwin specifies above, but the outcome is essentially that everyone will end up renting them. And as to innovation, forget it. Just plain forget it unless it is officially sanctioned or unless you are very rich (the two are usually one and the same). It may start out that everyone can be “innovative” with software as much as they want to, say in their own officially sanctioned play pens, but end up, as with building codes, being illegal to even own a compiler without obnoxious and often corrupt code inspectors sniffing around at every turn.
Yes, restrictions have costs, and reduce innovation. But while these costs may seem high to software engineering, it is the policy making that wins. It gets the lower costs of efficiency, and it is able to increase innovation at a policy level far more than innovation was lost at a software level. We don’t need better oars, we need better transportation.
I have the answer, and it’s simple.
Create a labor ‘shortage’ (more specifically: don’t flood the labor market).
If technical labor were not in oversupply, then there would be enormous pressures to make software development tools standardized and efficient. Of course, this would also result in much higher wages for software engineers – indeed, for all workers – and a smaller fortune for the sociopathic head of Facebook.
However, by using both immigration and trade, we have opened ourselves up to the third-world labor market. This means that software development can be inefficient, and piecemeal, and require constant re-writes and updates. This is first of all an advantage in that low wages for the many means high profits for the few. However, messy and overly-complicated software tools that require constant rewrites and updates provide another advantage to the big software companies: individuals in their garages just don’t have the manpower to deal with all of that needless complexity.
In high-wage countries food is harvested by a small number of people using machinery, and in low-wage countries food is harvested by armies of low-wage serfs piecemeal. Software development has exactly the same pressures. If technical labor is cheap enough there is no penalty for wasting it. Make technical labor expensive and things will clean up really fast, because they will have to.
Stick with IBM mainframe
Sorry ’bout the post fragment.
Stick with IBM mainframe COBOL. That’s my story (and my skill set) and I’mm sticking to it.
Yee hah!!!!!!
As a physics PhD with 25 years experience working in software engineering, I’ve heard and considered the building analogy before. I always find that it misleads more than it illuminates. Here are some of the reasons.
(1) Building is based on slowly evolving properties. The size of people does not change over time, nor do the properties of wood or the laws of physics. Software is built to run on systems which have increased in performance by many many orders of magnitude in relevant dimensions over just a few decades. Economics dictates that large software systems take advantage of this fire hose of new performance. Software projects that delay time to market in order to build in much future extensibility fail. Nor do the relevant dimensions change at the same rate. Over just a decade this can open up a relative order of magnitude or more between two of the dimensions (say, bandwidth to SSD vs bandwidth to disk). This rate of change means completely reorganized software systems become (un)economic over little more than a decade. Simple scale-up does not work for long.
(2) Software can be copied free, unlike buildings. We never build the same thing twice. True, applications that are popular and similar enough spawn frameworks and automation tools (think thick-client apps during the 90’s and three tier web apps in the 2000’s). Then, given (1), the very monoculture of these opens up competitive opportunities for doing better for some important subclass of the problem space (think, NoSQL databases).
(3) Software system testing faces more challenges than than do physical systems. There are several reasons which bear on this. Building materials have very good continuum approximations–we don’t have to model atomic scale structure to understand when a beam will fail. So far, there are few similar effective descriptions for software systems which offer compositional utility together with the kind of guarantees that can enhance reliability testing (operating systems are relatively good at hiding hardware details–maybe the most important success) Also most software has very complex inputs and outputs, and the flow of information down the layers is also very complex — it’s “all surface” compared with the physical interface of, say, a tunnel or bridge. Think of modeling a fractal bridge as an analogy and you won’t be far wrong. It might be objected that these are deficiencies of software engineering, and not inherent in the undertaking. I won’t argue it, but they are not easily solved, and building analogies won’t help solve them.
(4) Building codes involve the legal system. Now consider for a moment the speed at which the legal system adapts to change in the world (sometimes with very good reason), and the political forces which are involved. There is a huge impedance mismatch between the legal and the techno-economic evolution of (1).
So, it would be nice, and good luck to you with helping it happen. We shall watch your future career with interest!
If you’re driving too fast to be able to pay attention, the normal, logical step is to SLOW DOWN. Cursing the road or the car for being too old and slow does not help.
Wow PhD with 25 years experience? GTFO. Youre points are inane, wrong, and MISLEADING. You clearly DONT have a PhD in physics because if you did, you would never say that testing code is more complex than physical systems.
a). Testing physical systems requires sensors, a-d converters, error fixing code, feedback loops and control systems, etc. To say that debugging software is harder is like saying making an app is harder than making an iPad. If your talking about buildings, then u are also quite wrong; civil engineers spend hours making sure a building is structurally sound. Of course, building a house won’t have much in the level of this since its usually low to the ground, doesnt need to support much weight etc, but building a house is like coding an app: its quick and easy and many template are available.
b). Building and coding is actually a great analogy if you KNOW about both. Clearly, very few people here know about how a building is designed, engineered, and built, so they assume the analogy is fraught with ‘misleading’ comparisons.
Let me elaborate:
1. A building site can be compared to the environment the software will live/run in. Clearly, this determine A LOT, like if the code is going to be OO or not. If we are talking about computer vision, the environment would probably be some ARM microprocessor with 8/16 bit architecture.
2. Based on the building site, the architect constructs a possibility. He might use Revitt or AutoCAD, just like a programmer might use C or C++ or perhaps C imbedded with Assembly (which IS faster btw, if you know anything about low level programming-to whoever claimed that C is faster than Assembly due to a COMPILER. Jesus, most C compilers were BUILT in Assembly)
3. Software can be copied just like a building designs, its only at the implementation level that this deviates. And it is very very hard to copy encrypted code. CryptoLocker (the virus) is encrypted so well its actually considered infeasible to decrypt.
4. Building codes have building codes for a reason. Since improperly built building can have disastrous consequences, most building are designed to withstand 2-3 times the structurally nominal load. Does software have building codes? No why? Because software is akin to the DESIGN of a building. I can design a skyscraper with an inch diameter base if I wanted to. Does implementing software have codes? YES IT DOES! If it didnt, it would be in my right to write code that opened a backdoor into a banking system to trawl for data. Does implementing software on a system you own have codes? No it doesnt because you own its environment and its environment (hopefully) doesnt impede anyone else.
5. Brond states that coding environment changes rapidly, and cites completely irrelevant facts like the current change to SSD from HDD. x86 architecture has been around for a LONG time. Okay, so some old code doesn’t take advantage of your 12 core cpu, big deal. It still runs, or if it doesn’t its probably Microsoft’s fault, not the x86 architectures fault. Building properties have changed tremendously, maybe not on par to with software (because it takes massive amounts of capital to build with the latest materials and it only takes time to code, and in today’s low-salary-wage-long-hour workdays, this is cheap). My friend just won a competition in so called Zero Architecture. Of course building something that has zero emissions and is 100% self-sustaining would take 100’s of millions of dollars to build but the tech is out there, so don;t assume that because people and wood havent changed, somehow material engineering becomes irrelevant.
Lastly, there is this perception that there is no standard in coding, monkey/dumb coders, etc. There are bad architects, civil engineers (see bridge condition in the US) , etc in every profession who don’t follow guidelines. Coder s just happen to get away with it a lot more, but to say there aren’t any guidelines is completely false. Lets say your making a search algorithm that uses a binary tree. There are many guidelines to follow. Of course there are no standard guildelines, because each platform has its own guidelines, but you can compare this to city building codes, which vary from county to county.
I believe the term of art for building codes in software is “Design Patterns”. There are certain patterns that reoccur and these patterns can be reused to solve unique problems in ways that can be complex, but because they follow a well known pattern can be followed by others. That is the standard that is out there. Going beyond the level of design patterns does not work because software programming should never write the same thing twice. Unfortunately, various intellectual property laws combined with corporate legal department policies that can make using opwn source solutions difficult result in that happening all the time, so many opportunities for a mistake to be made just writing the same operation over and over.
Transaprency in software development means you make the code base available to all developers as much as possible. This provides greater reuse and greater visibility thus preventing the rewriting of the same thing described above while putting more eyeballs on things. I don’t understand this separation of UI, code, and data described they are all the same thing and any good software project is working all of these aspects together. Of course, many software projects are not “good” they artificially separate development into many components at a much finer level than UI, code, and data and assign developers to each of these components in the mistaken belief that deep understanding of a narrow component is better than a broad but more shallow understanding of the entire application. Then, spend a bunch of time trying to specify how all of those should work together in requirements and design documents. Then, all the various developers go work and try to build their components according to these documents. Then, at the end assemble it together in a massive unrecoverable failure of a software project.
Isolation of complexity does not mean a whole lot. I prefer design for testability. You design and develop in a way that makes each piece testable as a single unit. Then, you can have a good idea that each of those pieces work. As you assemble many small pieces to create complexity, then those assenblies need to be easily testable. When those tests fail, it is much easier to identify the problem because you can have confidence in the behavior of the smaller pieces. Even if all of that misses a use case, when a user facing problem occurs the unit tests exclude certain possibilities for the source of the error and provide a framework to use to replicate the missed use case and identify the problem.
“I prefer design for testability.”
This is key. You do whatever tricks you again to get things as correct as possible up front (OO, good languages, nice design, Agile development, SCRUM on steroids, code reviews, experience experience experience).
Ultimately though I think the best humans can do is to bumble along, detect their mistakes and keep trying to fix things.
The best programmers I believe are the vigilant ones.
The idea of finally fixing software if we can just get the proper lego blocks in place is not a new one, and it is well-intentioned. There have been many attempts to do so, usually leading into formal methods.
I don’t think it is possible though.
I don’t think we (as in the human species) have the ability to even define what those lego blocks should be. Whatever set we come up with will suffer from the same old bugs we ran into now just as soon as we use those lego blocks in new and unexpected ways that violate their assumptions.
Hmm, I recall writing something similar in comp.risks some years ago. [Digging].Yes, from Risks 23.74 (http://catless.ncl.ac.uk/Risks/23.74.html#subj2.6). I was responding to an article proposing to improve software reliability through the use of the appropriate components (i.e. lego blocks):
> If you have small components that you know are right, and you then combine
> those components to manipulate each other according to their published
> interface specifications, the results should be consistently correct. The
> results will be predictable, the usage will be consistent every time.
This is false. The results will not necessarily be correct at all. “Know are right” is not possible except in very specific and controlled contexts.
When components are used in new situations, any existing assumptions cannot be relied on at all, without tedious and careful work to reestablish them.
Software components are not physical components. They do not scale the same way.
That the software industry does not offer the same reliability and quality as physically engineered products is not because software practitioners are pulling at fast one (although they often are, but for different reasons).
They don’t offer the same guarantees, not because they don’t want to, but because they cannot. Getting software right is hard. Very hard. So hard that even really really smart people are not willing to be on the hook for it.
Customers have to tolerate software with mistakes, because that is the only way they can get affordable software at all. If they insisted on the same guarantees, they wouldn’t be able to pay for it.
Yes, we need to make better software. We need to try. Things can be improved. They must be. The current situation is not acceptable.
But it is not easy. Humans don’t seem to be good at it.
I just found this wonderful quote in the same risks article (http://catless.ncl.ac.uk/Risks/23.74.html#subj2.11):
Years ago I quoted to an architect Weinberg’s line that “If architects built buildings as programmers build programs, the first woodpecker to come along would destroy civilization.” “Oh,” she said, “but that’s just how they do build them.”