By Bob Goodwin, an investor and medical device entrepreneur who lives in Mercer Island, Washington
It has been a good generation to be involved with software. The scarcity of the skillset combined with the demand for the output have generated outsized incomes, while the work has been consistently rewarding. Our quirky group of builders has had an outsized influence on our industries, not to mention our culture and ideals. But that influence is looking less benign as the rigid procedures of computing are changing commercial relationships and the application of the law.
In our prehistory, the 50s, 60s and into the 70s, there were at first a handful of computers and eventually hundreds of thousands in the world. In those days data was impossible to crunch, and yet data was impossibly valuable to crunch. Any means necessary included costly, limited and ill-behaved combinations of experimental equipment. Memory was literally sewn together by hand, with a magnetic bead for each bit, and a pair of wires running through it along each axis of a grid. Longer term data would be stored on a rotating drum, and the programmer would try to time the placement of the data on the drum so that data would come around about the time it was needed. I was a teenager in the early 70s, and took a non-credit system programming course at Dartmouth when I was junior high school. But my memory of the time was writing programs with toggle switches, and not keyboard, debugging by watching lights, and patching paper tape programs with tape.
The software engineer had the tedious job of primarily overcoming obstacles, while trying to work the magic of his trade of extracting value from large investments in equipment. The best in the field resembled master craftsmen who could build rocket ships from tin foil and masking tape, and make it look easy to realize the potential of the computer.
But the golden years only started when the computer became a tool for human efficiency. It was easy then because there was so much inefficiency in life, and a simple tool gave its user an obvious and immediate edge. This in turn triggered an avalanche of imitation which changed daily life. The change was not immediate due to limitations in cost and talent, but none-the-less it was a rapid and confusing transition.
An innovation at the time only needed the benefit to outweigh the costs. The costs were astronomical: expensive equipment, steep learning curve, frustration, and lost data. But the benefits, even with buggy software, were even greater. Because the greatest cost for the user was (and is) the learning curve, monopolies were built on being the first with an innovation, giving the second movers in the market an impossible barrier of retraining the customer base.
The software engineer, however, had no barriers. The credo of the day was be first, solve a single common problem for a single user. There would be plenty of time to correct mistakes later. We were called computer programmers then, and were young, smart, opinionated and fiercely independent. The cultural shock from the Vietnam War was still fresh, and young people lacked any identity that might have tempered conceit. There are elder statesmen of software today. Computers, like language and music, must be mastered young.
The vacuum of obvious applications was quickly filled, but under the covers of the trade were a hodgepodge of ideas, methodologies and standards, much like the early days of the railroad where every stretch of railroad line would have a different gauge and standards, and so goods would have to be stored and repacked at crossroads. The true inefficiency of those railroads was that neither equipment nor skills were fungible. The evolving railroad system had high levels of abandoned infrastructure, and a continual investment retraining and recruiting skilled workers. The same is even truer in the software industry. Very little thought was given to this problem, because value was always higher than cost, and demand for talent was always in excess of supply.
The next phase of software engineering involved the connection from A to B. Warehouse to customer, booking to airplane, student to university: the centralization of data. The computerization of the government functions accelerated in earnest at this point. This time still predated the internet as we know it, but computing was now about interconnecting, but not yet in the democratizing sense of the world wide web. This was probably the low point for our field, and a lot of practitioners from this day have been flushed away. All of the limitations of the earlier land rush brought disappointment when things always seemed broken or at least brittle. This was the era when a lot of larger initiatives in computing were canceled or scaled back, and there was a cynicism that industrial management techniques were squeezing the soul and creativity out of building software, and thus the bad outcomes were predictable.
But the problem was with the trade, not the business or management styles. Connecting A to B very soon required Connecting B to C, and so on. There needed to be an agreement on how things should connect generally. If A, B and C were thought of as business desks operating by phone in three countries in Europe, Then ‘B’ might need to be fluent in the cultures, traditions and currencies of three countries. It didn’t take very long where the alphabet extended beyond C, D, and E, and software engineers found themselves writing connection code using multiple protocols, and being ignorant of the subtleties of each interface. Perhaps in France nobody calls you if a problem arises, where in Italy, transactions are silently kept for a week and combined to make later processing efficient. Ignorance of subtleties creates defects that are not detected until they are propagated to other actors. Tracing root causes across a network is a skill yet to be mastered. It was not possible in many cases to trace back from C to B to A. You don’t speak French, even if they would let you rummage through their paper trail. In this hodgepodge, only came greater anarchy, and a grinding sense that while computing was inevitable, the creaking massive gears of technology were as likely to crush as emancipate.
It was the mid 90’s when a consensus emerged on how best practices needed for connectivity of computers. The big consensus with the world wide web, which was actually a set of individual conventions, such as TCP, HTTP, HTML, URL, IP, XML, and of course the browser and java-script. But there was much more, languages (Java, Python, Perl), coding standards, rich and elegant libraries of common functions. There is a massive list of technologies that are now universally accepted. There was a short window of time when consensus overpowered individualism, and then reverted. Of course the browser has blossomed since becoming the window to the cloud, but this would not been possible without those crazy letter acronyms strung together like pearls. This began the second great land rush, and another batch of billionaires were minted, and the connected world of computing was built on the monetization of cloud data.
Only at this point in the story can we see the seeds of today’s crisis. Once, in the not so distant past, the software engineer could invent something that replaced tedium, impacted nobody except the person buying the product, and could help the world without unintended consequences. When those opportunities were exhausted, innovation involved two agents, with only one paying.
A consensus emerged on a set of rules for how to create these new channels of control, but there was no consensus on the rules for the proper use of those channels of control. They were inevitably built to the benefit of the party paying. The value of new innovation was still so high, and the talent still so limited.
So society groaned but adapted as the computer began to assert rigid rules on commerce that had traditionally allowed informal resolutions between people. Commerce had not become more predictable, but the costs were being shifted by technology. We were getting richer fast because of computers, but these same computers were also used to change the relationship from the consumer to businesses, and often between businesses. Efficiency was accruing to the individual, but power was accruing the institutions.
Growing economies are very accommodating to uncomfortable change. Everyone alive today in America uses a computer in some fashion, and is familiar with its lingo, traditions and values. They also know that software is buggy, inconsistent, complex, frustrating and inflexible. If you go inside the beast it is obvious why. We had a big bang of consensus in the 90s. Those gains have already been monetized. Computing is unbelievably different than it was then. Computers are more than 10,000 times faster and bigger, and there probably 10,000 times more computers in the world. Each of those computers probably talks to 10,000 more computers than in the 90s, many trivial transactions on a home PC may aggregate data from 3 or 4 other computers and perform tasks that are entirely customized to you. For all the consensus we achieved in the 90s, the emerging realities have long since outstripped the implausible consensus, and there is a new vast bloom of inconsistency and confusion, far bigger than the last.
The growth of information, and the resulting mayhem, have left us unable to propagate the needed skills of our software craftsman. Even amongst our best craftsmen, it is only with great effort that we create a day-to-day consensus on how to keep the tracks connected and the trains running, without destabilizing the system. These are excellent craftsmen, but each is seeking reliability with different perspectives and different biases, and each has seen the consequence of error a great number of times. As has always been the case, the engineer is given wide berth, but the businesses must go on. Here to the solution has remained consistent through time. Throw more machines at inefficiency, and clump wads of software onto the side of existing clumps of “legacy code” that is too valuable to discard, but too scary to touch.
The brittle code works, mostly. And it works because it has to. When it breaks, everything stops. Software engineers who once might have worked whenever they were sober, started to wear pagers. 5 engineers sharing a pager would allow an engineer to live a traditional engineer’s life 80% of the time. But now there was a week that required sobriety. Sometimes this week did not allow sleep. Engineers are smart. They like to sleep. So the software works. Mostly.
What changed, and is subtle to even most engineers, is that we had begun to depend on software. It was no longer making us better, now we depended on it to always work, and the when software failed us, we failed.
Today we see the Healthcare.gov web site is performing far below expectations. This feels quite a bit like the 90s again. Spectacular gaps between expectations and reality are being exposed. Like the 90s most people are shocked, and root causes will only be understood after lengthy analysis. But in both cases there were plenty of insiders who were very aware of the vulnerabilities of these big projects, especially government projects, due to greater rigidity (for instance, in the contracting process, in legal requirements) and politicization. We are not expressive, because we don’t expect people to even know how to listen to software engineers. We can be pathologically optimistic or pathologically pessimistic, and we can be dismissive. But most of all we are just busy. It’s pretty rough out here right now.
Here I have to go into pure speculation, but I don’t feel far out on a limb. The issues are very consistent across the whole industry. We know that the goal of Healthcare.gov was to instantly present a taxpayer with a set of options provided by insurance companies, subject to state rules, and with subsidies given in real time based on government information residing in computers run by departments within the government.
I have seen sentences like the one I just wrote in power point presentations created by ambitious new employees. I know from the sentence, just like everyone else in the room that the useful idiot giving the presentation will not survive to usefulness. If it were a serious meeting, and there was a serious business requirement, we would politely untangle the sentence and expose the obvious. The existing infrastructure is almost certainly inadequate for the stated task. But even if the legacy systems could be miraculously used, the bloody lessons from building cloud infrastructure have heaped new layers of constraints of how to put together systems that have the kind of reliability, responsiveness, interconnectivity, latency and interactivity of the world class commercial web sites we use every day.
Again, I am actively ignorant. Given the same challenge, and a constraint not to rip up all legacy systems, I would have put together simulations and stress tests, and added redundant pathways and servers, and adding caching, and implemented predictive decision trees. A lot can be done with ingenuity. But they haven’t even built the payment system at all yet.
The fact that this is a government system makes it unique, and I know many are hoping that blame can be confined to government practices or subcontractors. And software engineering will be left on its pedestal. I disagree, with this emerging consensus, but let’s explore this.
The very elite tech companies have been a magnet for the best computer top talent for a long time. It is not this cohort that writes government software. Of course there are good engineers in every school, industry, country, and within each specialty. But there is no question that technical collaboration looks very different between engineers at a big tech company versus a contracting company writing government software. Government processes seem to always flow explicitly from written law or written policy. In other words, Government process are by definition implemented top down. But every well paid software engineer has had a lifetime of practice intimidating his bosses to let them work bottom up. All of the most brilliant engineers I have worked with would be emotionally contorted beyond recognition if they applied themselves directly to processes common in government software projects. Independent of process, government software contracts are not continuous, they are lumpy. So contractors need elastic labor forces. Elastic is not a magnet for skill. It should not be a surprise then that government software mostly works because the bugs were eventually removed, rather than because a master platform had been built to improve the efficiency of projecting law into code.
The Pareto principle, aka the 80/20 rule, gets an especially big vote when it comes to government software. There is always friction between features, quality, and cost/time. Features are almost always the casualty in government software because of the top-down implementation, but also because retroactive changes are tricky in a democracy. The fun part of the hidden Pareto clause, is that software engineers are really good at the analysis, and do it subconsciously.
We more to say about Pareto later, but the end result is code usually falls short of the law, The difference was made by a software engineer early via prioritization of work, but then only discovered after it was too late that features would not be available. Software engineers are not guilty of specifically overriding a policy based on their policy preference. But software engineers do have extremely strong preferences for how data and decisions are stored propagated and managed within a computer, and for the order in which work is staged. These preferences are mostly forged of pain, not ideology, but the result is that we are probably the least adaptable builders on earth.
One other issue is unique to government software. To the government, software is a capital acquisition like an airplane. Big tech has no similar illusion when they manage big projects. We like to say that nine women can’t make a baby in a month. The contracting company’s primary purpose in life is to implement a software project but sell it to the government as if it were a capital acquisition. The processes used in each type of acquisition are very different, but the contractor must create an entire cargo cult to modify appearances. Software eventually completes because engineers are actually professional and competent, but it is painful to watch the process, especially the executions afterwards in the public square when the boldfaced lies force you to look away.
For all the specific handicaps facing government software, the root cause of the problem is that complexity has overtaken convention. Healthcare.gov is just the dead canary. The ominous point is that while policy makers may have ceded control of their institutions to software, it is no longer obvious to me that even the software engineers have control of the software anymore.
It is 1995 all over again for the software industry. And it is bigger this time.
Something new is happening. Software engineers have happily disrupted the world for 30 years. But now we are being disrupted by big data. Computing has moved from PCs with floppy disks to data centers that cost billions of dollars and are mission critical to massive economic activity. The rate of data that is being stored and processed seems to be stretching the supply of talent thinner at an even faster rate. The amount of talent that is left for secondary or tertiary activity is shrinking. Code in 2000 implemented a plausible imitation of law. In 2020 we are too busy to fix it.
So now we know that software is far from a highly elastic and lightweight method of propagating policy and improving efficiency. Instead we know that it is concrete that hardens into strange shapes that quickly devolve into Detroit. But why is that? What is the root cause for these limitations inherent in software engineering? This question is not a bit new to us. This question has been studied and written about extensively. Tools and methods continue to evolve, but at a place slower than emerging challenges. I’ll address this issue in my next post.
115 comments
Comments are closed.
Great post! On legacy code and government projects I suspect that there’s something else happening that’s simpler to understand: nobody wants to work on these systems. Connecting to or maintaining legacy systems is usually not interesting, the pay relatively sucks, the working conditions are likely a dismal combo of corporate and government, and the project/product managers are probably MBA’s whose knowledge of software engineering practices is even less than their sense of humor. Projects like healthcare.gov aren’t the type of work that will attract the best or brightest SE’s, or even their friends or distant acquaintances. Ironically, about the only positive, is it probably comes with a very strong healthcare plan.
I think this blames the employees entirely too much. Because while I have seen not very good programmers. I have also seen again and again and again and again, people who want above anything to do a good job, destroyed by impossible contracdictory demands coming down from on-high. I have seen good employees talents wasted for bad management too many times to count IOW.
Legacy systems are complex, poorly understood, and poorly (if at all) documented. Modifying them is scary: high risk and high expense. Managers would rather work on the latest sexy project they can bullet point into their yearly self review. No one gets credit for fixing software before it breaks.
[W]hile policy makers may have ceded control of their institutions to software, it is no longer obvious to me that even the software engineers have control of the software anymore.
The fact is organizations are not even trying to control the software. It’s no different than the crumbling infrastructure throughout the country or cut-to-the-bone R&D throughout industry. Everything is about short term profit. Legacy systems are hard and expensive.
Programmers aren’t fixing these issues because no one wants to pay them to do it. We aren’t even trying.
I loved this post. It reminded me of Lambert’s “Code is Law” post. Software engineers have lost control of the software because “complexity has overtaken convention.” It used to be that convention, however staid, was code. And code is another word for statute. If software engineers see the value in bottom-up coding and government only wants its quick simplistic efficiency, at some point the whole thing has to become dysfunctional. If “government software works because bugs were removed rather than because a master platform was built to improve the efficiency of projecting law into code” then that really says it all. The software infrastructure is too SIMPLE for the stated tasks. I still conclude that there is no such thing as too complex – only too impatient.
I’m not sure I understand what this is all about… But as an ex-SW engineer, I have a few comments.
First, software is malleable, and thus ideally suited to create Potemkin’s villages. If your contractor builds a bridge or an aircraft carrier from papiermache, it’s going to get noticed pretty quick. If you build a SW that looks nice but does nothing (or nothing well) it can take weeks or months to notice.
The complexity doesn’t help either, as you can always blame the above on the complexity (even if you did it on purpose to start with). Few people can understand complexity, and even with those who can, few can devote the time to do so. So again, it’s easy to Potemkin even more, something that’s much harder (not impossible, just much harder) in the hard-world.
Lastly, most of the large projects gets given to large companies. The only workable theory behind that is the old “no-one gets fired for hiring IBM”. My experience is that pretty much all IT projects that actually deliver on time and budget have less than 50 people on them (the optimal seems actually less than 20, but over 50 it’s doomed. probably because with 50 you can get 20 working and 30 doing pretend stuff easily enough, something not possible with 2000 people). But hiring a small company to deliver Obama care is fraught with political risk, as there’s a very clear “your fault” there, which is easily shifted with a large company on the company.
IMO, the adage should be changed to “everyone should get fired for hiring IBM”, if we want projects that work.
Ad the last point, sw as elastic solution. That was always a pipe dream, for people who never understood complex system evolution. There’s always a tradeoff between solution elasticity (I like a term agility better), complexity, speed etc. Trade offs are rarely made with long-term in mind (fo rexample, if they were, we wouldn’t have etremely poor email system prone to hacking and spamming as we have now), and more a spur of the moment that gets reused, usually in situation which is only marginally similar. The result is that now and then you need an equivalent of cambrian extintion event to get out of a “dead end”, but again, that’s a property of all evolutionary systems.
“no-one gets fired for hiring IBM”
Government projects are unique not because they require complicated software. They are unique because no-one gets fired who works for the government.
Great points. From my POV, as a former gov’t contractor, projects depended on efficient and smart management and not as much on the capacities of programmers/engineers for big projects. The first task is to create a clear way of collaborating and communicating where everyone is empowered–sounds corny but it’s remarkable how rare that is. I’ve seen projects that operate like Ferraris on the road and those that run with square wheels simply because of management procedures.
The first thing that needs to be communicated is rational expectations.
Or even before that, to form rational expectations.
But hiring a small company to deliver Obama care is fraught with political risk, as there’s a very clear “your fault” there, which is easily shifted with a large company on the company.
This point, which is true across government at all levels, can’t be underestimated. I’ve been both a Fed employee and Contractor and well remember initiatives to “think outside the box”, “innovate”, etc. Never happened.
In Government, risk taking that succeeds is rewarded with indifference but risk taking that fails is punished.
NASA, the military, probably NSA and many banks and stock exchanges have complicated software systems. All these organizations function reasonably well supported by reasonably decently functioning software system. Although, most large software systems are complex and take much longer to develop and more money to maintain, we have many success stories.
Healthcare.gov is dire straits. My experience supports an estimate of a time frame of about May/April as the earliest time the system will be decently functioning provided: the team fixing it is: small, excellent, experienced with large systems and unhindered by management.
Admittedly, I am ignorant about the system itself. Yet, I took quite a few colossal failures and made them into functioning entities. Software engineering is correctly portrayed in the post. Yet that picture fits the average case only; there are many successful software efforts. Healthcare.gov is a crucial social and medical system; it should be given everything we have to make it available to the public ASAP.
The post says: “The ominous point is that while policy makers may have ceded control of their institutions to software, it is no longer obvious to me that even the software engineers have control of the software anymore.” I disagree. We know how to control complexity. We have done it time and time again. Even a passing knowledge of the technical people working for Google, Microsoft, Amazon and similar companies indicates that we can succeed making Healthcare.gov a success story.
It is obvious to me that we can have full control of software if we want to.
NASA, the military, probably NSA and many banks and stock exchanges have complicated software systems.
I will bet that: All of those running good IT systems – except the banks, surprisingly often bank systems are really, really, bad – cost significantly more in time and money per functional unit than “ObamaCare v 1.0” did, I guess 10 times more.
The main difference is that “ObamaCare” was put out for tender, the cheapest supplier got the job and even while they know that talent does not scale, they still will try to solve the problem with ridgid top-down requirements and procedures the same way McDonalds mange to make the same burger with the same global pool of dum-dums and casual workers.
Probably, when “ObamaCare” has cost the same per funtional unit as NASA software the system itself will work (which still does not fix the brokenness from relying on all kinds of crap … unvalidated … input data that one does not control, like credit reports).
IOW: “ObamaCare” will be very expensive, bring years of dysfunction and many, many conference papers as to why.
The notion that government contracts go to the lowest bidder is not quite correct. The many, many laws and regulations governing government acquisition require that the lowest bidder who is “proven” capable of doing the job must be chosen. That’s why CGI, the behemoth software conglomerate that is the main project manager, was chosen. Not because they are the most capable, not because their bid was lowest, but because they have absorbed smaller companies which succeeded at particular tasks, thus “proving” that CGI is capable of performing that task.
The government procurement system is probably impossible to correct. It is just too huge and complex, and there are too many people with vested interests. It’s like the War On Drugs — it’s not working, it’s causing great harm to our country, but there are thousands and thousands of people who make their (not very great) living from it.
HOUSTON, WE HAVE A PROBLEM.
Even rocket scientists can’t make things work right all of the time. The time it took NASA to safely launch a rocket and the toll in dead astronauts, here and in Russia, shows how complex things are and how much we try to deny complexity. Some the most prominent and forgotten deaths, aside from the test pilots who were slated for space flight, are the 3 dead astronauts from the Apollo 1 training exercise. Then there was the now famous Apollo 12, made into the movie where failure is not an option. They were lucky to make it back alive. Even with decades of space flight experience, 2 space shuttles were destroyed in flight with their entire crews. The introduction of new advanced technology is complex, takes more time than people want to admit to, and when all is said and done, we can’t escape our humanity or our mortality or the overpowering forces of Mother Nature. The persistence of vision must endure until manifestation.
https://en.wikipedia.org/wiki/NASA_accidents
Middle Seaman,
If you thought I was putting down software engineers or saying we couldn’t make excellent software, then I was careless, and sorry I give this impression. I am or have been a upper echelon engineer in two of the three companies you list, and my respect for top talent is extremely high. I could not have put this effort in without immense love for what software is and does.
This is the first part in a series, and it only starts with this edge because it is important to set context. Amazing software is being written, but we can and will be doing MUCH better in the future when we overcome the challenges that are highlighted.
The fish stinks from the head
Not the software
Yes, yes, yes. Management is the key–bad managers, bad results at least for gov’t work.
Private sector too,
I’ve worked in Silicon Valley and watched even big and once innovative companies struggle under bad management.
+1 while there are incompetent people in the field, projects usually fail due to bad management. It’s not the lack of the best and the brightest, it’s lack of letting the bright enough actually use their skills in ways that make sense.
I was thinking of the old saying about law and sausages. ACA is one giant grinding of pig, in order to save it, make it bigger. ACA law orders coders to come up with a much larger nasty pig. That can never be pretty. The exchange is entirely too complex almost as much as it is not needed at all.
Look at the UK NHS web site. No questions, no application, no enrollment, no class warfare, I mean tiered bronze, silver, gold plans. No questions of ones credit rating, no deductibles or co-pays! Just how to get you to the health care you need right now.
We are creating an entirely needless borg. I can only hope the code/software never works. I will not be assimilated in this obviously nasty code no matter how pretty the software might be some day. Coders should stand up, walk out the door en masse, rather than talk about themselves, their self-importance at times like this. No code can make this law/code healthy, caring, much less affordable for the always preexisting human condition.
I have been in the computer business since 1961 and have had the privilege of working with some of the best software people on some of the most interesting problems for both government(NASA)and industry. One of my mentors said to me early in my career that “there are a lot of people in this business but only about two percent of us really know what we are doing”. The folks in software development are generally fairly bright but rarely have the opportunity to tackle problems of great scope and depth allowing them to develop the skills required to implement complex functional systems. And the few who do have the skills are rarely in charge of the effort. Having said that, any competent experienced web development group should have had no trouble creating this web site.
Technical problems are solvable. Bad management not so much.
I disagree. I think this was always going to be a clusterf–k. They didn’t even finalize the requirements until 4 months before the site was supposed to go live. Even if they were using an Agile methodology, you still need requirements laid down to interconnect extremely complex legacy systems. And somehow handle reams of bad personal data. IRS, credit bureau income verification, individual insurance company systems. Is it possible to do? Sure. In 4 months? Hell no. This has typical “I don’t need to know software, I’m an MBA” elite blindness written all over it.
Congress wrote a law that required the creation of a technical monstrosity. On second thought, I bet some of them realized it would fail and bet it would benefit them politically.
Jazzbuff,
I think you are missing the point that Healthcare.gov is not just a website like 5-10 years ago, it is trying to be a full featured cloud app, where data from a broad range of sources is used in real time to implement policy. And they tried to build it on 1990s technology. The problem is not about overpaying for some web pages.
Wow this was an annoying article (rule of thumb – if an article mentions the pareto principle, you know it will be bad). From the bad sociology (most programmers were not hippies, and were actually quite conservative. This is still true today), to the weird views about technology (different problems require different solutions. There is no single language/technology to rule them all. Would you build a house using the same materials that you’d use to build a bridge?).
Healthcare.gov failed for reasons that are sadly too common, but it wasn’t a failure of technology. It was a problem of management. The specifications were finished very late. Testing was delayed. The schedule was ridiculously aggressive. The problem it was intended to solve was ridiculously (and needlessly) complicated.
And this is flat out wrong:
“These preferences are mostly forged of pain, not ideology, but the result is that we are probably the least adaptable builders on earth.”
Software is far more maleable and flexible than hardware. It can evolve, it can be adapted. There are techniques for doing this. And people manage it successfully even on the largest of projects. The problem is that the techniques do not fit well with prevailing ideologies of management, and most large IT companies (in particular) are trying to tailorise software. Which really, really, doesn’t work.
The biggest problems that the software industry has are very bad management. Projects are late because estimates are over optimistic, because management don’t want to hear the truth. Projects fail to deliver value, because requirements never emerge (or emerge very late) from committees where politics trump everything else.
I pretty much concur with Cian.
ACM (Association for Computing Machinery) did a survey of large projects back in the 1980s. They found that the majority of them either failed completely or were not used as they had been designed or intended to be used; that is, they were partial failures to which work-arounds were applied. As a ‘software engineer’ since the mid-1960s, this was also my general experience, in both corporate and governmental work.
The problems are only partly that programmers typically work intuitively and often don’t know what they’re doing. The larger problem is that the people who manage them don’t understand the work, its materials, and its consequences at all. This is a common characteristic of large institutions: the top authorities necessarily must deal mostly with political maneuvering and do not have the time, energy, or knowledge to deal with technological and front-line managerial issues. Consequently they have to rely on other, lower-level political manipulators, who are frequently merely good fabulists or con artists. But the reality of politics is fundamentally different from and antagonistic to the reality of technology. You can hype, mislead, and defraud personnel and get more work out of them for awhile, but you can’t succeed by lying to a machine.
Everything that is happening now, like the addition of new personnel (which will actually slow down the development process) is typical of this kind of situation. The rollout of the ACA is, thus far, classical. The same mistakes are made over and over again, whether the environment is private, corporate, or governmental. It’s the way we do things.
Cian speaks for my experience, a former SW engineer from the late sixties (started in high school). I agree, this is a poor essay.
I think it’s hit and miss, it hits a few points I relate to (the difficulties with legacy systems and the unwillingness to throw out legacy stuff that kinda works and thus the hardening of software Not that you could in this case anyway, it mostly being the partners you were communicating with), intuitive knowledge forged by pain, the ad-hoc nature of what keeps most systems actually running (it’s ugly if you look behind that curtain).
But I agree projects almost always fail due to bad management, unclear requirements (without the ability to easily adapt), impossible demands etc., and the poor employees are left scrambling trying to do the impossible, because management says it must be done and by last tuesday.
“From the bad sociology (most programmers were not hippies, and were actually quite conservative. This is still true today”
wow, maybe that was the case at Brigham Young, or maybe you just hung out w/ the exchange students?
In my albeit unscientific observation of the “older guys” doing heroic programming work w/ the likes of PLATO https://en.wikipedia.org/wiki/PLATO_(computer_system)
at the UofI C/U, it was all in for reefers & beer social hour to watch startreck reruns wearing the likes of “keep on truck’in” teeshirts .
Perhaps regional impressions?
“Information wants to be free” is a prototypical hacker statement. Not exactly a conservative value. Maybe the clockpunchers are politically conservative. The influential, social/technical leaders were all techno-utopians or soft libtertarians. Nowadays who knows — seems like a fair number are sellouts to the NSA.
Maybe NSA is so secret because all their programmers are enjoying frequent reefer and beer parties? I never worked in the industry, but from reading magazine articles it seemed like Edsger Dijkstra was pretty conservative, while Niklaus Wirth seemed pretty laid back. I really don’t know about Donald Knuth, but he sure was a good writer and so was Brian Kernighan. This article would have been a lot better with fewer metaphors.
Here’s the thing. Most programming didn’t (and still doesn’t) occur in universities. It occurred in places such as IBM (notorious for being conservative), Hewlet Packard (great engineers, who weren’t hippies), Microsoft (seriously, not many hippies), DEC (blah, blah) Oracle (blah, blah), banks (really not many hippies), insurance companies (pretty intolerate to hippies), etc, etc.
Cian,
you do accurately state the conventional wisdom, and a wisdom that I followed most of my career. But let me start by saying I was and am a very conservative hippy. I think it is fine in a piece to make light hearted pokes at the cast.
Regarding your point about the agility of software. With tens of billions spent on data centers, the average age of hardware is less than 2 years, and the average age of software is at least 3 times longer. There is no hardware over 4 years, and lots of software over 10 years old. Which is agile? This is a critical myth we need to reverse.
There is also the persistent myth of bad management, but this is not really useful, because bad management exists in every industry but the challenges highlighted here are unique. There are many successful industries that have learned to make software work with large amounts of investments, but you find that at the good ones that the management is staying out of the way as much as possible to allow complexity to sort itself out organically.
My goal here is to move our eyes past some of the older problems that we may still be tackling, and look a-new. The big dream of software was more wonderful and more awful than we could have imagined. Seek truth, and from there improve the world,
Although I disagree with Bob Goodwin, I find it unacceptable to fault his thinking or his intentions. Bob represents quite a popular perspective of software. That is where a comment is in order.
The human body and medicine represent huge complexity about which we sadly have only limited knowledge. (See all the diseases we cannot cure.) Software is different. Software is way less complex, but more importantly, we cannot change the body while we can change software. Bob is right, most designs are terrible. They don’t have to be.
That is our key to success.
although I’d broadly agree with you (designs are terrible etc.), I’d say that the reason for that is what Cian says – politics.
And, although I didn’t spell it out in my previous comment, the reason for that is exactly because SW is so malleable. If you were building a bridge no-one would even think of changing the requirements every couple of month all the way to its opening. Everyone understands that you have to pick a location and it’s hard to change it once you put the pylons in. That when you decide to build a suspension bridge, redesigning it into arched one is not a good idea. Software, on the other hand, looks like somethign where it’s easy to add/remove/change things on fly, so it happens.
Incidentally, NASA can controll its (even very large) SW projects because they can pin down the requirements quickly and easily. There’s little human interaction required (which tends to bring in the problems).
That’s, IMO, one of the reasons why you need small teams to build SW – you then see that the resources is more finite, and you focus (and change less, and think about it first). If you feel you cn just throw more programmers on the problem to get it solved (a common approach), you get bad solutions very quickly. I’d rather pay 2 exceptional programmers 200k each than a team of 10 average people 40k each – I’d be much more comfortable with the delivery and quality.
All I will say is that it is sad; sad and damn annoying to have such a pontification by Mr Goodwin about the history of software floating past my eyes.
I couldn’t read the article after the first few paragraphs. It made absolutely no sense and seemed to be based on a completely fictional view of reality.
“A ship on the beach is a lighthouse to the sea.” – Dutch Proverb
Frederick Brooks’ The Mythical Man-Month: Essays on Software Engineering [first published in 1975], while rarely heeded, still pays dividends today.
“Brooks’ Law” argued that when a software project is in trouble, throwing people at it makes things worse, not better; adding people to a software project that is late makes it later.
Project developers tendency to repeat such errors led Brooks to quip that his book is called “The Bible of Software Engineering,” because “everybody quotes it, some people read it, and a few people go by it.”
The tarpit that he described plagues Healthcare.gov today:
“Large system programming has over the past decade been such a tar pit, and many great and powerful beasts have thrashed violently in it…Large and small, massive or wiry, team after team has become entangled in the tar…Everyone seems to have been surprised by the stickiness of the problem, and it is hard to discern the nature of it…This book is a belated answer to Tom Watson’s probing question as to why programming is hard to manage.”
It is a helpful guide to understanding its failure.
“Systems program building is an entropy-decreasing process, hence inherently metastable. Program maintenance is an entropy-increasing process, and even its most skillful execution only delays the subsidence of the system into unfixable obsolescence.” – Frederick P. Brooks Jr., The Mythical Man-Month: Essays on Software Engineering
Let me throw out this comment to those who, not in this posting but else where, say something along the lines of “Well, Google / Facebook / XYZ built a large system, it just goes to show that the gov’t can’t.”. Google et. al. did NOT build a large system. They built a relatively small one, and their users / customers / marks did most of the valuable work of loading and linking the data.
Also, private corporations can bury their corpses privately. A very large brokerage house I shall skip naming spent a rumored half billion dollars over several years on a system to provide information to their customer service representatives; this was in the mid-90s. Politically, it was based on the CEO playing golf with some Microsoft prince of darkness. Generally speaking, it failed and was superseded before it was delivered by a much simpler web-based system run up by maybe a dozen people. Unless you worked for this company, you never heard about it. A decree came down one day from on high, heads were removed, contractors and underlings shown the door (or the plank), and personnel with connections reassigned. The huge project faded away except for some lingering war stories. You can’t get away with that in government.
Hehheh – I have personally seen two large SAP-projects, consuming millions of EUR in direct costs and in loss of talent and productivity, tank. Both were forced through with “the full support of the board” – right up until they did not have it.
The last one they had hired-in security people carrying bin-liners and escorting the entire SAP project team with management to rows of taxis waiting outside – at the same time as an all hands “Big Boss Meeting” were taking place right next to the exit path. My boss even had some cognac and cigars stashed in his desk for the occasion ;-)
This never makes the papers.
As I recall, MS did quite well selling their failures too, and then making everyone buy repeated revisions until around version 5.0 or so the program actually worked decently.
I have one big client who, from his long experience as an HTML jockey, also thinks that a UI prototype can and should be fit to deploy seamlessly into production. In fact, there’s far more to making a reliable distributed system at global-utility scale than ghosting ten million PCs with a toy version of the system.
You go back to the 90s – most of the software being produced was the type that intermediated transactions – from my view – it was the introduction of new middlemen even when it was marketed as removing the middlemen to top sell to executives. During that time as now, it was about patenting a transaction method or a battle of standards – deliberate standard tweeking to knock the other down….it was entropy in full swing. Software is mostly communication – the only usefull software is the type that controls machines or aides in translating spacial relationships, aids in diagnostics. What real use high speed computers that transfer ownership of assets a thousand times per second like HFT on wall street and skim off productive capital from productive use.
Don’t get me wrong, I appreciate the influence rapid communications has on culture, on debates and the disemination of information. But the dark side are the huristics that anticipate what the information seeker is looking for – I believe this has contributed greatly to the polarization of views and debate. We stuff huge quantities of information into data pools without any thought of the quality or toxicity of that info and data mine some of it out like taking a cup of water from the oceans in order to sell to some fool who does not know that drinking salt water is potentially deadly….. a pick your poison scenario….a way for the cup dipper to cash in personally at the detriment of another…. a way to turn away from scientific discovery and toward opinion debate despite the evidence (see global warming). Vapor ware. All of it is from the printing press, from the etchings on stone and the paintings on caves. My own filter is this; does it advance scientific discovery, does it raise the consciousness of the human mind, does it produce new materials, diagnostics, raise health standards, living standards (the ACA and its software do not) or raise real world efficiencies or does it just serve to drown it’s users in an ocean of data and support entropy.
I know I am rambling on….sorry
“The Pareto principle, aka the 80/20 rule” ?
My view is that all the advancements of our software/programming are on the 20% side….the other 80% is pure crap. My cynical side (a very small part of me) thinks the relationship is 95/5…..95% useless crap in my filter of things.
This is why I think Google is one of the most dangerous companies on earth. They want to tailor your entire online experience to what they think you would want to look at. So we each get trapped in our own ideological bubble and lose access to the vast and wonderful ability to connect with random people all over the globe. Their takeover of YouTube and integration with Google+ is terrible for anyone who is curious about the world outside of their own belief system.
“Something new is happening. Software engineers have happily disrupted the world for 30 years. But now we are being disrupted by big data.”
Memory used to be an expensive limitation, with space to store data at a premium, data bases were very carefully, designed.
I’d describe these carefully designed, relational databases as elegant.
The way I see it, the dawn of cheap storage, combined with a highly concerted effort to hold down the cost of engineering labor, meant being able to use less expensive programers building ugly flat databases because the cost of storage was no longer a constraint, and the government allowing the importation of low-cost talent via H1B visas.
At this point, the bigger is better crowd shouted the rest of us down, if data was your friend, if data was good, then the more the better.
Why not have all the data?
You can see this thinking at work in the NSA surveillance systems; they want it all.
It doesn’t make sense, but that’s the big boys think now days.
When it comes to business, it used to be that a business was satisfied to manage it’s own data; that being who it’s customers were, who owed them money, who did they owe, what did they sell and how much of it was in inventory?
Now days, marketing vendors are promising that through the magic of big data, they can deliver not only your own customers, the ones in your particular database, but all the customers in all the databases related to your product.
They make this promise based on the fact that they talk you and all their other customers, your competition included, into giving them complete access to their data.
Instead of becoming good at understanding, at mining their own data, short-sighted businessmen are giving it away, (often contrary to their privacy policies) and then paying to get it back.
The story in short is that the constraints of expensive infrastructure and engineering led to elegant solutions and systems well suited for their purpose.
This produced a short-lived period during which ‘things worked’.
That’s all over now; destroyed by the inability of managers to resist the illusory promise of ‘Big Data’ hucksters.
It’s not that things can’t work, it’s that the engineers in the IT industry are no longer as influential as they were, the industry is now driven by marketeers who earn their money by making promises, often unrealistic and/or stupid promises.
The problem is; everybody is buying.
I agree that data bases are elegant pieces of software, they isolated complexity, and had well behaved interfaces. But they were also designed to manage limitations of hardware of a different age. There are still plenty of good reasons to use data bases, but it is the engineers who are driving alternatives, not marketers. We now have new requirements on data within a cloud, and databases were built to solve an older problem.
“unhindered by management” – Dream on. These are politicians managing it. They regard IT people as menials. They want to dictate reality, change it weekly if not daily, and have no idea why it isn’t possible and don’t want to hear otherwise. They will leave it unhindered when the serve peppermint ice cream in hell.
That reminds me of almost all corporate management I’ve ever seen.
This is exactly the attitude, right here. And don’t get me started on former HTML jockeys who think that it is right, proper and expected to ship a prototype of a data management system…
This is a thoughtful post on the field in general but doesn’t quite apply to the Obamacare problem. Goodwin acknowledges that gov’t work is unique. It is very unique in that you have to please stakeholders with very different interests, requirements and cultures. You are dealing with 1) political people; 2) senior career bureaucrats; 3) bureaucrats that directly manage the project; 4) marketing people in the consulting company; 5) consulting managers who consider the company’s future, political matters and so on; 6) project manager(s); 7) engineers/programmers. Also, in my experience, each of those groups don’t trust the other so various systems of high-maintenance management tool are brought in which, with careful and artistic use can make things work but often, just create a high overhead and annoy people.
This particular software failure came as no surprise to me. I’ve seen a number of successes–but in my experience most projects under-perform because of management failures. It takes gifted PMs to pull off a success and MBAs even when they are programmers often lack the ability to handle the conflicting currents–but a few PMs have the ability to get most of those stakeholders on the same page and constantly communicating–I’ve had the good fortune to see the best at work so I know it can be done–but they are the exception.
couldnt agree more, US.GOV gets the healthcare industry into electronic records and now websurance and the whole thing is tied together with chrome/ie/firefox.
The browsers complete for security, not communications interfacing and standardization beyond HTML 2.0. The US.GOV should make their own browser or plugin and standardize all electronic medical records communications to use a single source for functions and connections for troubleshooting and deployment.
HP Dell and the rest on top of the glaring lack of platform also throw on to security profile junkware that further inhibits basic functionality that makes deploying workstations and troubleshooting AVG junkware a waste of standardization.
Goverment made highways, its been 15 years and they cant figure out networks for the private sector.
Mandates arent all they are cracked up to be, building code, standardization, and necesity are not something you get off the cuff every 6 months from google and microsoft.
Well said…with the enthusiasm for the Internet & the World Wide Web, most people outside the industry don’t realize that the core web architecture is surprisingly inflexible, and thus fragile.
Prior to the mid-90’s, software development focused on software that would be run locally, either on the computer someone was sitting at (like a PC), or on a dedicated server or mainframe. With the emergence of the WWW, the architecture changed. The only thing running locally is a web browser, essentially a formatted text display program, with minimal processing power of its own. Instead, most of the software is running remotely, on a server that not only does the processing, but also formats it to be loaded on your local web browser. And the server you’re using is entirely arbitrary. Also, in a distributed architecture, the data, in order to be arbitrarily available, has to be “dumbed down” to flat, semi-formatted text (such as XML), instead of the more logical structures of relational databases.
The distributed approach has advantages, but from a software development perspective, the programmer has to design and code primarily to this architecture, and to the each specific browser, then secondarily to the business rules (which are difficult enough to pin down). As the author and others have mentioned, the timing of the emergence of this architecture roughly corresponds with the beginning of the industry-wide struggle to keep the IT infrastructure that powers global civilization up and running.
The WWW was in the right place at the right time, in the mid-90’s, hence its success, but it has left software engineering with a difficult legacy, as shown with the healthcare.gov debacle. Alternatives, like Java or Ajax apps, or insecure or take too long to write.
Throw this in with the ongoing NSA scandal has exposed that, in their effort to vacuum up everything, they have “broken” a lot of the web infrastructure, such as browser encryption.
Now is the time to create a new Internet infrastructure, to make it more secure, to help protect privacy, and also to make it easier to develop for.
A lot of energy focusing on the new untested boiler that blewup in the enginroom, but suppose that it worked and the ship was launched flawlessly? Isnt is the case that the cargo that is supposed to be a healthcare solution for the masses is actually just a boatload of crap?
What happens when the website actually does sputter to life and starts spraying the public w/ bronze colored disappointment?
As concerns the ObamaCare issue, your’s is the only comment that matters.
ObamaCare is not a story about technological failure.
Yea the rest is just software people complaining about the problems in their industry and why (their problems doing thier jobs basically they know all those troubles all too well). But that’s not what Obamacare itself is. It may depend on software but it’s not itself software.
I understand your point, but don’t agree.
Obamacare is a story about elite incompetence. That is, it’s a story about how today the U.S. has an elite, supposedly providing governance, which is in reality narcissistically dysfunctional at anything but looting for itself and its masters — and, indeed, ineffective at that, given that the ACA was a preemptive bailout of the U.S. health-insurance industry, otherwise projected to break around 2020.
So there’s a pathological dysfunctionality here we need to comprehend in detail, including how it played out in the context of this very large IT project.
Seriously, the Eisenhower administration oversaw the building of the interstate freeway system, nuclear reactors and aircraft and submarines, and the first rockets — which in turn provide the impetus for the development of the microprocessor (in 1960, 100 percent of all integrated circuits made in the U.S. were bought by the DoD).
The Obama administration can’t even build a website.
I disagree, there is no pathological disfunction involved, the system is functioning perfectly from the perspective of the health insurance industry.
Pathological may be a good word to describe what’s happening, but disfunction, no.
We’re ‘led’ by greedy psychopaths, and they are working from a plan.
That’s the pathological dysfunction I mean.
So my point stands. Because it’s now reached the point where the ACA scam can’t even function as intended to deliver the results promised to Obama’s co-culprits in the US medical-industrial complex.
Having worked as an IT contractor on both governmental and private projects and on a HIPAA implementation, much of this post is dead on target. However, the difference between government and private is not as much the problem as the difference between inside work and contract work. And it goes to the transaction costs and miscommunications at the boundary between client and contractor and contractor and subcontractor. Actually it goes to the politics of those relationships. In inside communications, there is more allowability for pushback and a political hierarchy of appeal, depending on the size of the project. With contracts, the customer (meaning the contracting official who is the key to repeat business) is always right and there is no appeal to their supervisors under usual circumstances.
With healthcare.gov, you had the added complexity of negotiating with private insurance companies over which you had no direct control at all. And each and every one of those private corporations (and there are likely hundreds involved) could appeal to a Member of Congress, a number of whom were not invested in the success of the project.
And each and every one of those private insurance companies had to complete their own inside work on time for the entire system to work.
Something like this, so new, so complex, was never going to work on the first try.
Unfortunately, advertising, PR and the media have removed all the sweat and struggle from our narratives (except personality conflicts), so no one knows how to deal with it when it appears anymore.
Quick there must be a personality conflict at work!
I see truth in every comment made on this page so far, and that’s with well over 50,000 hours of software engineering under my belt.
Just a small tidbit, but in my experience what separates the elegant and long-lived software systems from the ugly failures is the motivation of the team that creates the initial implementation. Teams that expect to live with their creation for years will, if they have the talent, create great software. Teams that don’t have that feeling of ownership and responsibility won’t invest enough energy to do it right, even if they could.
There are many, many programmers–especially working for large contractors–who feel a nice sense of accomplishment after an eight hour day of effort. What you want, if you care about the result, are programmers who only feel accomplishment when the vast mechanism they are building actually is running smoothly. Most cannot defer gratification long enough.
Healthcare.gov was a success, for the people who were paid to create it. They put in the effort and they got the money. If we, the customer wanted more–like a functioning system–we should have arranged that the contractors had the same incentive.
“Just a small tidbit, but in my experience what separates the elegant and long-lived software systems from the ugly failures is the motivation of the team that creates the initial implementation. Teams that expect to live with their creation for years will, if they have the talent, create great software. Teams that don’t have that feeling of ownership and responsibility won’t invest enough energy to do it right, even if they could.”
and that gets back to management, many times as employee I have felt I cared more about if something actually suceeds than either immediate management or the uppity ups whose goal sometimes seemed to be assigning us to as many distracting projects as possible. That’s demoralizing. Am I the only one that cares about doing a good job? Why again?
Hmm. In my view (agreeing with a poster above), this project flailed for the same reason many major software implementations flail – just more so – because it was bigger, more complicated, more poorly planned, more hindered by politics, and more over-sold than perhaps any project before it.
However, I disagree that this is limited to the public sector. In fact, after thirty years of watching, I have yet to see a major project in the private sector that did not flail out of the gate, ultimately require reworking and work-arounds, and always, always, fail to deliver on its initial promise. And I rarely see private sector people fired for it either. Software is viewed just like the crazy aunt who lives in your attic and comes down at every meal to torment you.
The software companies are not without blame. Their sales people are notorious for overpromising, and lying. Vaporware is rampant, product bug cover-ups, support staff turnover, and for promising, – above all- that the client can still keep their [insert sacred cow ridiculous legacy workflow requirement]. I once had a CEO suggest that all sales presentations ought to be videotaped and included in the contract.
It is not limited to software either. Think: Osprey aircraft boondoggle, or the Big Dig. [I read an essay a while back that argued to be forgiving of the big dig, that things have to happen this way. It’s like giving birth I guess: messy, dangerous difficult, time consuming. That’s why the government has to be the one to do it]
That said, it is my experience that management people leave their heads behind when they take on software implementations. They do a lot of magical thinking. They defer, even genuflect, to software firms like they are brain surgeons: inscrutable, masterful, infallible. They leave behind the basic project and contractor management skills they use of they were hiring a firm to build a building, for example. They think that software can overcome bad planning, rube goldberg workflows, human/organizational dysfunction, and leadership gaps.
RE: Software Engineering.
A really good and informative post. The most important extract is: “The existing infrastructure is almost certainly inadequate …”
Try to get a web site with an AVERAGE connection. It probably will work, but sometimes will not, and for completely unknown reasons. The system was created by people who believe implicitly in the theology of “free enterprise” (more fast bucks Bubba!..) and of course the result is often completely nonsensical, as well as very shaky otherwise.
Conceited (and ignorant) MBAs believe that language can be read and understood, by simple programs playing “twenty questions” with data, and are rudely surprised when that only leads to massive and far worse confusion. That way, they get a big bonus from firing the sensible female clerical types (very often), who would be institutionalized if they made the same mistake.
Result: My credit report (from the same Experian oracles as starring in the present crisis) had me living in Center city Philadelphia (known and familiar, but never lived in), and a place called Centreville (nineteenth century spelling deliberate, but completely unknown … had to look it up), Illinois), both at the same time!
Result2: It took six months of emails and calls to try to cancel an extra credit card issued by mistake (and fortunately never used), and ultimately a letter to a responsible person (obviously intelligent and female) to complete the simple process.
Software engineering (programming) can be quite difficult, but nothing is as trying as the attempt to work with the jumped-up, conceited graduates of “Bufoon Business Academy”, who in the vernacular “could F** up a box lunch”.
The reason the credit reporting system is broken is because there is a financial incentive for it to be so, and no incentive for it to be too accurate or responsive to citizens.
The banks are the people who pay them, and they would like too much information that is a litle too conservative if anything, rather than too little information.
In the meantime, so what if your credit rating is wrong? That’s your puny problem.
Think about it, they run a faulty system that holds your financial life in its hands, and there is no recourse for you.
But, FOR A FEE, they will sell you an extra product to protect you FROM THEIR MISTAKES.
The tagline for this article: “Software Engineering in Crisis” does not really deal with the issues of the failure of Healthcare.gov. A better tagline would have been: “Large Project Management In Crisis”.
Software projects, as well as other large, complicated, projects have been failing for a long time. The problem is always the same—BAD MANAGEMENT.
There is a must read book that outlines the issues: “The Mythical Man Month”, written by Frederick Brooks–back in 1976. This book outlines the problems IBM had developing its OS-360 operating systems during the 1960s.
One of the underlying problems is that programmers do not make particularly good managers, and companies do not like to spend money training people. Universities do not understand the real world–and generally do not produce “shovel ready” employees when they graduate their students.
So, in the long run–it’s the “Blind leading the Blind”.
There is an old saying: “Projects don’t go bad overnight–they go bad one day at a time.” Fix the management problem, and you’ll go a long way towards fixing the software problem.
Indeed. Had anyone followed the management lessons in the Mythical Man Month (circa 1975), the ACA rollout would have gone off well.
MMM is the bottom up bible, and it is hard to overstate the influence. But 40 years later, as you rightly note, non practitioners are falling behind.
This from an Old Old programmer , systems engineer, manager and Director.
You see the problem is that there are “people” involved in this deal.
I will offer as an example a customer who used software we designed distributed and supported. There was a method to be followed when closing out a day that should be followed. This lady just used the on/off switch at end of day and lost data. I explained the proper method and results, her response was, “I am going to just turn it off”.
People!
Tides of Empire
The American middle class is about to be awoken, and it’s not going to like what it sees, having worked so hard to float downstream with the empire. Labor is migrating to a 24 hr workweek, by eliminating compliance entitlements, whether anyone likes it or not, because capital’s demographic ponzi has passed top-dead-center, globally. Look at the data.
America as an empire is a scam, as are all empires. Capital presents the economic lifecycle as labor, middle class, capital, creating the perception of upward middle class mobility toward capital, the rising tide, with a rigged lottery compliance ponzi. Now that the ponzi is collapsing, middle class participants are falling, competing to be last to fall, hoping for another iteration, which is not coming, which is why you are looking at Nazi outcomes.
The cycle, beginning to end, is labor, capital, and middle class, which repeats with the so-distilled kernel. The capital education propaganda system simply shifts the point of origin to capital and employs the credit debt system to run the media presented economy backwards, delaying only the outcome perception.
Normally, capital would reverse Family Law polarity at this pint, to favor males again, and continue priming the ponzi with new shills imported for the purpose. The problem/solution is globalization.
Accept a lower standard of living because the rest of humanity suffers from a lower standard, equality under global law, and bow to government for privilege, on the way back to the DNA churn pool, NOT. China is not the future. That empire had its turn.
America was the rabbit, and the dogs caught it. A little common sense folks…capital isn’t formed in an office building with words on a piece of paper. The beginning, middle and end of capital’s plan is greed, which works until it doesn’t, depending upon how much gravity you want at TDC.
Capital is a derivative of labor. It’s not about the job and it’s not about the property. It’s about people, and you cannot change human herd behavior in real time, which is why capital is always seeking another shell to hide under. Don’t expect Silicon Valley to change course as the cliff erodes to its position. The peak exposes the reef in the valley. That’s why all those ships are down there.
Accounting paper profit on artificial asset inflation, a dance between Wall Street and the Fed, is not a business. Corporations are just pieces of paper with bankruptcy preference over people in a ponzi that is always bankrupt. Summers and Yellen wouldn’t have jobs if the critters didn’t want to use their house as an ATM machine to get something for nothing, sunk purchasing power.
You only need a constitution and money to grow a middle class beyond natural proportion. Labor doesn’t suffer from such a lack of constitutional character. A paper contract is not worth the paper it is printed on. Time, not money or property, is the measure of human endeavor.
You may want to adjust to the new workweek before the lights go out. Discount your RE values, the distillation gradient slope of Police State Power, beginning with the natural course of water, to close out the empire developers. Sometimes, you start over, as an investment in your children, to show them the process.
The Internet is just the Xth derivative of the global communication system, implementation of QEInfinity, reaching for the limit of diminishing returns. Opensource has been training kids since before Kissinger’s Monsanto Crop of Bay Area programmers were born. Let’s see who ends this war. Communication doesn’t begin with a handshake by accident; watch out for the current, The Imperative.
It always floods after the drought. Wire around it beforehand, so you have lights while you work. Run a temporary to the new motor, add features by floor, and put on the dress. It’s always the same process, and it helps to have a new motor, with new forward gears, ready to go.
There is always an economy being built and an economy being dismantled. Dismantling is a job. Building is a lifetime avocation. They run in opposite directions. The empire teaches dc. Find an old-timer with patience if you want to learn ac.
Remember driver’s ed, how the critters looking in the rear view mirror always ran the red light? Farm kids learn to drive when they are 5. Let the critters in a hurry pass you by; they are all going to the VIX casino, to be shorted by another short squeeze until they are slaughtered.
Don’t waste your time trying to explain the kernel. It’s About Time. Render unto Caesar, and get on with your own life, minus the artificial anxiety.
“There is always an economy being built and an economy being dismantled. Dismantling is a job. Building is a lifetime avocation. They run in opposite directions.”
Beautifully said. Poetically concise. Thank you.
I believe this is missing the point. There are two important issues that are obscured by the observations above, no matter how relevant:
1. The Right To Personal Computation and Storage. Even an encrypted cloud, while it might or might not hamper inspection and surveillance (and modification) removes the ability to control storage or protect it from deletion or access denial. In a few decades of current trends extrapolated, the mere desire to own and use processing power not under cloud control will be cause for suspicion,
2. The Replacement of Documented Policy by Opaque, Incomplete, Faulty and Compromised Software. Open societies require explicit and accessible definitions of policy for debate, accountability, judicial review. We are rapidly escalating towards badly written software implementations provided by private contractors hidden behind national security or corporate trade secret/IP protections used for algorithmic interpretation application of “expert system” codifications of the law – running on infrastructure compromised by “intelligence” backdoors and engineering flaws that cannot be discussed due to national security concerns being used to protect those responsible in government and corporation. This same dynamics also leads to responsibility diffusion and obfuscation, and to an end of accountability and liability for a new category of “procedural” errors – imagine the extreme case of “autonomous” law enforcement and military equipment (preprogrammed heuristic drones).
By and large, the article describes the same dynamics that affects “hard” engineering disciplines – see NOLA, levee, Army Corps of Engineers. I believe that the two issues above are genuinely specific to the intersection of software, governance, and open societies.
“The software engineer had the tedious job of primarily overcoming obstacles, while trying to work the magic of his trade of extracting value from large investments in equipment. The best in the field resembled master craftsmen who could build rocket ships from tin foil and masking tape, and make it look easy to realize the potential of the computer.”
I don’t think you’ve meet a Software Engineer. You’ve probably met Computer Scientists or Computer Programmers who have the title of Software Engineer and they think that means writing some code and writing some stuff down in a word doc and calling it a Archtiecture/Design/Requirements Document.
You wan’t software built with the same permanence and rigor as a bridge/airplane/building then you have to treat it like such which means a SE is more about Project Management then coding and the fake Software Engineers will start howling in pain.
I think you might have taken that quote out of context. I was talking about the 50’s and 60’s programmer. Programming then was indeed like that.
think this blames the employees entirely too much. Because while I have seen not very good programmers. I have also seen again and again, people who want above anything to do a good job, destroyed by impossible contracdictory demands coming down from above. I have seen good employees talents wasted for bad management too many times to count.
This post is a series, and the blame is never with the employees, managers, contractors, or government employees but rather how we think about software in the context of policy. And by We, I include you and me. It is also about that this is hard stuff, and we haven’t figured it all out.
Is that the same Bob Goodwin who sometimes pops up in the comments as a libertarian defender of the Tea Party? It was pretty funny how he bristled when, during the shutdown debacle, Yves wrote that TPers should be “collared and leashed”. LOL! But it’s good to see that everything’s been patched up between them, that it’s all hunky-dory now. It’s all good. It’s all good. Yves can sometimes be very blunt (in a surprisingly unlady-like manner).But we always come back to her, after some sulking. What can you do?
I confess I haven’t read the whole thing. I started but couldn’t finish. It seems too long-winded and rambling. One has to hope Goodwin doesn’t conduct his company meetings the same way.
As always, I’ve been educated by the comments: vlade, concise and insightful as ever, Banger and many others. There seems to be a fair number of software engineers among NC readers.
I was particularly interested by the discussion on complexity.
Middle Seaman: The human body and medicine represent huge complexity about which we sadly have only limited knowledge. (See all the diseases we cannot cure.) Software is different. Software is way less complex
I agree. Software, as a man-made construction, has orders of magnitude less complexity than the natural world. Its complexity is more comparable to that of other human contraptions such as Law, where complexity seems to come from all the loopholes that have to be added to accomodate our most “productive” citizens.
However computer programs have also been used to mathematically model biological processes. Consider the following thought experiment: let’s say we want to write a computer program that would realistically simulate a human body so that so that we could virtually test some new drugs without endandering any patients. In that hypothetical case, wouldn’t the sofware be as complex as the human body?
The same could be asked from all those climate modelling computer simulations that tell us that global warming will eat us. If they faithfully reproduce the natural reality, shouldn’t they be incredibly complex?
I disagree. All ladies should be like Yves.
I do keep trying to convince her that the Tea Party is the other great anti-corporatist voting block, but she sees them as primarily anti-liberal, which is probably true due to the current party alignment. I also think that degrading metaphors should be avoided in discussing loyal opposition, even when they are really annoying.
But Yves is right anyway.
This is a managementfailure first and foremos, and second a reflection that government solutions are not fast or as flexible as private market solutions.
I’ve seen many, many major systems done entirely in the private sector which had exactly the same problems as the ACA. As I said upwind, corporations can bury the dead quietly and out of sight.
In any case, in the U.S., most of the government computer disasters I’ve heard about were done (or, usually, not done) by private contractors.
When people sit down to design and build computer systems, if they’re any good and they have any luck, they’re out of the realm of ideology.
No, that’s not what the post says.
It basically says that most software engineers take it upon themselves to either overtly or covertly respec projects on an 80/20 basis, since most projects are actually undoable, at least the way the client wanted it done (in terms of performance, features, time to get it done, and budget). They don’t necessarily make the best calls in terms of the client’s interest.
With government gigs, you can’t do that because you have requirements that are a matter of law. That forces a top-down approach.
So the private company programs “fail” as much as the government ones do, in that what was ordered does not get done. But they have the appearance of success because they get reshaped enough that something sorta within hailing of the original spec gets developed, eventually.
Isn’t that the reality of most work, not just software projects?
Things that are reshaped enough that something sorta within hailing of the original spec are successes, for the most part, if they’re useful.
Especially when the original spec wasn’t that great anyway, and was reshaped as a form of compromise that are sorta within hailing of the original vision/goal.
Which was reshaped as a form of compromise that are sorta within hailing of the original problem.
Which has invariably changed, or expanded, or became outdated since the problem first surfaced and were noticed by people.
In my experience the biggest source of failure in IT projects is not technical or managerial problems but in incomplete requirements definition. No one realizes what has been left out of the requirements until they get their hands on the system, and by then it is too late. In the case of healthcare.gov, they were on a ridiculously short timeline and still didn’t have nailed down what they had to do when they began building.
I would bet any amount of money that everyone concerned knew six months ago that the task was impossible, but there was no way the deadline was going to be slipped no matter what. Anyone who’s ever worked on a big IT project knows what that’s like.
Just before I retired I was put on a project that was due in less than a year. I took one look at it and laughed in their faces. There was no way it was going to be done in less than three years. Actually it took five. But the deadline only got slipped in three month increments all along the way. Top management simply refuses to hear bad news. They think if they keep saying Do It or Else that miracles will occur.
top down combined with complex always produces incomplete requirements. It is a neat categorization for shit happens.
The most common example is when expanding legacy systems. This starts with the bottom up assumption of using the legacy system, which often conflicts with the unstated goals of the top down requirement. Its complex, opaque and confusing. So we call it an incomplete requirement.
So… as a lead programmer with 20+ years experience working at major software companies including projects where we built large scale internet marketplaces… the problem with the federal exchange isn’t indicative of the technology.
It’s just a crappy implementation.
Plus, almost all of the problems can probably be traced to program management and technical leadership. Who does a rollout to millions of users without a solid beta test including large scale load testing?
The specifications were poorly designed and thought out. If the idea was to emulate amazon.com, why create an account (expensive transactions) before shopping? No marketplace does this. Plus, they could have had people creating accounts for months instead of Oct 1st. They could have rolled out credit checking before Oct 1st… ect. Obviously, they needed more milestones.
Why do validation in real-time and not batch? I would have tried to make all server to server database transactions between legacy system batch. Send email with the results and let the user know the status.
And finally, as any good PM knows, you have to decide on priorities… Quality, Timeliness, or Features. I would have cut, cut, cut features to focus on the first two… Quality and Oct 1st.
There are so many bad software project, the Federal Exchange is just one example. People can create good ones… Even government.
P.S. Most of what I pointed out were architectural based on specifications. You can’t fix these easily… they’re not bug fixes. They’re architecture changes (which require beta and load testing).
P.P.S. If I was in charge, I would consider maybe going to CA or WA (or some state that was working) and maybe investigating which is harder; Fixing the fed exchange or modifying/cloning a working state exchange for each state on the fed exchange.
We agree on all points, I am placing blame on the software discipline, you are breaking blame to stakeholders. you are making a distinction between a bug and a bad architecture. I am not.
Fair enough.
I don’t think the industry is crisis… this is just a bad project. Heck they might even have great programmers. It’s just a badly managed project.
“software engineer”
PE? Any kind of accreditation or responsibility after the fact? Insurance?
Calling them “software engineers” is another way that their employers can stick them with overtime and not pay them for it. “exempt employees”
“But, I’m an engineer!” while typing away, unpaid, at 2am.
Hearing those two words together drives me insane. A software engineer does not exist. HR likes to think they do, and the execs like HR to think that.
About ‘software engineer’ — in the 1970 academic institutions came to realize that a lot of money was being made by a variegated, one might say, motley group of people called ‘programmers’. In the office I first worked in, half of them were women. Because, you know, programmers write stuff and type stuff and can learn and use language — very femmie. But big money was supposed to go to guys, and the academic institutions needed a slot for their programmer-creating mechanism. This turned out to be in the engineering department, because, you know, machines and everything. And big money. At the time, 95% of engineering school students were male. So, after a while you had to have a degree in ‘computer science’ to get a job as a ‘software engineer’ (regardless of whether you could write working code). And a few years later ACM was having conferences about why there were so few women in ‘software engineering’. Anyway, that’s where ‘software engineer’ comes from.
This is probably a stoopid question.
But….could we just leave all of the engineers alone and let them work their magic, and in the meantime, just use the U.S. Mail to shop for our Obamacare Plans?
And then just go back to using the internet when the engineers get done doing whatever it is that they do? Whenever that is?
I would rather have them do it right than fast. And, while they are at it, I hope they approach it from the standpoint that eventually we probably aren’t going to want health insurance companies involved. (So, I hope they make the infrastructure in such a way that easily allows for us to remove the toll booths that are the insurance companies.)
Okay. You smart people can start talking to each other again. I’ll go stumble my way on to an easier blog for the rest of the night. Bye! :-)
What a great idea. Get a form. Lick your pencil. Fill out the form. Post it snail mail. Let some one at the other end do their quiet competent magic just has they have done for Medicare and Social Security for decades on end in times past. Problems? Make an appointment and meet a government agent trained in ACA who will resolve your issue right then and there a solid majority of the time.
Such is not Obama’s method of operation. For one thing, that would give people jobs and Obama doesn’t create jobs, he only talks about creating them.. His actions always TAKE JOBS AWAY. Then, for another thing, Obama, like everyone else in business, is in love with the idea of making the herd dependent on computers for every move they make which greatly facilitates the ability of government and big business to record said data, to analyze it, and to exploit it, far into the future, at the expense of the fuc*ing retardz who entered it. Finally, under those circumstances, if people could successfully and easily sign up in large numbers for “afordable” (ha, ha, ha, ha, ha, ha ha ha) healthcare insurance, they would almost immediately see the ACA for the incredible crock of crap, financially bankrupting SCAM that it is.
How are you going to compare plans by snail mail? Michael Olenick in Florida has 132 plans for which he qualifies.
If one was actually going to implement an old fashioned form and bureaucracy based sign-up system for ACA, I suppose there would have to be give and take in both the nature of the product and the steps to sign up for it. If we did not have computers, or if we were still capable of thinking that way, or if big business (government included in that) didn’t insist on our doing all the leg work for them to profile us in 256 ways per nano-second, then the solution to your question would probably be clearer.
But keeping the whole business friendly and utterly too complicated comparable plans for the sake of argument, I suppose a non computerized sign up procedure would have to involve multiple steps. You would sign up and send in your qualifying data using one form. The competent friendly bureaucratic dispenser of magic would then send you back some list of comparable plans for which you more or less qualified and you would then fill out another form in which you selected, perhaps, your top three preferences and then send that in and so on. You would be generating a lot of employment if nothing else.
Ultimately, you might have to meet with a government agent (or private contractor) at least once to finalize your application and you would then be registered. Now, you would still have to go over to the private insurance company you finally selected, bringing with you your bottle of Crisco in one hand and a suit case full of cash in the other, but on the bright side of things, your personal data and all manner of heuristic analysis (infuriatingly accurate in places and inaccurate in others) probably wouldn’t be up for sale to anyone in big business that was interested.
I wonder if this is part of the reason for the move to an “online” exchange- Not giving any hard copy policy to the insured.
I also bet that they can change the terms, at their whim, without notification, on their website, buried in page 345 of the site user agreement.
1 page mortgage? How about a one page policy.
I have a few banks really trying to get me to enroll in “online banking”. I can only assume they are pushing it because it is good for them, on top of the paper statement savings (a few cents). What are they able to get away with now with an “online banking account” that they weren’t able to do before?
The push for “online banking” reminds me of the torrent of mail and notifactions about “opting in” to overdraft robbery via a debit card.
If anything, the marketing push for “enroll in online banking” is even bigger. They just won’t take no for an answer.
That’s a nasty twist of thought, Bob. What if insurance ended up being like a software license?
I have no idea what the author experienced in his career to arrive at his conclusions; if they are valid, I weep for what has happened to the software systems industry. My experience was in the aerospace industry on large military systems, 1958-1961, 1975-1995. All the people I knew, some well known at the time, were dedicated and skillful in all they did to make the systems work. By 1995, I was responsible for the ground system software of a prestigious and worldwide known software system; and it worked without problems. Recently, I heard there are problems with that system but that is probably because thoroughness is no longer valued as much as minimalism and completing satisfactorily on schedule, hoping that all will end up well. The current health care system concept is simple, in my opinion; even if there are variables, the straightfoward approach could have worked. In retrospect, I suspect developers are revisiting state systems that are experiencing problems, and making fixes. Software cannot be built correctly if logic is warped, design is illogical, protyping of the user interface is not accomplished, and all aspects of the system have not been thoroughly mapped out so that testing runs smoorthly and the resultant product works without hitch.
I say in my post that most software works. It does. Your examples were closed end programs, which do not suffer from the problems discussed. The post may be glib in style, as to make the point of the anarchy of rapid change. Each challenge has had a solution, but technology keeps changing. No blame, just discussion. But the representations about hard this stuff is, and the problems that happen are real. And that is why the engineers are getting more, and not less, rigid.
Mr. Goodwin’s post seems rather broad for the issue at hand. He assumes good intentions overall but a difficult period in technology due to the confluence of broad historical trends in software, hardware and developers.
The problem with ACA is more likely to be one of simple corruption in the bidding/selection process for contractors and a general “yes sir, you’re the greatest” attitude in the Obama administration itself than any far reaching pivotal moment in computer science and big data.
If the Commander-In-Chief says it’s going to happen, then by DC, it’s going to happen so everything will be fine; nod vigorously while sinking..
On top of that, I really think Obama feels ACA sign-up is beneath him. If it was ever a priority at all, it was a priority like putting his shoes out at night to appear presto, brilliantly shined in the morning. Indeed, it is not obvious he was ever all that interested in Health Care beyond demonstrating his enthusiasm for, even more than his fidelity to, the corporate elite and their goals. But that is done. The betrayal is over, done. Forcing citizens of a so called “free country” to pay homage to the profligacy of private enterprise without corresponding government oversight and price control is the law of the land and the template of the future. Obama has used the Democratic brand the same way he has used the constitution, like toilet paper to achieve his own deeply conservative ends and he has succeeded. The details, the software that herds the cattle to the pen is mundane and even a little embarrassing as the first cries of slaughter are heard (also, too many “yes” men to take note that a failure was possible and farthest from imagination that it might eat him alive).
Obama is more interested in software that gives him power, such as power over life and death at 6000 miles away (of course in reality only to take life, not to give it). Obama loves control and he has come to love power and therein lie his real priorities – regardless of how they might mesh with other powerful entities in his world.. He has keen interest in anything that gives him control and unconstitutional spying on your own citizens, never mind the rest of the world, even if done through an agency, gives him that feeling in spades.
And make no mistake; he has had an affect on the circumstances surrounding drones and spies, digital and other. His lawyers and his direct reports have been busier than those of Bush making secret frameworks that give him powers that only the most abusive periods of aristocracy have indulged in the past.. And much of the software that makes it happen so invisibly and so SUCCESSFULLY has been written while Obama was in office. His blessing, his lawyers, his active interest have been instrumental in ensuring and maintaining the type of quality, the load testing, the stress testing, the regression testing, that has so obviously gone into THAT software. Of course the NSA and the FBI and the CIA were all there before Obama and so were their software procedures and methods of assigning contracts and overseeing projects. But if the President of the United States decides to put a brake on something, if his interest is not active, then those agencies can not continue their efforts at those levels of success for long.
So clearly, at the very least, Obama has models for successful software projects that he can draw upon that involve great complexity, mind boggling scale as well as GIGANTIC data. These same projects have been scrupulously tested and re-tested before becoming operational. They have been so successful that few were even aware of them before Snowden made them public. The have involved not only in-house programmers and management and management philosophy, but contractors and contracting companies and small and large business philosophies and so prove the whole top-down (government), bottem-up (hot-shot contractors) conflict has been successfully navigated; again and again. Moreover, the application at hand, ACA, is not rocket science. It is a business application folks, that has plenty of successful precedent in terms of scale, of complexity and of transactional integrity in other business software that is out there all over the place already. An ambitious project perhaps, but this is hardly a revolution where ape meets big data.
Instead, this is an administration that indulges in too much ass kissing, too much pandering to powerful lobbies, and too much “hard nosed” acceptance of corruption not to get bitten hard on the ass by its own psychopathic delusions.
psychopathicsycophanticIt is absolutely true that we do not know root cause, and it seems really obvious to blame gov’t procurement. I make more of a case in the following posts, but the primary reason I focus on the software disciplines is that the same mistakes are made without gov’t procurement.
I do not blame the contractors for making a business of cargo cult. They are paid well to be scape goats.
Obama has a proven track record of being comfortable with all manner of corruption that benefits his corporate masters or his Executive power (indeed, if it benefits his corporate masters, then to him it is not – by definition – corruption), so looking in the usual places at the usual suspects (the procurement system) is particularly appropriate for him and his ruthlessly corporate administration.
That this is now bighting him in the rear end would be very satisfying if it were not also causing so much harm to individuals. Ultimately they will suffer, not O-Betrayer.
Note, I found your idea of a new pivotal point in software evolution, engineers confronting big data, to be quite interesting and plausible regardless that it doesn’t seem necessary to explain the ACA fiasco and wish I had made that clear in my comment above.
bightingbiting1. There are now at least 4 million apps for Windows. If that is failure, you are measuring the wrong thing.
2. Most software coders make OK money but nothing crazy, and they have to work hard.the median annual wage for computer programmers was $72,630 in 2010.
3. Most programs I buy and use are enormous, and work pretty well right out of the box. Updates arrive ovre time, but for the most part, these programs do what they are supposed to.
so, the entire article is incorrect.
Reading comprehension failure and as a result, irrelevant and inaccurate remark. This post is not about prepackaged consumer software.
Yves, Lambert: I have tried to access the Topic Archive: “Health care (171)” right here on NC a half-dozen times in the last 15 minutes and each time my browser (IE 10.0) has been blown out of the water. Is this likely to be a problem on my end or an instance of cosmic irony?
Hi — It works for me. However, I would anticipate that with the redesign, there will be a new format for the archives less challenging than the health care archive page, which includes a lot of complicated HTML.
I have to second some of the other posters in that I don’t see a failure of engineers as much as a failure of management. And this isn’t specific to software. Just look at defense acquisitions. Even the vaunted security apparatus of our nation isn’t immune from disasters. Just ask the FBI how well its planned integration with the CIA’s information systems are going…
Methinks Mr. Goodwin has rose-colored glasses as he waxes nostalgic for the past :-) There were plenty of failed projects in the past. We just forgot about them and moved on. And the management debacles that current projects demonstrate aren’t new. _The Mythical Man Month_ was written in 1975.
Finally, I’d like to point out one contrast. Democrats widely believed that Republicans had gained an IT edge during the 2004 election that was a major factor in their victory. They set about to correct that over the next several years. In 2008, Obama’s campaign had a truly impressive system that allowed campaign workers around the country to access massive data warehouses of whatever electoral data they needed to plan their GOTV efforts. It was all so easy to use that they didn’t even need any special workstations: just bring your laptop, connect to the intranet, and all the applications were instantly accessible over the web. Everything from individual state voter records to their email and social media campaigns to near-realtime data from their phone banks and block walkers was integrated into a nationwide system that anyone with a laptop, the appropriate passwords, and a few minutes of instruction could start using.
So even now, extensive computer systems can be built, fairly quickly, and fairly cheaply (I doubt Obama’s campaign spent more than a fraction of the ACA’s IT budget), with spectacular results. Obama even knew how to do it once. But I suspect the combination of a lumbering bureaucracy, conflicting goals and bosses (feds vs state govt vs insurance cos vs private citizens), and the use of outside contractors who could care less as long as their fees are paid, were far more detrimental than any inability to deal with ballooning complexity on the part of today’s software engineers.
Did I say earlier software had fewer problems? That is certainly not my memory either.
We’ve also got a lot of talent and resources tied up working on the problem of how to spy on everybody all the time.
Goodwin needs a good editor; his post is way too rambling to get his main ideas across clearly.
I’ve worked in this industry, primarily in systems startups, since 1985, and started programming in the early 1970s (and it is those, not 1990s, machines, that are 10,000 times slower than today’s). I’ve worked on large multi-company projects, some successful, and some abject failures.
There’s nothing fundamental about this business that should have made healthcare.gov such a disaster. Making REST/HTTP calls to a half-dozen databases as part of a transaction is not rocket science (I’ve done this stuff myself, even for fun projects like displaying shows from NetFlix).
There has been very poor communication between the government and the prospective customers. I’ve noticed, for example, that the commonly reported problem of not seeing available plans after creating an account happens 100% of the time when using Firefox or Safari, but not with IE9 or Chrome. Why doesn’t the web site warn you when you connect with an unsupported browser?
If the reports are accurate, major specs, such as whether you need an account to see plans and costs, were still changing the month before the site went live. That’s an example of insanely bad project management. And the final choice, requiring an account to *look*, was also a really bad choice.
Anyway, my point is that decent project management could have made this a much better experience. I have no idea whether it can be salvaged with an additional month of effort, or not — I’ve certainly seen systems that just needed a bit more debugging to work well, and I’ve seen others that look very similar from the outside, that were total write-offs.
One point that I *think* Goodwin was trying to make is that we’re dependent upon computers actually working for our economy to function. Transportation scheduling uses computers all the time, and without that, no food shows up in your supermarket. If payment systems go down, it doesn’t matter if food shows up, since you can’t buy it. This has been a real issue that’s been sneaking up on us over decades, and it’s important. But it isn’t white healthcare.gov sucks.
Ive been a lurker on NC for a long time, but have never posted anything till now. Why? Because I happen to know a LOT about software engineering, specifically operating systems and encryption, which is what my past research delved into. Bob Goodwin’s articles on software have been, IMHO, verry weak in argument, usually blaming the ‘bad coder’ and ‘no standardization of code to detriment of software’ jargon that is completely useless to talk about let alone discuss because software is very reliant on its environment, and for every example Bob Goodwin writes about how software fails, I can find ten examples of how it succeeded and vice-versa.
“It has been a good generation to be involved with software. The scarcity of the skillset combined with the demand for the output have generated outsized incomes, while the work has been consistently rewarding.”
–Bob Goodwin, delusional “software engineer”
Reality is the exact opposite. There are so many software engineers out their right now that most end up going into IT because its harder to outsource.
Another “gem” is:
“The growth of information, and the resulting mayhem, have left us unable to propagate the needed skills of our software craftsman.”
Did the internet stop working suddenly or am I missing something? We have spectacularly fixed so many problems with past protocols, such as the “cache poisoning” fiasco where in the past, one could literally redirect queries from, lets say BOA to a clone site, that functions exactly the same way. And how do you equate more computers, larger data transfer volumes, to badly trained software engineers? If anything, more computers and better access to information has lead to much more competent coding. Its like you extrapolated backwards from healthcare.gov not working, which is a managerial problem, not a coding issue.
This post is complete garbage, along with the previous “software engineering in crisis”
There is a lot of opinion similar to yours. Lots of people call themselves programmers, but to hire one person at Amazon and had to first approve 100 resumes. Facebook is paying people 200K on there first day of work after they get a PhD. And then stock options on top of that. Maybe there are a lot of bodies, but a real shortage of talent.
Two weeks ago I wrote down every time software caused a delay in my life. I stopped writing after 30, probably could have gotten to 100. Do you think software is good enough?
“Two weeks ago I wrote down every time software caused a delay in my life.”
I’d like to see that list. And why don’t we think of that time as a tax?