Ships, Chips, and IPs: The High Costs of Tight Coupling

I’m sure this line of thought has already occurred to many readers, so forgive me for belaboring it, since it appears it can’t be said often enough. The press on almost a daily basis is showcasing the cost of over-optimizing businesses and systems for efficiency, which regularly results in tight coupling. “Tight coupling” is vulnerability to cascading and potentially catastrophic failure. And it’s hardly a new concept: “For the want of a shoe, a horse was lost…..”

As we’ll discuss later in the post, tight coupling induced messes generally if not always impose significant costs on innocent bystanders. Yet bizarrely, this sort of “profit for me, costs for you when not terribly unpredictable events occur” is seldom treated as an externality. The time is overdue to demand more disincentives, like taxes or prohibitions.

Tight coupling occurs when systems are so closely integrated that failure or unanticipated interruption of one activity propagates so quickly that the breakdown process can’t be halted quickly. This concept was popularized in the runup to the crisis by risk manager Richard Bookstaber, who saw first hand how easy and dangerous it was to create tight coupling in financial services. He’d done so by selling portfolio insurance to institutional investors in the 1980s. Its automatic selling in market swoons turned an October 1987 stock market decline into a rout. From Bookstaber in a 2007 Wall Street Journal interview:

Financial markets are like jetliners and nuclear-power plants, he says: They are highly complex systems that require numerous interrelated steps to work together. But the “tight coupling” of these steps means when one of them goes wrong, the error rapidly ripples through the entire system, producing disasters like the ValuJet crash in 1996 or the Three Mile Island nuclear accident in 1979.

We’ve seen numerous examples of this syndrome. Some of that is due to Covid pressures on supply chains. But some of this is Shit Happens.

After decades of worry that Iran might interfere with oil shipments via threatening the Strait of Hormuz, instead a supersized container ship has wedged itself so well in the Suez Canal that it is now expected to be days or even weeks before it is freed. From the Financial Times:

The 400m long Ever Given, operated by Taiwan-based Evergreen Marine, is one of the world’s largest container ships and is weighed down by thousands of tonnes of cargo, with its position suggesting its bow and stern are wedged on the shallower banks of the canal edge at the southern end.

While salvage experts hope higher tides may help release the vessel if the dredgers can remove enough sand and soil, fears are growing that refloating the Ever Given may prove more complex.

Salvage companies may need to remove fuel from the ship’s tanks to help lighten the vessel and consider offloading some of its containers — an arduous task given the relatively remote location, sheer height of the ship and lack of infrastructure on the ground.

At CBS (hat tip furzy) an expert said that even if the Ever Given were freed today, it would still take a week to clear out the traffic jam.

Ships that entered the canal are being backed out by tugs. The long way around Africa to destinations in Europe and the UK takes roughly 10 days longer.

The knock-on effects go well beyond a short term increase in oil prices, pricier goods in the UK and to a lesser degree, the EU, and the potential for manufacturing disruptions if any “just in time” supplies are delayed. The vector for more widespread problems is containers. Even before this monster cock-up, Lambert had pointed out in Water Cooler that containers were in short supply, creating a scramble among shipping companies. Now we may see toilet paper shortages! From Bloomberg:

The world really doesn’t need more toilet paper problems. But unfortunately the biggest producer of wood pulp — the raw material for products including bath tissue — is warning that the global crunch in shipping containers could start creating supply snags…

The container crisis, sparked by huge demand from China, has been playing out for months. But Suzano’s warning is among the first major signs showing the spillover into other shipping markets. If the squeeze continues to increase freight costs, it also raises the specter of accelerating inflation.

The question is why was such a ridiculously hypertrophied container ship even permitted to go through a teeny choke point controlling 12% of the world’s sea traffic when it was well known that it was particularly vulnerable to this type of foundering? Lambert hoisted this explanation from FT Alphaville:

The Suez Canal is basically just a 24m-deep ditch dug in the ground to let the ocean in. When a ship comes by and displaces the water, the water has nowhere to go; it gets squeezed in between the ship’s hull and the floor and the sides of the ditch…. [Evert Lataire, head of the Maritime Technology Division at Ghent University in Belgium] wrote his dissertation on a similar phenomenon as a ship passes close to a bank: the bank effect. The water speeds up, the pressure drops, the stern pulls into the bank and, particularly in shallow water, the bow gets pushed away. Stern one way, bow the other. A boat that had been steaming is suddenly spinning. It’s a well-identified phenomenon; in 2009 Ghent University’s Shallow Water Knowledge Centre put together a whole conference about it. Clever pilots on the Elbe, according to Lataire, will use it to shoot around a bend. However: the more water a ship displaces, the stronger the effect. And the closer the side of the hull is to the shore, the stronger the effect. The bigger the ship, the faster the bow shoots away from the bank.… [On the video,] everything happens quickly, in a way that looks a lot like the bank effect. Bow shoots away from the bank. Stern continues to hug the bank and move north. Ship spins. Bow bulb punches through the riprap.

There are plenty of routes that don’t involve going through canals. East Coast of South America to the US or Europe. North America to Europe or Africa. East Coast of Africa to Europe. TransPacific. What moron decided to use monster ships for Asia to Europe? And why did the folks who run the Suez Canal allow it? If the Suez Canal Authority won’t bar this type of ship in the future, maybe the insures will jack up rates so catastrophically so as to accomplish the same end.

Chips are another example of neoliberalism meeting with “Shit Happens” but with potentially more serious long term consequences. It doesn’t require a lot in the way of observational skills to recognize that chips are an essential resource. How well would the US get on with no new chips (that includes “new chips” embedded in products) for eighteen months? Specifically, how would our vaunted military-industrial complex fare? How deep is their stockpile of spare parts and inventories?

It wasn’t hard to see, once the trend got going, that major manufacturers would increasingly buy chips from lower-cost providers (or who offered other favorable terms), which often meant Asia, as in abroad. Yes, it would have been less efficient to force manufacturers to incorporate some domestic production, but safety and redundancy come at a cost. This exposure intensified via the propensity of manufactures to increase the software content of their products, as we’ve seen with cars, ostensibly to increase safety and create feature bloat satisfy consumer demand for more goodies like entertainment centers, but also to force obsolescence down the road.

A bit over a month ago, we posted on the auto industry’s chip woes:

US automakers have had to cut production due to limited supplies….Short chip supplies are also hurting some other consumer products companies. From CNBC:

Automakers across the globe are expected to lose billions of dollars in earnings this year due to a shortage of semiconductor chips, a situation that’s expected to worsen as companies battle for supplies of the critical parts.

Consulting firm AlixPartners expects the shortage will cut $60.6 billion in revenue from the global automotive industry this year. That conservative estimate includes the entire supply chain — from dealers and automakers to large tier-1 suppliers and their smaller counterparts, according to Dan Hearsch, a managing director in the New York-based firm’s automotive and industrial practice….

Automakers are scrambling to get supplies of the chips, which have extremely long lead times due to their complexity. The shortage is far down the supply chain, causing a ripple effect through the entire network….

A 26-week lead time is needed to build the chips before they are installed in a vehicle, according to Hau Thai-Tang, Ford’s chief product platform and operations officer.

The origin of the shortage dates to early last year when Covid caused rolling shutdowns of vehicle assembly plants. As the facilities closed, the wafer and chip suppliers diverted the parts to other sectors such as consumer electronics, which weren’t expected to be as hurt by stay-at-home orders.

Amusingly, Toyota has no chip problems. Per the Wall Street Journal:

A sharp rebound in car sales has paradoxically threatened the auto sector’s recovery as car makers contend with severe shortages of automotive chips. Except, apparently, for Japan’s automotive colossus: Toyota says it’s doing just fine…

The problem could last for months as it takes time for chip makers to ramp up capacity and readjust their product mix.

Back to the current post. The take above proved to be optimistic, thanks to a chip factory fire in Japan. From AutoNews at the start of the week:

Toyota, Nissan, Honda and other Japanese carmakers scrambled on Monday to assess the impact on their production of a fire at a Renesas Electronics chip plant that could aggravate a global semiconductor shortage.

Renesas has said it will take at least a month to resume production at a 300mm wafer line at its Naka plant in northeast Japan after an electrical fault caused machinery to catch fire on Friday and poured smoke into the sensitive clean room.

The fallout on carmakers could spread beyond Japan to auto companies in Europe and the U.S. because Renesas has about a 30 percent share of the global market for microcontroller unit chips used in cars.

Let me stress that it would take an extraordinary level of inside knowledge even to guesstimate what kind of tradeoffs would be involved in creating more robust supply chains and/or having more inventory cushions in critical parts. But part of the issue is that “just in time” practices appear to be so deeply baked in that I suspect incumbents have an almost emotional resistance to rethinking them. And also recall that at least among US automakers, and I suspect many of their competitors, another long-term trend has been to shrink the number of suppliers. The arguments in favor were to deepen relationships, allow for more knowledge transfer, and of course, get better pricing. But the tradeoff again is more fragility.

Another way of deliberately creating fragility is intellectual property. Some of that can come about, per above, by the use of proprietary or otherwise hard to buy chips and parts, or software. We’ll get back to our bigger point of how this can create exposure to catastrophic risks, as we can see even on a more local level via a favorite NC topic, right to repair. ZDNet tells us that service problems under Covid have created a bigger impetus to cut back manufacturer restrictions:

Much of the legislation is about medical equipment because, in states like California, during COVID-19, hospitals were in desperate shape. Patients, of course, need ventilators to survive, and there weren’t enough to go around. If one happened to break down, it required a technician employed by the medical device manufacturer to repair it, and in-house staff couldn’t do simple repairs to life-saving devices. There are now two legislation pieces for medical equipment under consideration for California and Hawaii: SB 605 and SB 760. Texas has also introduced a similar medical equipment bill: HB 2541…

The pandemic has driven a lot of this because people are spending more time at home, using their products, and stuff is breaking down.

As a result of all this impending legislation, the hope is that the benefits of the new laws also trickle down to the average consumer. It’s not just a “right” to repair; it’s an “ability” to repair. The prospect of opening up a device like a smartphone, a current-generation Mac or PC laptop, or an IoT box scares many consumers, even if they have access to the parts network and the proper documentation.

I have more distaste for fixing things than just about anyone you will meet, due to lousy manual dexterity and low tolerance for frustration. But even I would regularly swap out memory cards in my computers. There are lots of basic repairs and upgrades that mere mortals could once do and should be able to do again.

Now to a more serious example: drug patents. As we can see in a much narrower focus from right to repair, intellectual property protections create differentiation and for patents, monopolies by design. There’s been ample discussion of how unduly Big Pharma favorable the current regime is. US prices for insulin is a disgrace.

We’ve been featuring posts by Anis Chowdhury and Jomo Kwame Sundaram on what they have called “vaccine apartheid,” that rich countries, prototypically the Global North, are hoarding Covid vaccines, so that supplies won’t be available to poor countries until 2023 (no typo!). And to add insult to injury:

Reputedly the cheapest vaccine available, Oxford-Astra Zeneca’s is sold to EU members for around US$2 each. Although trials were done in South Africa, it still pays more than twice as much, while Uganda, even poorer, pays over four times as much!

US negotiated bulk prices, for Moderna and Pfizer-BioNTech vaccines, are much higher, at US$15.25–19.50 per dose in several contracts, yielding 60–80% profit margins! Moderna will charge the rest of the world US$25–37 per dose.

Developing countries seek a temporary suspensions of the World Trade OrganizationTrade-Related Aspects of Intellectual Property Rights agreement to increase supplies and lower costs of Covid vaccines, as well as tests and equipment.

Now we pointed out this picture not be as dire as the one Chowdhury and Sundaram anticipate, with 2.5 million avoidable deaths. They will be forced to use preventatives and prophylactics like Vitamin D, zinc, ivermectin, hydroxychloroquine, aspirin, quercetin and progesterone as their first line of defense. But if they aren’t successful at making these interventions work at scale (remember, these treatments need to be ongoing), we can expect SARS-COV-2 to remain active in large swathes of the world and keep mutating. Great for Big Pharma, not so hot for the rest of us.

And before you pooh pooh the vaccine-making capabilities of poorer countries, as some readers did, microbiology prof KLG (who has been doing basic research as well and so has plenty of bench experience) said China, India, and Cuba clearly had the capacity to make any of the Covid vaccines. And that’s before you get to advanced economy drug companies who didn’t devise a winning Covid vaccine and want to keep their hand in the game. Key sections of his comment:

In the present context, there is nothing about an mRNA vaccine that is not obvious, going all the way back to the Central Dogma of Molecular Biology (Francis Crick, 1957): DNA makes RNA makes Protein…

And while it is true that mRNA vaccines are (relatively) novel, there is nothing at all remarkable about them based on the past 60+ years of research on gene expression and protein synthesis in prokaryotes and eukaryotes. They are certainly not transcendent in the sense that a microprocessor transcends a mechanical clock. There is essentially nothing in the manufacture or delivery of mRNA vaccines that is not commonly used in thousands of molecular biology laboratories all over the world, which is not to say there is nothing proprietary involved or that this is not technically demanding. This is also true of the “genetically engineered” adenovirus-based vaccines. My laboratory used essentially the same adenovirus platform to express a large muscle protein in cultured chicken cardiomyocytes 15+ years ago, and the technology was not new then. Whether the non-replicative adenovirus will present cellular or immunological problems in and of itself is a somewhat open question….

The mechanisms of the fusion/entry of LNPs [liquid nanoparticles] with host cells after inoculation have not been described in detail, but the underlying cell biology is well understood. The LNP fuses with the host cell membrane and delivers its cargo (mRNA) directly into the cell (cytoplasm) or is taken up as a vesicle (endocytosed)….

So, in summary, mRNA (and adenovirus) vaccines are either obvious or ingenious, depending on your point of view. Their production is technically demanding, but not beyond the ken of thousands of molecular biologists, biochemists, medicinal chemists and the institutions around the world, public and private, in which they work. There is no good reason that these essential, if experimental, vaccines cannot be produced simultaneously and at scale in any country with the necessary technical and labor infrastructure. There are many such countries, not all of them in the Global North.

Yves again. The point of this technical discussion is to drive home the fact the press has been moaning about how delays in the delivery and/or uptake of vaccine in the US risks new variants getting the upper hand. That bleating conveniently ignores that most of those vaccines arrived on airplanes. New variants will continue to do so unless governments put their boots on the neck of the vaccine and test makers.

Since there seems to be no willingness to do so, we’re about to see the worst sort of externality: inability to tame Covid, with resulting deaths and disability. All due to fealty to overreaching notions of property rights. And since not enough countries are willing to cut the tight coupling created by air travel, the first world won’t escape the blowback.

Print Friendly, PDF & Email

20 comments

  1. Sound of the Suburbs

    It never did occur to Western leaders that no matter how much you cut costs in the West, costs in China would always be lower.
    China was always going to the winner in an open. globalised world.

    Maximising profit is all about reducing costs.
    Western companies couldn’t wait to off-shore to low cost China, where they could make higher profits.
    China had coal fired power stations to provide cheap energy.
    China had lax regulations reducing environmental and health and safety costs.
    China had a low cost of living so employers could pay low wages.
    China had low taxes and a minimal welfare state.
    China had all the advantages in an open globalised world.
    It did have, but now China has become more expensive and developed Eastern economies are off-shoring to places like Vietnam, Bangladesh and the Philippines.

    The West never really stood a chance.
    An environmental disaster was also baked in as environmentally friendly measures cost money and reduce profit.

    1. doug

      I think the ‘western leaders’ could see what would happen. It is not rocket surgery….
      Third wave coming. It did not have to be this way…

    2. XXYY

      The official history now is that China somehow sneakily got hold of US technology and know-how by espionage and used it to build up their technological base at US expense, despite the valiant efforts by US leaders and industrialists to hold them back.

      In reality, US leaders literally could not move fast enough in the 90s and 00s to transfer US knowledge and capabilities to China, and to close down their US plants, in order to make higher profits. Chinese leaders played their US counterparts like so many violins, setting up research and manufacturing centers which foreign companies paid to fit out with the latest technological machines, technology and know-how, which the Chinese then ran with via a huge domestic investment program and energetic industrial policy.

      Now that the predictable (and predicted) results have happened, US leaders survey the ruins of their own country and portray themselves as the victims of foul play.

      1. Synoia

        In reality, US leaders literally could not move fast enough in the 90s and 00s to transfer US knowledge and capabilities to China, and to close down their US plants, in order to make higher profits break Unions and lower wages.

        Please pardon my edit.

    3. drumlin woodchuckles

      I think the Western leaders, especially the American leaders, both realized it and quietly sought to design and engineer for it on purpose, in order to exterminate industry throughout America so as to exterminate industrial unions throughout America.

      It was all part of Plan Haiti for America.

      On purpose. That’s what I think.

      ( And now that I see Synoia’s comment just up above, I would agree and go even farther. On purpose, with deliberate and well-informed malice aforethought).

  2. Rick Shapiro

    Tight coupling is merely one aspect of theft by concealment of risk, abstractly not different from marketing (and purchasing!) of complex financial derivatives. Executives reduce near-term expenses in order to bulk up their option income by tight coupling, by slashing r&d and by purchasing assets with concealed risk. The result is theft from long-term stockholders or from taxpayers who bail out systemically important firms.

  3. XXYY

    I believe it was sociologist Charles Perrow who first popularized the dangers of “tightly coupled systems” in his 1984 book Normal Accidents. In fact, he used the Three Mile Island nuclear plant accident as his lead example.

    It’s a super fascinating read even now, also putting forward the then-controversial thesis that accidents in such systems could not be reduced below a certain rate (i.e., would always be “normal”) because measures meant to eliminate them increased complexity and coupling and so made things worse. This idea was a sharp stick in the eye to “high-reliability” theorists who believed all accidents could be eliminated if we just tried hard enough.

    He has been a controversial figure ever since.

    1. Synoia

      Single points of failure feature in that discussion, of which the current Suez Canal blockage is an example.

      Given the Suez is Canal is surrounded by desert, is very busy, and has a one way only traffic management system, why was a second channel never discussed?

      Many of the locals make a living selling goods to stationary ships, so I suppose killing off a large part of Egyptian commerce may intrude on the decision to keep a one way Canal.

      I traversed the Suez out bound and return in 1966 on a LPG tanker in my wanderings. Egypt was not impressive.

      I don’t believe the Roman rule improved it over its old civilization. In addition, Christianity compounded the decline by burning the Library in Alexandria.

      1. RMO

        The library of Alexadria may have suffered a large fire during the reign of Julius Caesar (historians disagree about it) and it did decline under the Roman Empire but the historical burning happened under Aurelian who was certainly not a Christian. Historians are also skeptical about the account of Caliph Omar supposedly ordering it burnt in 643.

        1. JBird4049

          Also, the books were usually scrolls made of papyrus, which need to be stored properly, checked, and IIRC every century or so copied again. A large library like the that of Alexandria required a large workforce constantly working and a lot of floorspace. I forget which which Roman emperor cut off the library’s funding, but no funds means no upkeep. It might take two, even three, hundred years, but eventually the scrolls would just fall apart.

  4. Mary

    And for a massive example of institutionalized complexity vulnerability, California has mandated that new homes have no natural gas hookups, to help prevent climate change.

    https://www.scientificamerican.com/article/california-is-closing-the-door-to-gas-in-new-homes/

    In an accidental, or pre-planned power shutoff, that means you have no hot water, no heat from a gas log fireplace, and cannot cook with a gas stove, which one can still light with a match if there’s no power for the ignition spark device.

    1. Jokerstein

      Virtually all ignition spark devices use the piezoelectric effect to generate the spark: physical deformation of some material creates the spark.

    2. XXYY

      I recently moved into new construction in CA which has (a) no fireplace, and (b) a flash hot water system that uses gas for heat but also requires electric power to work. The house also uses fiber for the phone hookup so no land lines without electricity.

      At least when the power is out we can huddle around the gas stove!

  5. rusti

    Tight coupling occurs when systems are so closely integrated that failure or unanticipated interruption of one activity propagates so quickly that the breakdown process can’t be halted quickly.

    In the world of navigation and sensor fusion, tight coupling entails fusing inputs from several different sources (most often a GPS receiver and inertial measurement unit) into one estimation process, rather than getting independent estimates from each and blending them together. The consequences are pretty much analogous to what’s described in the post. For safety-critical systems it’s easier to separate the inputs and monitor them independently, but performance can be enhanced on average with a tighter fusion.

    Our supply chains might be more like “deep coupling” where the tracking loops of the GPS receiver are steered by the inertial measurement unit, so it works great until you get perturbations and then it really can spiral out of control.

  6. Mikkel Fishman

    I’m very glad to see this post explain such a widespread issue with extreme clarity. It is very near to my heart, as I have dedicated much of my professional life to addressing it.

    I have decided that efforts to correct over optimization are all in vain.

    For me the primary issue goes much deeper than simply reconsidering JIT. Whiles Yves is totally correct that those practices have moved well into dogma, Odum’s Maximum Power Principle predicts that tightly coupled systems are inevitable.

    It states, “systems which maximize their flow of energy survive in competition” which means strategies utilizing a combination of over-optimization and resource exploitation will always win out.

    The only way to address this is through regulating design at the systems level and necessarily limiting the ways that actors can compete. At the very least it requires a critical examination of capitalism, and perhaps even the basis of Western thought, which is fundamentally incompatible with full systems approaches.

    Even then Odum’s law has proven true on the cultural level. Although indigenous and eastern cultures have harmonizing as their primary goal, they were quickly overwhelmed by confrontation with the West.

    I have had fascinating conversations with Chinese and Indigenous friends about their trauma of being forced to become maximizers in order to survive. Through these conversations I have learned that the issue goes far beyond a technical problem and instead represents our deepest ingrained worldviews.

    1. KiWeTO

      An optimized mutation that thrives in one environment often has massive difficulty adapting to a rapidly changed environment.

      One could be the most efficient, cheapest, highest margin manufacturer of CDs. There still isn’t a lot to do once the market isn’t interested in CDs anymore. Or buggy whips. Or twisted pair copper phone lines.

      Unfortunately, humans aren’t so good at the creative destruction of their own optimized (from their own perspective) systems to build the next optimization. There will just be a lot of unexpected consequences when such structures actually break. Resilience might be better achieved by being less than optimized, but how can any decision maker defend such a position, since it will always be made with imperfect information?

      Systemic flaws require systemic solutions, not band aids.

  7. Kouros

    Yup, Ireland has still not achieved the population levels from 1840s, prior to the Great Famine…

Comments are closed.