By Don Quijones of Spain, the UK, and Mexico, and editor at Wolf Street. Originally published at Wolf Street
Internet banking has become a crisis-prone business in the UK, as the online platforms of big banks suffer regular outages and other forms of IT disruption.
Friday morning, the online systems of the Royal Bank of Scotland, Ulster Bank and Natwest — all part of the RBS Banking Group — crashed in unison, leaving millions of customers unable to pay bills or view their balance on their online and mobile accounts. The group has 19 million customers in the UK and Republic of Ireland and 5.5 million active mobile app users.
After around five hours of chaos, the RBS Group announced that the problems had been resolved. The failure had apparently been caused by a “technical glitch” — a word that is being used with increasing frequency by high-street lenders — in a regular update to their firewall. The bank emphasized that it was an “access issue” and there is no evidence that customer data was compromised. But then, it would say that!
On Thursday, it was the turn of the UK’s largest bank, Barclays, whose website and telephone banking service crashed for around seven hours, leaving frustrated customers locked out of their online accounts.
Fed-up customers took to social media to vent their anger, with some complaining that they were unable to access their accounts not only through the Internet platform but also ATMs. Barclays has around 24 million UK customers, though it’s not clear how many of them were affected by the outage.
The bank told customers that they should still be able to make payments to existing payees through mobile banking, though new payees weren’t possible due to the incident. It also claimed that payments into accounts were unaffected by the issues.
One alarmed customer begged to differ, complaining to the BBC that a payment due into his account had gone missing, while another customer reported the systems inside branches being down, preventing customers from carrying out transactions even in the old fashioned, pre-digital way. By mid-afternoon, the IT “glitch” — that word again! — had been “resolved,” though no explanation has yet been given as to what caused it.
A few days earlier an outage at online challenger bank Cashplus, which targets people with poor credit histories, left customers unable to access their accounts, make cash withdrawals, or make or receive payments. The problems prompted Nicky Morgan, chair of the Treasury Committee, to ask Richard Wagner, chief executive officer of Cashplus, for an explanation of what happened and how victims of the outage will be compensated.
In other words, over the last two days, dozens of millions of UK bank customers have been locked out of their online accounts at different banks.
In terms of RBS, this is not the first time this year its subsidiary Natwest has suffered an outage. Its banking app went down briefly in April and in July a glitch with its card payments left customers unable to use their cards in shops or online.
RBS, the largely state-owned lender that has cost British taxpayers almost a hundred billion pounds in bailouts, losses, fines and legal fees, also has a rich history of outages, including a major blackout in 2012 that lasted for over a week, disrupting customers’ wages, payments and other transactions. The outage was allegedly caused by an “inexperienced” RBS tech operative’s blunder. For the duration of the blackout, the only means many customers had of accessing basic banking services was to visit the local branch.
That, however, didn’t stop RBS from embarking on a branch closure rampage, blaming the growth of internet banking for its decision to close one in four of its branches. Now, it can’t manage to keep those web-based services up and running, leaving customers even worse off. Even as the lender has increasingly digitized its services, it has consistently downsized its IT services team. In 2017 it revealed that it planned to axe 900 IT jobs by 2020 and is doubling down on its outsourcing of IT roles to India to reduce costs.
This underscores one of the major problems high-street lenders have with technology. They never treat it as a mission-critical aspect of their business, even as that business becomes increasingly dependent on technological solutions to stay competitive.
Bottom line-obsessed bank executives are always looking for cheap, short-term shortcuts to IT issues, with the result that lenders — particularly, but not only, in the UK — have for decades under-invested in their sprawling, creaking, accident-prone legacy systems dating back to the primeval age of COBOL and mainframe technology. And if some banks had been thinking about trying to finally move off their legacy systems and drag their IT platforms into the 21st century, the recent botched IT migration at mid-sized TSB, which continues to sow chaos 23 weeks after it was supposed to be ready, will not encourage them to do so.
Time is running out. March 29 is the deadline. Urgent action is needed. But it’s not happening. Read… Disorderly Brexit Would Trigger Mayhem in Derivatives Market
20 comments
Comments are closed.
This is what happens with under-regulated businesses that are abusing the public square, and the public in general. One can only hope that government regulators would rely on IT experts to right the ship. My sympathy goes out to the bank’s customers, not to the bank management.
Daring to re-post my one little personal recent experience with a large regional bank:
“Big regional bank I happen to ‘have money in” just had a “payments system malfunction.” Caused the president of said bank to have to go all “strategic apology” to reassure us mopes who put our ones and zeros in their Cloud that “hey, the money is still there, and we will work really hard to be sure that something like this is as unlikely as possible to happen again…”
Dear [customer]:
I want to apologize personally for any inconvenience the disruption in our digital systems earlier this week caused you or your business.
I understand and regret your frustration, and our team stands ready to work with you on any concerns. I assure you that for every SunTrust teammate, including me, the goal and purpose of our work is to ensure your financial success through convenience, trust and confidence.
Issues during a scheduled system upgrade led to the access problem. Those issues have been addressed, but more importantly, SunTrust will continue to invest substantially to bring you the most reliable and valuable technology, combined with outstanding service.
Thank you for being a SunTrust client.
/s/
Bill Rogers
Chairman & CEO
I am, like, totally reassured now… How little most of us realize how vast our mope vulnerabilities are — or at least we really, really do not want to think about all that.
Interesting to me that the various New Deal programs discussed here recently worked as well as they apparently did. I wonder what the organizing principle that drove the people that activated and managed and worked in them could be stated to be?”
That was then — this is now.
“Dear Chairman,
From now on, I will deposit checks in your bank and withdraw enough cash to pay all my small bills to local merchants, or I will continue to write and mail checks to medium sized businesses. Corporate and tax bills will be paid by autopay. If you have I.T. problems, that’s between you and them.
Thanks,
A customer
Don Quijones, you have made an error in your report. Not about the IT banking debacle but about the bailout. Taxpayers’ money was not used to bail out the banks. Government money was used for the bailouts but taxes had no role to play. Taxes are not used for government expenditure. It is a Treasury bookkeeping exercise. Taxation does serve essential social functions but underwriting government expenditure is not one of them.
Absolutely correct larry. It’s high time commentators ceased the neoliberal meme that taxation pays for stuff when anyone with half a functioning brain cell knows that taxation drains fiat from the economy. A better explanation would be that propping up banksters with Government money deprived the public of claims on the fiat thus potentially depriving them of real resources that may have otherwise been exchanged for that fiat. Even this is shaky as a Government may issue any amount of fiat provided there are sufficient goods and services to exchange for the fiat. The scab of neoliberalism will never be scraped away when the use of their jargon continues – it actually perpetuates the ideology.
On a broader front, while it’s fine for Government money to support institutions that provide a public good, it’s tantamount to treason for democratic Governments to throw fiat at an institution and not hold to account the perpetrators of the collapse. While no one goes to jail the mistrust of the relationship between Government and quasi private institutions will continue. It is sovereign fiat after all and owned by us – the polity are just a middleman.
One part of this story jumps out at me:
My guess is that there was a failure of their primary cloud services provider that forced them into a Continuity of Operations (COOP) mode, which resulted in a period of limited capabilities while operating on their failover backup infrastructure. This type of event would typically be the result of a connectivity loss and not a cyber security breach, however some contingencies may activate COOP if certain cyber sanity checks were triggered.
My experience is that many current COOP architectures are built on design assumptions from brick and mortar days when most all infrastructure was under one roof, or at least, under single ownership and control. As infrastructures have predominantly migrated toward cloud services, COOP designers forget that cloud services, while being very agile and affordable, cannot be considered to be equally reliable as brick and mortal, and perhaps more importantly, they fail and restore differently. All in all, cloud services are more desirable than brick and mortar, but the COOP designs should plan for much more frequent failures and therefore build in significantly greater redundancy in computing and data infrastructure. This redundancy was typically limited to key infrastructure because of expense, but cloud services can offer affordable (for large enterprises) solutions for computing resources, applications, connectivity, and data.
One of the key benefits of cloud is greater redundancy, cheaper virtual computers that can be cloned, spinned up, and shut down in seconds. But you have to design for it. One tool is the Chaos Monkey: a script that randomly switches servers off during office hours (https://en.wikipedia.org/wiki/Chaos_Monkey). Everybody is ready for it because it happens all the time.
But banks are notoriously bad at changing their processes. They have always been able to just throw money and bodies to problems, and at least muddle trough. But the internet is not like that: one little slip, a certificate that lapses, a misconfigured firewall, and systems cannot talk with each other and shut down and start throwing errors to user’s screens, and transactions pile up and accounts get out of sync. Bad!
“You’re a mean one, Mr. Glitch.”
Lets see if i get this right, the web pages and apps crashed, but the parts that kept working didnt? and the crashing parts were the shinny new parts? now i wont say your wrong on how banks and finance tend to under fund IT, but it usually applies to any business now days. they will keep the servers they bought years ago running until they wont run at all. now they have the new toy, cloud, care to guess how that will work out?
If the banks can’t keep their systems up and running now during normal times, what are they supposed to do next March when Brexit hits? If I lived in the UK, I would withdraw all that I can in cash in the lead-up to Brexit just in case. I bet that all those voices in the UK bleating about abolishing cash and going digital are staying quiet at the moment. I wonder if countries like Sweden are paying attention?
Quinones ends with a Cassandra bit on “Disorderly Brexit Would Trigger Mayhem In Derivatives Market.” As one who fails to see any redeeming social value in said market, might I ask for an explanation from the author on why ordinary people should give a sh!t if the whole derivatives scam collapses? I read his linked article, and other than hand-waving in the direction of “carnage,” what really happens to food, shelter, medicines and the rest of the stuff and services that ordinary people depend on, if this —
Time is running out. Until now, the European Commission and ECB have shown scant willingness to accept any form of equivalency between British and EU financial services. And without that, there is a genuine risk that a disorderly Brexit on March 29 could set in motion an unraveling of an already hugely volatile, highly interconnected derivatives industry.
were to eventuate? The whole “industry” is froth and corruption, far as I can tell, and even the “players” in it don’t understand how it all works — just chase personal gain, knowing that each transaction has a designed and pre-built losing position built into it, and most of the elements of fraud and three card monte, and only exists due to absence of “pro-social” and not-captured regulation, and the corruptly obtained guaranteees that “government” will pay off the losing bets and rescue the improvident and corrupt from their behaviors. Said rescue being underwritten by accelerating extraction of limited physical resources and the drudge labor of the lower orders in the Real Economy.
I see it as just a chance to kill the beast. But then I am just a little mope, not versed in the virtues and value of these “volatile” and vulnerable and “highly interconnected” ever-inflating balloons full of quadrillions of “notional dollars” and buoying up yuuuuge fee-profits and bonuses…
“Erin go Brexit?”
Burn it to the ground.
But let’s get rid of cash. Every since my bank at the time deleted my online bill pay record for my mortgage server presumably to prevent my providing evidence of no default, I have been a cash only girl. Luckily (Ha) I am not wealthy so it works for me. Just remember, you only have the money in your account THEY SAY YOU have. And you only paid the online payees THEY SAY YOU PAID. Print everything and if possible get the printouts stamped as accurate at the bank occasionally. Defeats the purpose of ‘online’ banking but you may need it someday as I did.
Ditto, Dolley!
Anyone think this may be a problem of the legal regime? If you try to represent a legal system as code what you’re doing is mathematizing a political system.
If that system is inconsistent the code must be inconsistent or a poor representation of that system. Common law systems are particularly broken legal systems, old and riddled with case based hacks.
The natural prediction is that banking, finance and billing codes in the UK and US will be particularly prone to breakage and cost over runs.
They will also be particularly prone to abuse by those able to hack these legal systems at the top and bottom – not societies of laws but of lawyers.
Probably need adverserial deep learning systems to handle billing!
Usually inconsistencies become apparent during design or implementation, and are kicked back to the policy analysts if it’s based on inconsistencies in the underlying law. It then either goes on the calendar to get resolved legislatively, or (more likely) a special treatment or accommodation is agreed for that particular case and translated into system specifications. Generally it’s possible to handle them robustly if you spot them in advance.
Unexpected inconsistencies are more of a problem and can indeed cause incidents. However I wouldn’t expect them to cause widespread outages of this kind unless the system was extremely poorly designed, because by their nature they tend to apply only to special cases and very small subsets of the population.
The use of the term ‘glitch’ is indeed worrying as it has no informational content and suggests that they don’t yet know what the problem is. They did say in the first case that it was related to a scheduled firewall update. That suggests that an access rule was changed in a way that prevented online systems from functioning, but for some reason nobody knew in advance that it would do that. Either the change itself wasn’t communicated properly, or the approval process for changes is lacking, or the underlying system dependency isn’t understood sufficiently by approvers that they knew what the impact would be, or… (pick your own variation).
In any case it’s most likely a management problem. They made a scheduled change that broke everything. Clearly they didn’t know it was going to do that, or they’d have implemented the change differently. Therefore, their understanding of their own systems is insufficient. That’s always true to some extent at any sufficiently large organization with complex IT infrastructure, but “what systems will break if we make this firewall change?” is absolutely the kind of question for which you’d expect to receive a definitive and accurate answer.
You miss my point. Essential underlying incosistency leads to hacks – case based solutions that are increasingly complex. Giant if else trees. More and more subclasses. State machines proliferating.
Case based common law is like this – hack upon hack that code must represent. Attempts to impose consistency that only create more inconsistency.
I’m not arguing that a particular inconsistency is glitching. I’m claiming that certain representations of law are simply too much of a mess to practically put into code using nonsuperhero coders and managers.
You can’t write code to describe us supreme court decisions because they are inconsistent shit for example. More so for the financial system.
Yet if you listen to the Cognoscenti that fill the footnote-splattered pages of all the Law Reviews of all the Very Special Law Schools, there are “threads” that can be teased from the hairball of precedent that Make It All Completely Consistent And Rational (in accordance with the world-views and political and economic preferences of the smart-ass pontificators in their bow ties and raised noses…)
Of course, then, there’s the reality:
Supreme Court Justices Admit Inconsistency, and Embrace It
https://www.nytimes.com/2014/12/23/us/supreme-court-justices-admit-inconsistency-and-embrace-it.html
Explaining how “Justices” go about the “strategic apology” game. And excuse themselves for not “developing a consistent jurisprudence and sticking with it.” Because one must maintain the “right to be wrong,” so “the law can grow” into cancers like Citizens United. Which with appropriate rhetoric can be fit neatly within the “four corners of the Constitution.”
Hey, it’s the “genius” of the system, ya know?
That part I certainly agree with. If I was designing an IT system and encountered one of those cases then I’d create an exception scenario and flag it for manual processing. If the laws themselves are inconsistent, it’s a problem that’s outside the scope of IT to solve. Let the lawyers and the courts deal with it – that’s what they’re paid for. IT is best suited for quick, automated execution of clear-cut processes – essentially the stuff that front line government employees would otherwise handle. (Try asking a USCIS phone support person for a legal interpretation and see how far you get).
My point was that the article was about widespread banking outages and you were (or at least seemed to be) quoting legal complexity/inconsistency as a possible cause. While the problems you describe are real, I don’t think they are of the kind that would contribute to instability and outages.
It isn’t just banks that have this problem. And it’s caused by a perfect storm of C-suite types who don’t understand tech but feel that moar of it will increase profits, and tech companies who are only too happy to take advantage of the naivete of the C-suiters and sell them buggy software that doesn’t work well and that they probably don’t really need in the first place.
Meanwhile an entry level office job now pretty much requires if not IT experience per se, then at least a strong ability to figure out how to work around all the IT ‘glitches’ to get one’s work done. Because the CEO who just got a bonus for the cost savings accomplished by firing employees and replacing them with software sure isn’t going to put that $$$ at risk by hiring a larger IT department to deal with all the larded-on tech.
Better to have the company just limp along with pissed off employees and hope you get the golden parachute before it all comes apart.