By accident, CalPERS has let us, and perhaps a lot of other people, know it’s not doing a shipshape job of keeping its network secure. Since CalPERS has the names, addresses, and Social Security number of 1.6 million beneficiaries and also executes billions of dollars worth of securities transactions per year, this is not a good look.
CalPERS exposed a mid-level network security vulnerability to us when an employee using CalPERS’ network and equipment left a comment on Naked Capitalism. This security shortcoming in and of itself is unlikely to have exposed any sensitive information. However, it is an indicator of a failure to do adequate firewall testing. That raises the question of what else on CalPERS’ IT security front might be handled in a casual manner.
CalPERS declined to respond to a request for comment.
This post reflects extensive input from four computer professions: our regular bank/payments system expert Clive, two other professionals with considerable experience in running large networks in the banking/securities industry, and an academic who specializes in cybersecurity. One of the three practitioners also conferred with other colleagues to make sure his assessment jibed with current best practice. This was typical of the comments he got:
One response was “It looks like their network security monkey is pretty lazy”. The general perception is that if they are lazy about trivial tasks, how attentive are they to more complicated serious threats?
To be clear: we are not saying that this security shortfall is a big deal taken in isolation. In fact, despite being classified in software standards as a medium-level deficiency, everyone agreed that by itself, it wasn’t consequential. As one IT professional noted:
The cyber security world is all about splitting hairs, so pulling on a thread like this is likely to produce collisions of technical interpretation. Obfuscation of internal IP addresses in external packets have become such a standard practice, that failure to do so has a stigma of laziness in the biz because there are so many competing options for mitigation, but is also not considered to be a great concern per se.
It’s my experience too, that it is these types of “medium” issues that tend to bring out the most ‘how many angels can dance on the head of a pin?’ geek arguments.
The real issue is whether this is just an isolated case of sloppiness or a symptom of slipshod network management. Clive stresses that an institution like CalPERS needs to have internal procedures that are far more rigorous than for most organizatons:
IT security is in the eye of the beholder. What is fine for one kind of enterprise or institution is definitely not okay for another.
We get people from small and even medium sized companies in as new hires and very often they are totally blasé about the need to step up their thinking to match the environment they are now in. Screw up the PayPal interface on an organic cosmetics wholesaler and who cares. Screw up our interbank payments gateway for more than a few hours at my TBTF and a good chunk of the entire UK economy goes tits up.
How Did We Find Out About This Problem?
An accompanying post on CalPERS today describes how a member of CalPERS’ legal department left a comment under an assumed name on a post criticizing the censure of board member JJ Jelincic.
In our backstage, I saw two IP addresses for that comment, which is unusual. One was the IP address associated with the computer’s connection to the internet. The second address was from a range (10.x.x.x) that is reserved for intranets only1. We’ve embedded CalPERS’ response to our related Public Records Act request at the end of this post.
As CalPERS stated, “Christopher Phillips was the user identified with the internal IP address noted in request number three for the time period noted.” Phillips is a senior staff attorney who has worked at CalPERS for four years. As we stated in a companion CalPERS post today, CalPERS confirmed that it was indeed Phillips who left the comment.
What Are the Immediate Implications of Exposing an Internal IP Address?
The fact that Phillips’ internal IP address was exposed externally does not mean his computer or the network files under his account were outside CalPERS’ firewall and therefore readily hackable. However, even a deficiency like this does increase CalPERS’ security risk. And as one professional put it: “This demonstrates basic firewall configuration incompetence.” In addition to the limited information it provides, a deficiency that reflects laziness might encourage hackers to poke around and see what other weaknesses they find.
We’ll rely heavily on Clive’s explanations in this post, since by virtue of having to deal patiently with un-tech savvy and often stubborn senior managers, he gives accessible explanations of fine points.
Publicly available security standards almost universally show exposing internal IP addresses as a medium or Level 2 vulnerability on a typical 3 level scale. For instance, the security standard, issued by the the PCI Security Standards Council shows a worked example of vulnerability testing and analysis on page 26. “Internal IP address disclosure” is cited as a medium vulnerability.
The UK Special Interest Group which created this the data security standard is 20 years old. Internal IP address disclosure has been in it from the start and was, and remains, a severity 2 vulnerability.
Clive explains why medium risks are worth worrying about:
IT security pros usually class security flaws in terms like “critical” or “medium” vulnerabilities. “Critical” vulnerabilities are those which if left unfixed would directly by their presence allow access controls comprise and/or data loss. The large-scale cyber attack on 15/05 was as a result of systems which weren’t patched to prevent a known vulnerability being exploited. Microsoft classed the vulnerability as “critical” – rightly so, because if it wasn’t patched against, your machine was toast.
“Medium” vulnerabilities, by contrast, are those which do not by themselves allow for compromise or data loss but they do potentially help and enable this to happen. Usually it is because they increase what is commonly referred to as the “attack surface” – the size of the target area a malicious actor can aim for. Exposure of an internal IP address falls into this category. Once you know one internal IP address, you can make an educated guess at a network’s entire range of internal addresses.
This is like an art thief wanting to steal a particular masterpiece because they’ve been contracted to heist it. Initially, they don’t even know where it is in the world. But then if they get some inside intelligence that gives them the address, they are hugely further forward in their ability to plan an attack. They may still have to contend with whatever security arrangements have been put into the address where the painting is (alarm systems, guards, locks) but at least now, they can case the joint and find out what these are.
Armed with an exposed IP address, like a thief in possession of a property address, a hacker can now probe all ports looking for known high-vulnerability ones what have been left open. They can test for services which have not been disabled that contain hackable back doors. They can try the same on what they can now guess are other internal IP addresses.
It is also easier to probe which IP addresses belong to workstations and which are servers. Servers are more vulnerable and a potentially richer target to aim for, so if you can get to know the internal IP address of a server, you can concentrate your efforts where they may yield the biggest impacts.
Another IT professional added:
To follow the analogy of an art thief, a more sophisticated thief would want to map out the identities and locations of objects so that a specific plan could be formed to minimize the potential for alert or response while grabbing the maximum value in the briefest amount of time.
What an internal address exposure does is support the ability to map out a target network. If the mapping is thorough enough, nodes, types, and functions can be determined so that when they get in, they can quickly go straight to what they want to take out. They can also use this mapping knowledge to use some weaker nodes as staging points to load exploitation tools and to leverage any trusts that those nodes may have on the network.
And what about a worst-case scenario, that Phillip’s machine was exposed? Generally speaking, the files on that machine aren’t the target. A hacker is interested in any permissions which that machine has, especially if he can access the permission of the logged-on user. Even seemingly pedestrian permissions like being able to parse the print queue can expose all sorts of random but useful nuggets.
But that’s before you get to the fact that Phillips, as a member of the legal department, himself would be on the list of high-priority targets. Clive again:
If someone compromised my PC, then even if they got no further than the files I store locally, they’d be able to tell nothing much more exciting than that my TBTF is completely useless and wastes vast amounts of money in project delivery or that it gets taken for a ride by all and sundry. While potentially embarrassing, it would merely confirm a commonly-held (public domain) belief.
Conversely, if the CalPERS machine was used by someone on the legal team, then by the nature of their role within CalPERS they would reasonably be expected to have for more sensitive and compromising (compromising to CalPERS if it got out, that it) files – files which might be stored right there on their machine. Meaning that a hacker would need to get no further than accessing the machine, not anything else in CalPERS’ IT environment.
Unless the user was incredibly diligent and never held any high value/high confidentiality material on their local machine, they’d be a prime target for a hacker simply by virtue of who they are and what they do. At my TBTF, a lot of intrusion attempts are *specifically targeted* at C-level execs, audit, IT security operations and so on. High-value targets. The reason is obvious – don’t waste your time on mid level apparatchiks like me, go for the big fish who will have more juicy prizes at stake.
What Does This Security Failure Say About CalPERS IT Vigilance?
Turing the mike over to Clive again:
If I had to guess, I think that someone inside CalPERS intended to deliberately create what is known as an Exposed Host. There are a variety of legitimate and even necessary reasons why you might want to do this – running a web server, an email service, an extranet or similar. Videoconferencing is another possible use.
But that is okay for a small business who might run a “disposable” network which they don’t care about and don’t put anything on that anyone would be interested in. It is difficult to envisage why a institution of the scale and sophistication of CalPERS would find itself in this position. Especially with the risk that confidential data (from scheme beneficiaries, for example) could inadvertently end up on a network which it should not ever be put on (you, or rather they, are then depending on operational controls and these are the weakest because they are most failure prone and don’t fail-safe).
An excerpt from a typical firewall’s documentation has this to say on the subject of Exposed Hosts:
When a computer on your LAN is designated as the exposed host, it loses much of the protection of the firewall and is exposed to many exploits from the Internet. If compromised, the computer can be used to attack your network.
Whenever I implement a change at my TBTF which involves an externally facing system (and this interpretation is extremely broad; firewall changes and setting up publicly-facing services would definitely be included) are subject to a sign-off by a “White Hat” security testing specialist – hackers who will try to hack your infrastructure to test for weaknesses. One of the tests is for exposure of any internal IP address. Any internal IP address exposure is an automatic fail and must be fixed before the change can be promoted to a Live environment.
If CalPERS are not undertaking this sort of diligence and QA in managing their systems – and if we did indeed see one of their internal IP addresses it proves they are not – this is a significant governance and management capability shortcoming.
The mere fact that you can see the innermost configuration of CalPERS internal network shows that something is potentially amiss with their security. If I lived in CA and my data was being processed by CalPERS, I’d contact the appropriate state regulator and ask them to make enquiries.
If this had happened at my TBTF, it would be regarded as a serious security incident. There’d have to be an internal review and a post-mortem and a set of Lessens Learned drawn up.
As another IT professional put it:
The hiding of the internal IP addresses, as a matter practice, have become such a commonplace standard (because hiding them is a very trivial process), that security experts view it as sloppy hygiene.
Richard Smith explained why network security is often not given the attention it warrants:
Managing one’s managers is especially tricky in the IT world and it can get very dysfunctional: striking the balance between escalating the stuff that needs to be escalated and not escalating non-urgent problems just gets harder and harder, especially when non-urgent stuff suddenly gets urgent, for instance if the CEO gets a bee in his bonnet about some trendy IT boondoggle he vaguely remembers from some recent conference.
This recent NHS cyberattack is a classic instance of a better-justified sudden change of priorities, and in fact IT security is pretty much always the orphan child that suddenly claims attention. IT pros who are nonchalant about security are cruising for a bruising. But as for when and how that bruising will be delivered, it may be impossible to be specific enough to make a business case if the company culture doesn’t work that way. The perverse end result is that elementary IT security precautions get sidelined as speculative investment. This is precisely why IT security’s orphan child status will continue until it can be packaged up as a service offering, and overall we are a massive, massive distance from that Nirvana.
Maybe this little incident will get CalPERS to revisit its priorities.
_____
1 Technically, it’s called a bogon. For instance, on home computer networks, they are usually of the format 192.168.x.x
17 comments
Comments are closed.
Tally-ho!
Not too surprising since calpers hasn’t had a real Chief Information Security Officer for almost a year… something funny’s going on inside of calpers IT and no one will take the job.
You think calpers could do a better job on IT security considering they spend around $100 million a year on direct IT costs and over 20% of calpers employees work for calpers IT. Is calpers a retirement system or perhaps a safe haven for state IT workers? Have they been found out and is this the reason calpers IT management is fleeing?
In my experience, it is often the bosses, the executive class if you will, that refuse to be constrained by the network security rules that the rest of their organization must live with.
It is often the case that there is a subset of users labeled ‘execs’ whose technology goes virtually unrestrained because they feel personally affronted by even the smallest of inconveniences imposed by network security policies.
Powerful institutions are hacked, $millions are lost, it’s covered up, it’s never admitted, and it’s blamed on North Korea, the Russians and China.
I’ve removed wireless network profiles I’ve found installed on execs laptops that originated on ‘Gentleman’s Clubs‘ ‘Free’ wifi networks.
Hubris
I am stuck trying to wrap my mind around the idea of someone going to a Gentleman’s Club and having the thought “Hey, I wonder if they have Wifi?”
Poor Chris P. (Esq) — outed by his ISP.
VPN coulda fixed that. *wink*
Hope Yves don’t sic 4chan on him. :-0
Anonymous is a fickle beast. “Not Your Personal Army” is a catchphrase of 4chan’s /b/ board; any attempt to directly recruit the slabbering hivemind for some task has a better than even chance of backfiring on you, with you ending up as the target instead. It’s generally better to draw their attention to something, and hope that today they’re feeling generous and view it as a worthy target.
They’re a mob that’s just as likely to dox some harmless Tumblrite or perform an organized raid of a kids multiplayer videogame as to take up any actually worthwhile cause, like the time they hacked the Steubenville High School rape photos and made them public.
I don’t know what’s funnier, the fact that they have such wireless profiles installed, or that they felt compelled to bring and *use* their work laptops to such a place.
I mean, what are they thinking, doing work when they’re supposed to be watching porn?
“In my experience, it is often the bosses, the executive class if you will, that refuse to be constrained by the network security rules that the rest of their organization must live with”
Make that most rules, especially at a place like Calpers where the entitled, annointed and extemely inept ones lead
Hilarious. Yes, too many execs think ignoring network security rules means they’re part of the Big Swinging D*** club. (see Gentleman’s Clubs). Too bad ignoring network security rules really means they’re part of the Clueless Ditz-brain Club. Would they leave their office doors unlocked after hours to prove they don’t have to follow building security rules? Dilbert’s pointy haired boss might.
Reminds me of a story I heard on the radio, seems some robbers pulled off a multi-million dollar heist by taping a note on the office door shortly before closing;
(readers of a sensitive disposition may well wish to skip the comment below and move onto something more wholesome!)
I once was peripherally involved in an investigation which centred on an upper level exec who had tripped a system / behaviour monitoring tripwire in the building access control. He worked too late too often and there wasn’t an obvious reason why the work had to be done in the office. He’d logged on to a PC at a desk other than his own. Nothing incriminating about that, but it was inexplicable. The office cleaner had also reported a difficult to clean sticky gloop which she assumed to be spilt soft drinks or something similar at the same desk and was getting fed up of trying to get the congealed gunk off the wood effect drone worker bee’s work area (the woman who sat at the desk was a junior nobody; she was though by all accounts very easy on the eye and you could quite happily look at her all day).
The woman whose desk the messy spilt drink habit was supposed to belong to denied not following office policy and failing to clear up if she had spilt her drink. She claimed she never spilt her drinks anyway.
The office CCTV was redirected to the woman’s desk, covertly.
It was then that the mystery was resolved after company internal security reviewed the footage. The exec was indeed logging on to her PC and was actually doing genuine work-related tasks.
When finished, however, quite late into the evening, he also repaid the woman’s unknowingly kindly loan of her desk by removing his trousers and leaving a parting gift on her desk. Let’s just say that the sticky mess weren’t no spilled drink.
It takes all sorts, I thought to myself. But the real oddbods do seem to be over represented in the upper echelons of big businesses.
Oh just wow. How can I be both amazed and not amazed? Well, um, I guess it’s because I have worked in an office.
My theory about that is that the effect of money and power works as a personality amplifier. As one rises in significance, one does not become different, one just behaves more in accordance with ones nature.
Once one has at least enough* money to not having to adjust ones thinking to whatever the local convention is out of fear -> one does not Need This Job, then one becomes both an oddball to others and also capable of thinking things that the surroundings are simply not capable off, which feeds into the cycle, assuming the ideas are good ones. With power comes both the capability to enforce ones ideas directly and also to bend the local environment to ones will so one is no longer so odd. For better or worse, if nobody ever dare saying ‘No’ one ends up like Michael Jackson or Enron.
Now, the next-level problem is that the human mind is not wired for being happy or even content for longer periods so, when left to its own devices, it will ceaselessly invent problems or threats and amplify the few that actually exist to unreasonable proportions in relation to reality. This is one of the reasons why so many wealthy, very successful, people (and nations too) go from “eccentric” to “totally off the rails”.
*) “Enough” relative to the mandatory expenses (food, housing, medical, …). When one manages to have more than 2/3 of the spending being in cash and discretionary, then life improves A Lot, almost regardless of the absolute level of spending. “No New Ideas” is why “they” why want to keep us poor, fearful and struggling, to avoid a repeat of the 1960’s.
One has to design the security around the people one unfortunately must deal with.
A very smart person I know specifically engineered a CEx Interface into his building management systems – it works in almost the same as the real interface the engineers use, except it is simpler and more visually impressive, and it affects changes by updating the screen interactively while quietly queuing the CEx-issued commands for the engineer to approve or not.
If a command is not approved after some time, it expires. So there would not be a stack of random stuff sitting on the job queue for the shift engineer to sift through on Monday morning.
All this because the CEx will occasionally show off to someone, touching buttons and settings that are better left alone,”Like, ooh now I am controlling this 2.5 MW pump here from my Philippe Starck styled office”. That CEx interface saves thousands of EUR per year in on-call service fees and my friend provides it as part of the package because he do not want to be be that guy they call for support because “… his system does not work”. The CEx will never admit to having tampered, of course. So The System is broken.
Still, the moral of “Fargo” season 3 seems to be that stupid, incompetent, people are far more dangerous for everyone than the professional criminal and the hardened assassins. The, stupid, incompetent, people are as basically as unpredictable as Russian traffic.
I have searched on CEx, results not helpful? Can someone provide a translation pls?
Sorry, unclear, replace the ‘x’ with ‘O’, or whatever the particular flavour of the specific critter is. (Of course there is also now ‘CTO’ and ‘CIO’ etcetera, bricking the simple naming scheme).
One big reason Sysadmins and DBA’s don’t get any love from management and morale is usually low, low enough that nobody wants to fix simple issues like this:
A: Everything is hunky dory: “What are we paying you for?”
B: Everything is FUBAR: “What are we paying you for?”