Gridium: Hello everyone, and welcome to this conversation with Fred Gordy, Director of Cyber Security, Building & Facility Control Systems at Intelligent Buildings. He is responsible for the technology strategy and cyber security for control systems and his project portfolio includes military bases, Internet data centers, REITs and national retail chains.
My name is Millen and I’m with Gridium. Buildings use our software to fine-tune operations.
Fred and I will discuss the growth in BMS hacking attacks and the NIST-based framework for assessment and defense that Intelligent Buildings has developed. This should be a popular conversation given the interest building operators have in maintaining safe and secure building systems. And it’s a special pleasure to have Fred back on the podcast—our first repeat guest!
Fred: Thank you, Millen! I really appreciate you inviting me back. It was my pleasure to do it last time and again. You know, I was looking forward to this opportunity to kind of catch up and let you know how the world has changed quite a bit since last time we talked; it’s gotten a little bit more violent.
Gridium: Let’s start with a quick story. Tell us about the growth in BMS hacks since last we spoke, which was in November 2016 in our “Is Hacking a BMS Easy?” conversation.
Fred: Well, you know, the thing is I noticed that the other day that the last time we spoke was in 2016 and it’s really funny because at that point in time, the growth of cyber security in building control systems was stagnant at best.
But what we’ve seen is, since that time that you and I had the conversation that we did, we’ve seen a huge spike in the number of attacks on control systems; in fact, it’s over 400% increase since 2017 because during 2016 there were really no recorded events. Now, working at Intelligent Buildings and going out and doing assessments all around the United States, Canada and I just got back from Australia, what I’m seeing is these attacks are happening. They’re happening almost on a weekly basis. The thing is that’s interesting is, you know, when you see or hear about a credit card breach or a data breach, that information has to be shared; in other words, there’s government regulations around that. That information has to be shared.
A building control system, as long as it doesn’t affect personal data does not have to be shared, so therefore the general public doesn’t hear about all the attacks.
Gridium: Well, and for our audience that missed it the last time we chatted, can you share some more details about what happened with the Target payment hack and what their response should have been?
Fred: Sure! You know, one of the things that I preach over and over is having a really good instant response plan. And what an instant response plan is is it’s a lot of things, more than just, you know, “Okay, let’s get in here and get the bad guys out of the systems.” It also is about how to respond to the public.
And in the case of Target, there’s a company by the name of Fazio Mechanical. I used to work for a mechanical contractor and we would go, just like these people, and they would go and do their work orders. In this case, it was Target; so their guys would go do the work order, bring it back, and the admins would plug that information into a portal which they had access to into the Target system. Well, the thing is, everybody still to this day things that the hack came through the HVAC system because that was the first things that was said; but one of the things about an instant response plan, it has a well thought-out in advance plan of how to respond and how to react to these things.
Had Fazio and even Target had a well thought-out instant response plan, they could’ve answered some of the fodder that was hitting the “air waves” if you will, and they could’ve curtailed at least some of the brand damage. So, in the case of Fazio, I know a gentleman in the area where Fazio does business, and they are a supplier to Fazio. Well, he told me that their business—and I’m not going to quote a percentage—let’s just say it’s fallen off dramatically. And why is that? That’s because the people that were doing business with them now see Fazio Mechanical as maybe it’s not a good idea to do business with them because they’re the reason that Target got attacked. Now, Target on the other hand, was quiet for days.
I would like to point out at this point, how to show the difference between a really good instant response plan and one that’s not so much, is: most of the people here probably are not going to remember this, but the Anthem breach—do you remember that?
Gridium: No sir, what was that? I’m not familiar.
Fred: (Laughs) Okay, yeah so, not long after the Target breach, Anthem who is I believe the parent company for Blue Cross/Blue Shield, got hit with I can’t remember how many hundreds of thousands of patient records—patient information came out of there.
Gridium: Oh, yeah.
Fred: Do you remember that?
Gridium: It’s coming back to me.
Fred: Well, yeah… well the reason it’s not in the forefront of your mind is because what really impressed me about the CEO of Anthem, was he had a response ready-to-go within 12 hours and he blasted it out to all the media and to all the subscribers, and everybody got a response; whereas Target kind of sat back and held everything close to the vest. Well, that’s why you remember Target and that’s why you don’t remember Anthem. Does that make sense?
Gridium: Yes. And I hesitate to belabor the point, but this can all be very serious, including examples of some of the most sophisticated software code ever written. And I don’t think I’m exaggerating in that extent… what I’m talking about is Stuxnet. Can you summarize that story for us?
Fred: Yes I can. I’ve studied that one as well. One of the experts that actually dug into that from McAfee, they called it the first “weaponized” virus. And what I mean by that is it did more than just steal things; it actually went into the systems and destroyed equipment. And so…
Gridium: This is Iran’s SCADA data system with their centrifuges.
Fred: Exactly—Natanz, the nuclear facility. And at the time, nobody was really claiming responsibility for it, but I will say this: that the guys at McAfee who did the forensics on it, they made some very astute observations, one of them which is the code in there had very little errors.
And why that’s important is because it was a signature that whoever did this had a whole lot of prior knowledge to the Siemens PLCs and controllers. So, still nobody has actually come out and said, you know, “We did it.” But it’s pretty well accepted that the US and the Israelis were behind it. But however, what it did was it introduced this “weaponized code” meaning, “Hey, we can attack something. We can actually do physical damage.”
Now, one of the things that I hear and I’ve done work with a lot of government agencies, both in the US and in Canada in military facilities. And the first thing that I hear is, “Well, we’re totally air-gapped.” And if you don’t know what “air gap” means, it just means that it’s not connected to the Internet; it’s not connected to any other system. It’s totally isolated, so therefore I should be okay. Well, Natanz was exactly that—it was totally air gapped, there was no Internet connection or anything like that. So, in this particular case, the theory of how it got in was us all being technical and just like the engineers at Natanz, the theory is that some entity was dropping USB or thumb drives in the parking lot. Well, stop and think about it. If you saw a thumb drive, what would you do Millen?
Gridium: Well, I actually have seen this story play out before, so I know not to go exploring. But yeah… most people of course are going to plug it right in and see if they can use it, if it’s empty, if it’s worthwhile to keep.
Fred: Exactly. And that’s the theory behind what happened: is one of the engineers saw that in the parking lot and they picked it up and they took it inside and they plugged it in. So, once the virus was in there, it opened up. It probably said here’s pictures of dogs or something, or who knows what.
Gridium: Right.
Fred: But it unleashed a virus and what it did was, if you’ve seen the movie Ocean’s Eleven, they tapped into the video system and the guards were looking at it and it looked like everything was fine, but guys were running up and down the hall… that’s kind of what Stuxnet did, is it sat back and watched and it recorded a good operation, or a “correct” operation, let’s put it that way.
And then when it decided to unleash itself, to the operators who are sitting in a remote control room, they saw that the system was running completely fine, but what was actually happening were the centrifuges were spinning up and wide open and then they would slam on the breaks and go the other way and they just began to destroy themselves. S
o, the bigger picture here is hackers are curious beasts and they also are looking for the path of least resistance. So, what this kind of awoke in the hacker community is “Hey, wait a minute. These systems are not watched over quite as tight as an IT system.” And the more they begin to explore, then they begin to see things like, “Well, if I need to know anything about a UPS or an air handler or a York chiller or whatever, I can go online and download everything I need.” And then you had the Integrators—who I used to be one—and we would create remote access into these systems, but remote access only really equated to a public IP and therefore, to get into the system was not that hard. And then once you got to the system, all you had to do was look up the username and passwords, or use crackers on the system—password crackers. Because the other thing about these systems in like an IT system, you know if you go to your bank’s website and you fat finger your password three times or five times, what happens? It locks you out.
Well, a password cracker, what it does is it just bam, bam, bam, bam. And it just keeps hitting the system over and over until finally it figures out the password, and it has literally a library of common-used passwords. So, if you don’t have little things like that, like the auto lockout, then once a bad guy finds your system, he can just unleash a password cracker on it and after a while, he’ll get in, so it’s not that hard.
Gridium: Given all of the growth in hacking activity, Intelligent Buildings has developed a NIST-based framework for defense. What’s that look like?
Fred: Well, what we did was you know, NIST foundationally is one of the most well thought-out platforms or frameworks that’s out there; it’s accepted widely throughout, even I’ve found throughout the world in some of the places I go. I mean, that’s foundationally what Australia and Canada even look at it. But what the NIST framework does is there’s five very basic what they call functions, and they’re eloquently simple to understand. The five are: identify, detect, protect, respond, recover. And so what we did was we started working with the NIST framework and to be quite honest though, it’s not an exact fit because building control systems have their own nuances and quite frankly, they don’t follow any kind of standard policy or procedure, obviously.
So, what we did was we took foundationally that if you think of the 2x4s and the sheetrock of the house, that’s what we did. We took that and that’s what we use, and then we layered on top of it the rest of the house: things that are specific to building control systems: BACnet protocols, typical installations that we’ve experienced over the years because inside of our company, there’s more than just myself. There are a lot of old integrators, if you will, that have designed and implemented systems over the years and they are the ones that know, as myself, of how to protect and destroy a control system.
So anyways, long story short, is from there we created BCS-CAMP which is Building Cyber Security. BCS-CAMP takes those foundational components of NIST and has built it into a multi-pronged approach: one being what we call BSET, which is not unlike CSET, which if you look at ICS CERT—which is Industrial Control System Computer Emergency Response Team—CSET is Cybersecurity Evaluation Tool. Well, BSET is Building Cyber Security Evaluation Tool. So, it’s taking those things that we’ve learned, it’s taking the things from CSET—CSET is in the public domain, but we’ve layered on top of that a series of questions, because that’s what CSET does. It asks you questions so that you can get a pulse read of where you are, okay? But that’s not the end.
So, the next step to that is once you establish—and we’ve got a grading scale that uses backend algorithms, not unlike what CSET is. So, it’s not like we go do an evaluation on your system and we guess on a score. When we run through BSET, you will actually have a score and I will tell you that the industry right now is running Ds and Fs. We have run into some Cs. There’s no, well hardly… we have run into a B, which that was a military base, thankfully. (Laughs) But, as a rule, the commercial real estate is running Ds and in some cases F.
But anyway, so we do that. The next step is we actually go in and look at the configuration of your systems, because one of the things in this is you know, modern configurations. Nobody’s been doing that for control systems, so we’ll go in and we’ll run scans against the network.
We’ll pull the files from the control systems, look at the configurations. And you know, like I said earlier, where we were talking about the auto lock-out, we’ll look for… we have distinct things that we look at in the configuration files: things like are strong passwords enabled, is auto lock-out enabled. If it is enabled, what is the threshold of the number of attempts? What is the lock-out time?
We look at things like if we go in and we see that there are two users but we know that ten people work in facility engineering, then we record the number of people that work in facility engineering and we see that there’s two users and we ask, you know, we ask the questions. “How many people actually use the system?” And they say, “Well, 8 of us do.” Well, what that tells us right out the bat is there are people that are using the same user, and that’s not a best practice.
But anyway, it goes on and on. There’s a lot of things that we check and we, like I said, scan the networks and we look for public access; you know, how would the public get into it. And then we also go on what we call hunting and phishing exercises. And our phishing exercises are not like what you get in a company where IT sends you a fake phishing email that says, “You won something from Amazon.”
Our phishing is things that would trip up a facility guy; you know, like one of my favorites—we’ve retired it so I’ll give it away—(Laughs) is you know, in building services, over time air. The tenants are used to being able to go into a portal and say, you know, I would like to request air until 9 o’clock at night and of course they get billed for it and that kind of thing.
Well, we’ve sent out phishing emails to the facility guy saying, we’ll look on websites and find people’s names that are actually in the building and then we’ll send the phishing email. And it’ll say something like, “Hey, this link was provided to me for overtime air. I can’t get it working. Can you click it and check it and see if it’s working for you?”
Gridium: Mmhmm.
Fred: And we do malform emails enough that if you’re looking, you can tell that it’s a phishing expedition. So…
Gridium: Yup.
Fred: And I’ve run my mouth way too much, I’m sorry. (Laughs)
Gridium: No, it’s fascinating. As you’re describing the BCS-CAMP defense approach that IB’s developed, I’m curious if it’s one-size fits all?
Fred: At this point in time, you know… if you’d have asked me that question 18 months ago, I would’ve said, “No.” But here’s the thing: experience has taught us because we’ve done now, I think last count almost 2,000 assessments in 18 months; that tells you how busy I am. And it’s not just me, obviously. I can’t cover… I’ve got a team of guys that we go out and do this.
But anyways, so I would still say there’s a degree of it that one size does not fit all; however, we are in the, you know, when you think about crawl/walk/run, the industry as a whole is still in the “crawl” mode. And even, again not mentioning names, but there’s some large organizations that I’ve worked with that have high levels of criticality. But they need to do the same things just to get started that some of the other companies that don’t have nearly the criticality. Does that make sense?
Gridium: Yes.
Fred: In other words, just getting those basic things like vendor policy—none of them have vendor policy. Like, for instance, if you have let’s say a company that has 5 employees that come to service your building from one company, one of the things that we teach is that you should have vendor policy that says, “If they let go an employee, they need to notify you within you know, 8-12 hours that that employee is no longer there.” The reason being is psychologically, if you think about it, if somebody gets fired, they’re going to—and this is not me. Studies have proven that within 24-hours they’re going to do something stupid if they’re going to do something.
Gridium: Mmhmm.
Fred: So, just little things like that, the vendor has more control of your system than you do. They have control of remote access, you don’t. They have control over all the usernames and passwords; well, if you were hiring in an IT firm or you bought servers, would you have DELL administer your users?
Gridium: No.
Fred: No. So, those are the very basic things that everybody needs to do. Now, then when you get into… I do a lot of policy review. I have one company that said—and rightfully so, I was really actually glad to hear them say this, the head guy—he said, “Look, we need policy, but I don’t want 100-pages. I want 2-pages because we need to introduce this gradually.” And I said, “I hear you because if you stop and think about facility guys, what are facility guys? They’re fixers. They figure out how to make things work. They figure out how to get around things, right?”
Well, if you overwhelm them, they’re not really going to follow what you’re asking them to do. You go to another company where they do have a higher degree of policy, then yeah you can get a little more aggressive with your policy there. But that’s where the one-size does not fit all…
Gridium: Sure.
Fred: … you’ve got to figure that out going in.
Gridium: Let me ask an obvious question: why have buildings been asking for this service?
Fred: That’s a really good question. Two reasons: one is there’s the obvious, which is brand damage. Just like in the case of Fazio Mechanical, that’s a particular instance where brand damage has done monetary damage to the company and that’s not something that you can actually really… you know, people said, “Hey, can you help us figure out what our brand damage would be.” And I said, you know, “I can’t.” You can’t really figure that out because you don’t know, you know, what the repercussions are going to be and at the point of brand damage. So, anyway, the end user—the person that owns the building or owns the data center or has the responsibility of protecting the military base and so on and so forth—has awoken to the fact that they are a target now. That’s one reason.
The other reason is I’ve actually seen that people are using it as a marketing tool. They’re saying, “Come to my building because we do our due diligence and so you’re safe and you know, we’re going to protect the infrastructure of the building.” So, think about it like, you know, in the day when a building had installed security cameras and access control for their tenants—well, not everybody was doing that. Well, when companies or buildings started installing that kind of stuff, then that kind of drew a certain clientele, right?
Gridium: Yup.
Fred: So, that’s the two ends of the piece of the puzzle. But the bottom line to it is, the people like the job I used to have of being an integrator, they are not driving this; the manufacturers are not driving this. It’s the end user who’s driving it and it’s all business—bottom line.
Gridium: Once you’ve got the audit done, what happens next? Can you walk us through a little bit more of the high-level steps of a project?
Fred: Sure. So, one thing I didn’t… I’ve got to kind of address before we go there that I didn’t cover earlier is, when we’re doing the assessment it’s not all about a bad guy; it’s also the operational risk piece of it. And what I mean by that is, now you have these building owners and everybody who’s saying, “Hey, we’ve got to get Cybersecure! Oh, and by the way, we have an IT department and we’re just going to inject them into the process.”
Well, what you can do is you can actually introduce operational risk, and what I mean by that is policy and procedure that IT use can, in some cases actually go in and screw up a system. So, when we’re doing our assessment, we’re looking at the cyber risk but we’re also looking at the operational risk.
Now, to your question is, once we’ve gathered all our information and we’ve done our site visit and we pull all the data, we go back and do an analysis on it, some of which as I described earlier is through you know, algorithms and that type of thing to where, you know, it’s easy enough; you get the score on that.
But then when you get into the intangibles like, you know, how’s the system being used, where is it located, what are the people? Because we interview people and we try to get a feel for what their concept of cybersecurity and whether or not it’s important. Those are those intangibles, so we have to take that information all back and we’ll create a report and we’ll score. Now, we do use some metrics; we try to make this as scientific as possible. We will score and produce a report that is focused around the BCET area, the identification of what’s on your network.
I’ve got to tell you a really quick story: I went and the last assessment I did, the guy—I mean, he was very helpful. He was a vendor and he said, “I don’t have a problem with you scanning my network, but there’s only four things there.”
Gridium: (Laughs)
Fred: And I said, “Okay. Well, that’s fine.” And I said, “We’re here for verification.” After I ran the scan, I found 32 devices.
Gridium: Yup.
Fred: And he just looked at me like, “What?” (Laughs) And I said, that’s what happens with these control systems: is over the years, people just plug things in. There was a Raspberry Pi in this network. Now…
Gridium: (Laughs)
Fred: …this was probably there for innocent reasons, but do you remember a few weeks ago what happened with NASA?
Gridium: Now wasn’t that also a thumb drive story or am I confusing the two together?
Fred: Yeah, well it was a Raspberry Pi. Somebody had just innocently plugged in a Raspberry Pi—he was probably doing some little project box kind of thing…
Gridium: Another project, yup.
Fred: …yeah, yeah. But then a bad guy found it and that’s how they got through to the network. So, I mean… I don’t think that was the case here; I have no validation or whatever…
Gridium: Right.
Fred: …around that, but I mean it was interesting that literally a week after the whole NASA thing I find a Raspberry Pi on this network.
Gridium: Mmhmm.
Fred: But anyways, so after we take those scans, we identify, we give them a full inventory of what’s connected to their systems. We also, inside of that, we’re identifying operating systems, so obviously if you have a Windows say, Windows XP—which a lot of older systems are running that, we score that.
I mean, that’s a definitive scoring, and we bubble it up into an executive summary and then a detailed summary so that if somebody does want to dig into it, which hopefully they will. And then, beside each one of the findings is a remediation recommendation. Now, it depends on the value of the report that you’re getting. If you’re just wanting to get a temperature read of where you are, we’re going to give you a reading of where you are and we’re going to give you some recommendations. If you want to actually do outlines and projects that you ought to attack, we also have a tool to where we can score those based on criticality, risk, lift and money. So, by identifying those, it actually helps bubble up which projects you need to identify first.
Gridium: I’m only a quarter joking with my next question.
Fred: (Laughs)
Gridium: If my BMS is bricked by a hacker with a Bitcoin ransom, should I pay it?
Fred: No. And here’s the thing is—going back to processes—if you’re actually doing your processes like backing up your systems to a remote location, a control system… they’re not going to get to the supervisory controllers because ransomware doesn’t work out there, but it does work on the Windows side.
So, think about it like this, is: if you get hit with ransomware, well first off you more than likely were doing something with your frontend application server that you weren’t supposed to be doing, like checking email or checking Facebook—and I see that all the time. If I can, a side note to that is, would you take your email server and go look at Facebook on it? No. IT’s not going to let you. So, but the facility staff has gotten used to using that server as just another PC, so that’s what usually happens. Anyway, if the system is being backed up on a regular basis and you have a really strenuous…
Gridium: Thorough?
Fred: …auditable—yeah, thorough backup, what’ll happen is—and we’ve actually had one customer that was doing the right thing just recently. Was, they got hit with ransomware and they were back up and running in less than two hours, and they never paid a dime. And that’s because they were able to get another machine—I mean, you know, machines are cheap now. Got another machine, took their backup, reinstalled the backup on the frontend. The backup was only a few days old, and they were up and running in less than two hours.
Gridium: That’s great.
Fred: Well, and the other reason I say, “Do not pay it” is because you don’t, even when they unlock your files, you don’t know what they left behind.
Gridium: Yeah. Speaking of which, and as we wrap up here, do you have any basic best practices building operators should perform?
Fred: Absolutely. So, the first and foremost one is, if your system is on a public IP, get it off a public IP. There are other solutions out there that you can get that are not expensive that give you remote access. Right now, I mean we’ve all probably heard of Shodan and Censys.
Those two search engines are cataloguing everything that’s connected all the time. Just by getting it behind a firewall and off a public IP, you now become blind to those two search engines and make it harder for you to find yourself. The next thing is just what we were just talking about: is that frontend application server is a frontend application server. It is not a workstation; it is not your personal PC.
Move it off the desk. Unplug the keyboard, mouse and monitor and don’t let anybody do anything to that other than if you need to access it, the whole thing is with all of these different manufacturers is they have web interface. You access it through a web interface.
The next thing is, is everybody must have a unique username and password. The reason being is—and you need to restrict access based on role. There’s a thing in this called least privileges and what it means is you only get those privileges that you need to perform the job that you’re supposed to do. So, everybody needs a unique username and password with their specific roles and duties; that way, if somebody leaves, you can kill that user.
If somebody does something they shouldn’t have done, you have an audit trail to figure out who it was that did it. And then, you know, if you just do those three things, you’ll be so far ahead of everybody else. I mean, you know the whole story of the two guys running from the bear, right?
Gridium: I think so. Hopefully, yes. (Laughs) Yeah, I do, but I’ll let you tell it.
Fred: So, think of it in terms of…
Gridium: You don’t have to outrun the bear, basically.
Fred: Exactly! You just have to outrun the other guy. Well, remember I said, bad guys are looking for the path of least resistance. Just like when somebody’s driving through a neighborhood: they’re looking for the house that has a sign on it that says “security system”.
Well, I’m not going there, even if there’s not a security system—they’re not going to chance it. Well, not exactly the same thing, but if the world is blinded to your presence, you can’t find it. If you’re not misusing that system, you can’t get ransomware; I don’t want to say can’t, but the probability of you getting it goes way, way down.
Gridium: Right.
Fred: And then you’re controlling your users: both your employees and your vendors.
Gridium: Okay, Fred. Thanks for sharing all of this with us. The work you’re doing is important. What’s the best way for folks to get in touch with you?
Fred: They can absolutely go to our website, which is IntelligentBuildings.com. We’ve got a cyber security section in there and if you click that, more than likely, I’m going to get the notification and we’ll be happy to talk to you. Of course, you can hit me up at Fred.Gordy@IntelligentBuildings.com. And I do try to answer everybody that sends me something; it’s getting a little harder, but I will do my best. (Laughs)
Gridium: Roger that. Thank you Fred for taking the time. This has been really interesting.
Fred: Well thank you Millen, again. It’s always a pleasure. Thank you for having me back.
