Tim Brown, CISO of SolarWinds, on Sunburst

Show Notes

In this episode:

00:00 — Highlight Clip

02:15 — Introductions

03:02 — Sunburst Incident Overview

07:13 — Difficulties Of Handling An Incident During The Holidays

06:44 — How Tim Stayed As CISO

09:15 — Pivoting From Internal To External Facing CISO

11:24 — Organization Reporting Obligations

13:06 — Finding Help For A Large Incident

14:24 — Reaching Out To National Defenders

16:04 — Cooperating With CISA For Messaging

16:55 — Lessons And Improvements Going Forward

19:06 — Validating A Digital Supply Chain

21:03 — Assume Breach Before And After

21:32 — Sign Off

Tim Brown:

Orange Matter — https://orangematter.solarwinds.com/author/tim-brown/

LinkedIn — https://www.linkedin.com/in/tim-brown-93639a1/

Links in this episode:

SolarWinds RSA Presentation — https://www.youtube.com/watch?v=7DHb1gzF5o4

Thanks To Our Sponsors:

Heinz College CISO Certificate — https://www.heinz.cmu.edu/programs/executive-education/chief-information-security-officer-certificate

CISOWise vCISO — https://www.cisowise.com/

Follow CISOWise on all podcast apps.

Website — https://www.cisowise.com/podcast

Show Notes & Transcript — https://www.cisowise.com/podcast/001-tim-brown-on-sunburst

Transcript

[00:02:00] Introductions

[00:02:00] Earl Crane: Today, I'm joined by Tim Brown. Tim joined SolarWinds in 2017 as the vice president of security and now leads their security efforts as their chief information security officer. After the Sunburst attack in December 2020, Tim was the lead in the response and remediation efforts.

[00:02:18] Earl Crane: Tim has spoken to thousands of customers and has been instrumental in all customer remediation, support, and services. He's worked closely with the SolarWinds CEO in designing the future state of security and their "Security by Design" philosophy. His trusted advisor status has taken him from meeting with members of Congress and the Senate to the situation room in the White House. He's also an avid inventor and holds 18 patents on security related topics.

[00:02:44] Earl Crane: Tim, welcome to the program.

[00:02:46] Tim Brown: Thanks, Earl. It's great to be here.

[00:02:47] Sunburst Incident Overview

[00:02:47] Earl Crane: I really appreciate you taking the time. As many people are aware, the SolarWinds launch to the national spotlight, as suffering a significant supply chain security incident as I mentioned in December of 2020, could you give a little bit of background on what that was?

[00:03:05] Tim Brown: Yeah, absolutely. So December 12th, we were notified by FireEye that we had shipped tainted code in our products. Turned out that people had downloaded about 18,000 copies of those affected versions.

[00:03:17] Fast-forward a little bit. The tainted code was not in our source code control system. It was further down in the supply chain where it got inserted into the product, by what we now know as the Russian SVR really were the perpetrators. That's what everyone is saying from other external research.

[00:03:36] Tim Brown: So they inserted this code in there, and it lived between essentially March and June of 2020. And the threat actor shut down their command and control server in October. The really affected period was March to October.

[00:03:51] Once that was announced, the incident kind of took a life of its own. One of the things that people don't realize is that about under a hundred customers were actually impacted by this.

[00:04:04] Tim Brown: Nine agencies, this is according to public information, so it could be off, but nine agencies up to then, 90 commercial entities, could be lower, but that's a maximum amount. And it's simply for a number of reasons. A lot of people put their Orion servers behind firewalls, and they weren't on the open internet.

[00:04:25] Even if they weren't on the open internet, the threat actor had to care about you. They had to go to the next level to do something. And for many of them, they didn't, so that brings a window down from a much larger number to a more reasonable number of what was actually affected.

[00:04:41] Thank you, Tim. I really appreciate the overview. As you said, I think a lot of people could get more details. A lot of it's been very public. But what's different here is you're in the unenviable position of being at the helm of cybersecurity during one of the most public cybersecurity incidents impacting a wide number of organizations. So what was that like?

[00:05:05] Tim Brown: Yeah. So, we quickly realized that we want to get as much information to our customers first. So really it was a customer first response. And getting information out, getting people to have clear information on whether they were affected or not, working with government agencies to help get that information out.

[00:05:25] Tim Brown: The CISAs of the world, the FBI for investigation. Just working closely with a number of both, government entities across the world, Along withprivate researchers, as well. So it was absolutely an experience you'll never forget. The first, three weeks was kind of no sleep. We didn't have Christmas, didn't have New Year's. Those things didn't exist in 2020.

[00:05:48] Difficulties Of Handling An Incident During The Holidays

[00:05:48] Earl Crane: I remembered that resulted also in a lot of frustration with your customers, because they were dealing with the response in the middle of the holidays. And it's not a coincidence that a nation state would pick that timing. So being a CISO in the community, how has that been that this incident disrupted a lot of your friends and colleagues and others holidays as well? What's that been like?

[00:06:11] Tim Brown: Yeah. And that's the worst part about it, right? Is that, even those who weren't affected, their boss's boss's boss heard SolarWinds on 60 minutes or SolarWinds on CNN or SolarWinds in somewhere and said, "Hey, I don't know what this stuff does, but if we're using it, what's going on?".

[00:06:26] Tim Brown: And just the investigation that they had to do, in order to figure out was I affected or was I not, was, yeah, it was terrible for them. And you're right, that, I wasn't the only one that lost holidays. Many, many companies were trying to figure out what they owned, what they had, were they affected or not? Were they targeted or not? So, it was a difficult period for the world. And not all information was clear and correct. Just a lot of misinformation going on. So it's hard for people to understand what should they do?

[00:06:58] How Tim Stayed As CISO

[00:06:58] Earl Crane: So you missed your holiday. A lot of other people miss their holidays, but you still have your job and fire the CISO is something we hear frequently. Why are you still the CISO at SolarWinds?

[00:07:12] Tim Brown: Good question. So that was a question that during a RSA, presentation, they asked Sudhakar, who is our CEO. He basically said that, if I was going to go hire a CISO I'd hired Tim. If I would hire somebody with his credentials, his background, his knowledge.

[00:07:29] Tim Brown: I think the other part of that is some of the things that were needed during that period of time, especially and still now, are more of executive presence, the ability to communicate, the ability to talk to all levels of an organization, the ability to take ownership and move forward, the ability to get on the phone.

[00:07:51] Tim Brown: My little intro said I've talked to thousands and that's probably about right. So the role from an internal CISO quickly changed to an external facing CISO overnight.

[00:08:05] Tim Brown: And, that's one that I think everybody needs to be prepared for. They need to be able to be prepared to get on literally the first week was of course the largest governments and the largest companies of the world. Now we've switched to some of the smaller ones, but still, just a lot of people want to know what you know at a moment in time, how you go about explaining things to that level is different.

[00:08:28] And it's important you're able to do that. So throughout this process, especially in the first three months, it was simply that voice for the company, being that voice to the world, which was critical, and they absolutely needed it.

[00:08:43] When something like this happens, the more critical you are to the response, the more critical you are to the resolution, the more critical you are to getting things better, the more valuable you are to the company. I think that is really the reason why I'm still here.

[00:09:00] Pivoting From Internal To External Facing CISO

[00:09:00] Earl Crane: So I want to go back to this idea of pivoting, and you had the support of the CEO. In pivoting from being an internal facing CISO to an external facing CISO is a huge amount of trust that you put in that person, because not everyone can be out in front of the media or talking to customers or meeting with regulators or with elected officials.

[00:09:23] Tell me a little bit about what it was like making that pivot. Did you have the support of the board as well? Did they try you out first before they let you go? You became the face of the security for SolarWinds, which was effectively safeguarding the future of the company at that point.

[00:09:43] Tim Brown: Yeah. Luckily my career and my past careers had been very public facing. As you noted, I testified to Congress a few times, talked to the White House. I did a lot of different things externally.

[00:09:56] Earl Crane: And were those in response to SolarWinds or those were before SolarWinds?

[00:09:59] Tim Brown: No, those were prior lives.

[00:10:01] Earl Crane: Okay. So you had that experience you had been through.

[00:10:04] Tim Brown: Absolutely, at Dell I was out there in front of everyone. I've keynoted in front of 10,000 people at RSA. So the idea of speaking to large executives, and luckily my experience from the outside, and from prior positions that I'd held, really lent themselves to the switch.

[00:10:27] Earl Crane: That's not what you were hired for initially.

[00:10:29] Tim Brown: No, no. So, in my past life, the way I like to say it is I built a lot of security products. In SolarWinds, I switched more to operations. Because they wanted to really do an operational role and I wanted to be able to have my team, build my team and really focus in on the practical areas of the world. So I was really focused on more of the operations side and the engineering side for three years prior. But when I needed to switch back outside I had the skills. I'd been media trained for 20 years from three different companies. So when something like this came up and said, "Hey, you need to go talk to people." I was prepared for it, I guess.

[00:11:09] Organization Reporting Obligations

[00:11:09] Earl Crane: Messaging to the customers being forward with facts, those were some of the most important things, But you're uniquely also a public company. And so if I were to ask that question to the CFO, or maybe the CEO, they might have a different response because they want to get out and message to the market or to shareholders rather than customers, or maybe to regulators. How did that conversation go?

[00:11:35] We had both going on at the same time. That's why nights were lasting until 3:00 AM. We were looking at literally 10-k's going out that next day with every comma, every period, every word, every sentence, everything needed to be as accurate as possible. And the other part of it, to be able to make sure that our facts were correct, knowing enough of the details to be able to validate our facts. So making sure that those statements were correct, and they aligned with the customer, they were just different. One was for the financial markets. One was for the regulators. One was for the customers. But, I think right along the path we would have said our customers where our first priority is get them safe.

[00:12:18] Tim Brown: And then one of the things we also operated is multiple streams, right? So we have one group looking at customer information, customer focus, all those things. We had an investigative team doing a threat hunt inside the company. and we had another team doing deep forensics inside the company. And then we had basically coordination between all of those groups on a daily basis to bring everything back together and make sure that we basically had good plans for each one of the streams but very aligned.

[00:12:51] Finding Help For A Large Incident

[00:12:51] Earl Crane: You almost had the collision of all these different streams of everything together, right? You were infrastructure, government contractors, nation state, public company reporting. You had so much at once.

[00:13:05] Tim Brown: A lot of CISOs have had to deal with an incident at some point, but yours is kind of a perfect storm of a lot all at once. How did you manage that streams? And what advice would you give to CISOs dealing with how to manage theirs? Bring the right folks in to help. We did a really good session at an RSA, so that's online the YouTube video with all of our partners. Believe it or not our legal team DLA Piper has a cyber division, and they came in and acted as a good quarterback.

[00:13:33] Tim Brown: So they quarter backed a lot of the organization. Got the right folks in from a threat hunt perspective from CrowdStrike essentially to do threat hunt.

[00:13:42] Tim Brown: We need somebody with, deep expertise in building products. So we brought KPMG's forensic team in. Again, great job from those guys. Make sure you have the right folks on your call list and make sure they're available right then, so that you can have the correct program and project management. Remember you spin up 5, 10, 15 independent programs, plus all the splinter programs. So you've got to make sure you've got all of those things going on well coordinated, well organized.

[00:14:09] Reaching Out To National Defenders

[00:14:09] Earl Crane: You mentioned 15 or so different streams. You mentioned some real, A-list names here. DLA Piper and CrowdStrike, KPMG. Did you have them under retainer? Did you have to scramble to get them? How did that initial coordination go?

[00:14:23] We had them ready to go. DLA had been a partner for a while. So it's important you have at least a primary. And from that primary you can get others associated with that. So the relationship to be able to have who to call quickly. Very, very important. Then the other part of it is relationship to the outside world. The CISAs, of the world. And that's one of the other things we found a little bit later on in the investigation was hey, we need some ties into not just CISA but the national defenders around the world.

[00:14:57] Earl Crane: So, government contacts. Law enforcement contacts.

[00:15:01] Tim Brown: Absolutely. So FBI was involved very quickly. CISA was involved because they are a good voice of truth. They also have sister organizations around the world. So all the national defenders we wanted to meet with. So we brought in the Krebs Stamos Group and Chris Krebs, former director of, CISA, and Alex Stamos, who was a former CISO for Facebook. Chris had contacts around the world. It was easy for him to just send an email, pick up the phone, and we would get the right people on the phone.

[00:15:33] Tim Brown: What happened is the national defenders had no other play in the game except real information. So convincing CISA of recommending things. And then we tied very closely with their recommendations. Our recommendation matched very well, actually.

[00:15:49] Cooperating With CISA For Messaging

[00:15:49] Earl Crane: So feeding CISA your recommendations based on your understanding of platform technology capabilities, all the forensics you did, passing those to CISA to recommend you publish that and so more than just you reaching out to your customers, CISA did the outreach, and you supported by pointing to CISA's recommendations.

[00:16:09] Tim Brown: They amplified the messages for us. It wasn't like, you just call them up and say, "Hey, here's what my message is, just send it. It's a lot of work to get to them understanding "Hey, you know you said these weren't effective, how do we know?" Show us proof.

[00:16:23] It was not a rubber stamp that they'll go out and do those things, but they absolutely were a great amplifier and a really good strong partner. So if any of the CISOs here end up in these types of situation, they're a fantastic partner and tireless workers.

[00:16:40] Lessons And Improvements Going Forward

[00:16:40] Earl Crane: They're a bunch of passionate and technical people. I spent 10 years at Homeland Security. I'm so glad to hear that. So following with that theme, with, when you look at, say the NIST model, for example, you always have after response recovery. You ran through multiple work streams. You had all of these issues that you were juggling. How did you get better? Where did you grow? What did you learn? We quickly defined something called "Secure by Design". And secure by design came from a mantra from our CEO. And really that took into a number of different things. It took people. It took our infrastructure, and our technology or build environments.

[00:17:21] Tim Brown: All three of those were affected. We needed people. We need to make sure that people understand cybersecurity, to play a larger role in cybersecurity, that they support our, missions and model. People training, additional phishes, additional remediated training, creating a whole culture of security across the organization on a journey. That one's never done.

[00:17:46] Tim Brown: Infrastructure. When you have folks that are looking deeply into your environment, and you're paying them millions of dollars. They find things. You fix them, you find them, you fix them. We were working infrastructure every day with the CrowdStrike KPMG guys for the first, probably couple months. It was just, a lot of tightening down on the outside. A lot of tightening down of just infrastructure in general. And just providing less privileges for certain areas. We've implemented multi-tiered MFA. So administrators have YubiKeys now. Taking an assumed breach model from the inside. On the build side, this happened in our supply chain, this happened in our build system. So first thing was, okay, well, how do we put resilience into that more? January 25th I think was our build when we got to a hundred percent efficacy check.

[00:18:36] Tim Brown: So that means take a line of source and go through the build process, gets a product. We then install a product. We decompile the product and get it back to source. If we match everything in source code control, the libraries, the files, everything back to what we've shipped.

[00:18:51] Validating A Digital Supply Chain

[00:18:51] Earl Crane: So, because the incident actually tainted the code after it had been put into the build process, after it had been MD5 somewhere, after the certificate had been signed, and that's when it was then tainted. So what you're doing is validating the entire process forward and then backwards again.

[00:19:12] Tim Brown: Right.

[00:19:12] Earl Crane: Are you doing that with all code you ship?

[00:19:15] Tim Brown: We're doing that with all code we ship.

[00:19:17] Earl Crane: Is that something you recommend CISOs also look at?

[00:19:21] Tim Brown: Absolutely.

[00:19:22] Earl Crane: Sounds very expensive though I know there's a lot of initiatives to help with it. What advice would you give when you drop that gauntlet down of validating and securing your digital supply chain?

[00:19:34] There's some products that start you down that path. There's also things that you can do manually to make sure you're there. It is a very good step in the process, but it wasn't the only step. So other things that we've also done is made sure that everything from a check-in to code has not just a peer review, but a architect review at the same time. So what done is, is no longer just a rubber stamp. Your Kanban boards are making sure that it's really done. So simple moves, like moving everything to AWS and putting all your build processes in code allows you to say well, "Who has access to it?" Well, five people. Everything's audited everything's controlled.

[00:20:16] Tim Brown: So you put the check to make sure you still match. That's always there, the QA build. But then you also put three different pipelines in place. So you build in security pipeline. You build in a dev pipeline. You build in a lab or production type pipeline.

[00:20:31] Tim Brown: Building in three pipelines with no individual having access to all three, means that if you want somebody to affect your build again, you would have to get collusion. So again, assume breach model says, okay, we've got that there. We've got a safeguard of the two-way build. We've got a safeguard of the three-way build. .

[00:20:48] Assume Breach Before And After

[00:20:48] Earl Crane: Were you not in the assumed breach threat hunt mindset prior to Sunburst?

[00:20:53] Tim Brown: In some cases, but not as much. From the administrator perspective, absolutely. But assume breach to the max. That's going to add a lot of complexity to you. Every step of the way. Remember nobody can do anything by themselves any longer. Nobody can be responsible for two things any longer. Assume breach is not something that is a simple infrastructure change. It's not something that's simple.

[00:21:17] Sign Off

[00:21:17] Earl Crane: Well, Tim, I really appreciate you taking the time. Thank you so much for sharing your experience and wisdom here on CISOWise.

[00:21:25] Tim Brown: Absolutely, thank you.