Failure, Culture and Keeping Your Sanity in Cybersecurity

Show Notes

In this episode:

00:00 — Welcome

02:19 — Alan Levine On His Single Biggest Technology Failure

05:25 — Tim Brown On Advice For CISOs Potentially Facing A Large Incident

07:01 — Yiannis Pavlosoglou On Shortcomings Of No Resilience

09:39 — Mike Wilkes On Having A Social Contract With Your Team

10:56 — Brandon Hines On Example Of A New CISO Misaligned With The Organization

13:36 — Mike Wilkes On How Has Marvel Maintained Security Standards For So Long?

15:14 — Nick Shevelyov On Advice To A New CISO Dealing With Greater Responsibilities

17:10 — Brent Maher On What Works In Engaging Business Units With Strategy

18:57 — Joe Robinson On Not Taking Business Decisions Personally

19:55 — Outro

Alan Levine:

LinkedIn — https://www.linkedin.com/in/alan-levine-43a226a

CISO Street — https://www.cisostreet.com/alan-levine/

Tim Brown:

Orange Matter — https://orangematter.solarwinds.com/author/tim-brown/

LinkedIn — https://www.linkedin.com/in/tim-brown-93639a1/

Yiannis Pavlosoglou:

LinkedIn — https://uk.linkedin.com/in/yiannisp

Kiberna — https://www.kiberna.com

Mike Wilkes:

LinkedIn — https://www.linkedin.com/in/eclectiqus

Brandon Hines:

LinkedIn — https://www.linkedin.com/in/brandonjhines

Nick Shevelyov:

Website — https://www.nickshevelyov.com/

Cyber War... And Peace — https://www.nickshevelyov.com/the-book

Brent Maher:

LinkedIn — https://www.linkedin.com/in/ciso-brentmaher

Joe Robinson:

High Peaks Solutions — https://highpeakssolutions.com/

Thanks To Our Sponsors:

Heinz College CISO Certificate — https://www.heinz.cmu.edu/programs/executive-education/chief-information-security-officer-certificate

CISOWise vCISO — https://www.cisowise.com/

Heinz College:

https://www.facebook.com/heinzcollege

https://www.linkedin.com/school/carnegie-mellon-university---h.-john-heinz-iii-college/

Carnegie Mellon:

https://www.linkedin.com/school/carnegie-mellon-university

https://www.facebook.com/carnegiemellonu

Follow CISOWise on all podcast apps.

Website — https://www.cisowise.com/podcast

Show Notes & Transcript — https://www.cisowise.com/podcast/007-failure-culture-and-keeping-your-sanity-in-cybersecurity

Transcript

[00:01:27] Episode Introduction

[00:01:27] Earl Crane: Welcome to the CISOWise podcast, holiday edition. This is your host Earl Crane, and I'm excited to bring you a collection of stories that we couldn't fit into our regular podcast episodes. This is broken into three acts on failure, culture and keeping your sanity in cybersecurity.

[00:01:46] Earl Crane: Act one, on failure. We will hear from Alan Levine with Alcoa, Tim Brown with SolarWinds, and Yiannis Pavlasoglou with UBS, on the failures they have seen and experienced, and the advice they have from their lessons learned.

[00:02:03] Earl Crane: Act two, on culture. We will hear from Mike Wilkes with Marvel Studios, and Brandon Hines with Dimensional Fund Advisors, on the importance of building a cybersecurity culture with your team and the failures they've seen when the culture is misaligned.

[00:02:21] Earl Crane: Act three, on sanity. We will hear from Nick Shevelyov with Silicon Valley Bank, Brent Maher with Johnson Financial Group, and Joe Robinson with Fifth Third Bank, about when you're in the role of CISO, what do you need to do to keep yourself sane and healthy? And how to think about your responsibilities?

[00:02:42] Act 1: Failures[00:02:43] Alan Levine On His Single Biggest Technology Failure

[00:02:43]

[00:02:43] Earl Crane: You've probably gotten a really good sense of detecting from a vendor or a type of technology if it'll work or if it's not quite there yet. How do you think about some of the smart tech as it's coming in and say, "this is something I'm going to take a risk on".

[00:02:59] Alan Levine: You can march organization down the wrong road a couple of times, and people will suffer you. You can misspend a couple of times as a CISO and people will understand. Technology is difficult. The threats and risks are a moving target, but over time you finally have to get it right.

[00:03:16] Alan Levine: And the way that I learned to get it right was to do as much research as I could with my teams. I made sure that my teams each assigned people that specialized in research. I think it's a mistake to just rely on the gardeners and foresters.

[00:03:31] Alan Levine: I think they have value, but I think it's a big mistake if you just rely on them and say "boy, as long as I picked something in the top right quadrant I'm a genius". Invariably, it's a matter of will that solution really work? Will it really work as advertised in your environment, given your conditions? And then most importantly, what happens when it doesn't work?

[00:03:51] Alan Levine: I had a fair amount of buyer's remorse as a technologist, as a CISO. That IPS story that I told earlier, I ended up not buying that. We did at one point buy a new proxy server, which promised SSL tear down. So one of our issues, I think it's a persistent issue for cybersecurity folks, is that on the one hand you're doing the good deed by encrypting, including for example, enforcing the rules that you're only going to have your users interacting with secure sites.

[00:04:19] Alan Levine: However, once all that encryption is in place, it makes it much more difficult to see what's exiting your organization including exfil, right? And so this particular proxy vendor came to us and said, what they specialized in was SSL tear down. They could on the fly decrypt, give you a snapshot of what was in that file, reencrypt, and send it on it's way.

[00:04:42] Alan Levine: And there would be minimal disruption to network traffic. We deployed it on a Monday. On Tuesday, I got a call from one of our shipping organizations in the United States. I'm not sure, I'm from Brooklyn, New York and I'm not sure I've ever heard that many foul words in one phone call because everything they were doing was encrypted because we told them it had to be encrypted.

[00:05:04] Alan Levine: And the SSL tear down was not taking milliseconds or even minutes. It was taking hours. And in some cases was failing. So it would decrypt just fine. It would not reencrypt and because we required them to use HTTPS, guess what? The entire transaction failed. Because of that the shipping department couldn't ship. And because of that, we were losing money.

[00:05:26] And so, I think it took two weeks to undeploy that, to remove it from our environment. It was the single biggest technology disappointment of my entire career. I was fed a line of bull by the supplier. It was cutting edge technology and I took a gamble and I lost the bet.

[00:05:49] Tim Brown On Advice For CISOs Potentially Facing A Large Incident

[00:05:49] Earl Crane: What advice would you give to a CISO when they either have a fear of a massive breach or that they are needing to prepare for it? What was something that you thought was one of the most useful things you did in regards to dealing with this breach?

[00:06:06] Tim Brown: Yeah, I guess, a lot of it goes back to understand what your customer is. That's the big thing. Communicate to them with what you can and as often as possible, so that they have enough information to move forward. So the more sharing, the better you are in the environment. Absolutely be prepared for small and large incidents and test your processes.

[00:06:28] Tim Brown: Make sure you do have the right people on speed dial, make sure that you test that process. The model of response is not just the technical response. It's not just a security response.

[00:06:40] Tim Brown: There's a response that takes apart every part of your organization. So, it's important, you test that. It's important that you model that support and that you're ready for that.

[00:06:50] Earl Crane: So, to prepare for it, make sure that you test it, make sure that you have plan.

[00:06:54] Earl Crane: So, threat modeling all the different types of impacts and actors from your legal to your public relations, and elements. Then trying to figure out how would we do, and testing those.

[00:07:08] Tim Brown: Develop your community at the same. Develop your internal, right? So, make sure you know the right players in the middle. Right. Make sure that you are connected to all of the people that are part of your response plan. That way you're not trying to develop relationships at that last minute.

[00:07:25] Yiannis Pavlosoglou On Shortcomings Of No Resilience

[00:07:25] Earl Crane: I'm wondering if you could go back in your archives and think about situations where you saw the lack of that resiliency and the pain and the consequences. I'll give you an example is, I was with an organization and we suffered a DDoS attack and it was a significantly large one to the point that it overpowered the initial thresholds we had of our procedures put in place for DDoS, and it triggered a higher level of protection that resulted in a greater cost.

[00:07:53] Earl Crane: And it got into the millions of dollars that we were billed for in our DDoS protection. And ultimately the systems that we had, though we survived the DDoS attack, what was not resilient was the processes to handle the financial impacts and consequences of that event. So,maybe you can share some examples of when things were not as resilient as they needed to be.

[00:08:22] Yiannis Pavlosoglou: Yes, organizations are not often built by processes that tend to have the ability to absorb disruption.

[00:08:29] Yiannis Pavlosoglou: So, if you're building a process within an organization, not a side process, a generic process, that is able to adapt and evolve and become better, then it's very likely that that same process has been designed with the ability to absorb disruption because it has fail-safes in it.

[00:08:50] Yiannis Pavlosoglou: Examples of those fail-safes to just get very specific. Do we have within that process the Four Eye principle applied when a decision is being made. That is key.

[00:09:02] Earl Crane: So four eyeballs, I think of that as dual control. You're saying four eyes, two persons, four eyeballs. Go on, so you had examples of Four Eye principles.

[00:09:11] Yiannis Pavlosoglou: So if we have a process that has the ability for two people to be checking a key decision point, the Four Eye principle, if we have a process that is able to improve over time, that gives us the ability to make sure that when a disruption becomes part of that process we're able to react and potentially absorb that disruption.

[00:09:35] Act 2: Culture

[00:09:35] Earl Crane: Act two, culture. Mike Wilkes with Marvel Comics will share with us on the importance of having a social contract with your team, and then Brandon Hines with Dimensional Fund Advisors will share with us the misalignments he's seen between culture and a new CISO. Finally we'll wrap up with Mike Wilkes again, on how Marvel Studios has maintained its standards of security for so long.

[00:10:03] Mike Wilkes On Having A Social Contract With Your Team

[00:10:03] Mike Wilkes: If I'm paying someone and we have a business contract, time and materials for money, time and attention for money. I think I've failed. But if I have a social contract with my team, then that's a win because if I call them at four in the morning, they're gonna pick up and they can call me at four in the morning and I'll pick up. And so I think if your relationship with your team or with your colleagues has devolved to a business contract and you've lost that social contract, then you're in trouble.

[00:10:32] Earl Crane: I think that's a great point to take a health check of your team and ask yourself, as a CISO, what's the strength of my social contract? Versus what's the strength of my business contract? That's probably something we don't do enough.

[00:10:44] Earl Crane: And I think maybe one of the traits that you see of a leader that does highlight that social contract is someone that will ask, how are you doing? how's your family? and be able to build from there.

[00:10:54] Mike Wilkes: Yep. I think that's been an important part of the teams that I've built. Obviously you don't want that knowledge walking out the door after just six months of training and investments, in hiring that person.

[00:11:03] Mike Wilkes: But like I said, growing the talent is sometimes easier. I can teach people on my InfoSec team, but I can't teach them to be hungry and curious and have an appetite to learn.

[00:11:13] Earl Crane: Find the hunger first, and I can teach you security, but I can't teach you passion.

[00:11:20] Brandon Hines On Example Of A New CISO Misaligned With The Organization

[00:11:20] Earl Crane: Do you have any examples? Maybe from earlier on, where you've seen individuals who maybe over-focused on certain areas and they needed to course correct.

[00:11:31] Brandon Hines: Yeah, I have witnessed a CISO hired into an organization at one point and they did their own internal assessment and I think they brought a lot of their experience with them to an organization. Which has always important to bring your experience.

[00:11:48] Brandon Hines: But it's a new organization. It's a new set of risks and they escalated all the way up the chain all the way up to the CEO all of these risks, and they thought that the world was going to cave in and it didn't play to the narrative. Maybe it was true. Maybe it wasn't true. You never know, unless if you're the one in that situation.

[00:12:07] Brandon Hines: But he didn't last long, the world didn't cave in. So, maybe he was misaligned with the organization.

[00:12:15] Earl Crane: So it goes back even to culture. That might've been a CISO that wanted to come in and be a savior. They said that they saw all these problems, that they were going to come in and fix them. And that hit the pragmatic reality of an organization that knew it from the culture and said, yeah, we don't quite believe everything you're saying here. And that they hadn't established the trust.

[00:12:36] Brandon Hines: That's correct. And I see this all the time with folks, even in IT, where they wanted to escalate. How do you know that you're aligning with the real business risk? When you get into the technology and we've got so many technology vendors that are really bad at this, they've got all of these dials and widgets and buttons that you can press within the software and you can do all of this stuff.

[00:12:57] Brandon Hines: And that's an entire career just to know all of those buttons and widgets. And you go through all their training and their documentation and they're training people to be able to address all sorts of types of risks. But unless if you can translate that to what the business process is and what the business need is you have people who start escalating things left and right, saying we must do this, we must do this, we must do this.

[00:13:21] Brandon Hines: And that might be the gnome you have in the front yard as an ornament, and it might be a part of the crown jewels of the business. And unless if you know the difference between those, you're not going to have a very productive conversation between those folks in technology and those folks in business.

[00:13:36] You're just going to set yourself up for a lot of conflict that you have to manage.

[00:13:40] Earl Crane: Conflict and then exhaustion.

[00:13:42] Brandon Hines: Yeah. And especially when you're starting a function, anything that is not contributing to forward progress is pulling you back. You have to be always forward focused. And so anytime you can help get alignment and get people moving forward, that'll just put you in a much better position.

[00:14:00] Mike Wilkes On How Has Marvel Maintained Security Standards For So Long?

[00:14:00] Earl Crane: So do you think that Marvel may have been a little bit ahead of its time in data leak prevention, data leak security? I was at Marvel for two years and this had been part of the culture and history of the organization. I remember coming in, and interviewing people and asking them, what is Marvel good at and what shouldn't we be doing?

[00:14:19] Mike Wilkes: What should we be outsourcing? from the security and infrastructure point of view. Because I was working in dev ops, and also enterprise architecture, and InfoSec. And wearing these different hats gave me a reason to talk to all sorts of people in the organization. And a lot of that knowledge of what makes Marvel special is not coded anywhere, not written down. And so I wanted to try to capture that and try to understand what are some of the things that it's inherent in the DNA of the organization and passed on from person to person when they joined? Because that's a sort of tacit knowledge transfer that just comes from working there. So, yeah, they had a lot of, deeply rooted paranoia that was there for various reasons, potentially going all the way back to the founding of the company, and Stan Lee, and the competition for storytelling and for artwork.

[00:15:05] Act 3: Philosophy For A New CISO

[00:15:05] Earl Crane: Act three, philosophy. Nick Shevelyov with Silicon Valley Bank will provide his advice for a new CISO, and how to deal with their increased responsibilities. Then, Brent Maher with Johnson Financial Group will share with us his insights on how he engaged with business units to keep his sanity, and drive his program forward. And lastly, Joe Robinson with Fifth Third Bank will share with us the insights he gained in not taking business decisions personally.

[00:15:38] Nick Shevelyov On Advice To A New CISO Dealing With Greater Responsibilities

[00:15:38] Earl Crane: Let's go back to Marcus Aurelius, the principle of knowing yourself. Because, certainly we'll hear people who are still relatively new, if you if you could take a moment and talk to a new CISO or a CISO that's rising into a position. They just got this position, and so they may have imposter syndrome. What advice would you give to them? Who's now been given this greater responsibility and their first time in that situation.

[00:16:06] Nick Shevelyov: Say, take a deep breath. Think about the long game. Build relationships, both in security and outside. The CIO should be your best friend. You should be able to have creative tension and have good, sound discussions about needs. Let's take a look at your budget and think about what portion of your budget would you give to technology to fix problems at their root before they get to you?

[00:16:29] Nick Shevelyov: An example is, are your developers shifting left and thinking about security and privacy by design? If the CIO doesn't have the budget, help them get that budget or give them a portion of your budget, right? You will be solving problems at their root before they get to you and manifest themselves in vulnerabilities.

[00:16:48] Nick Shevelyov: And so surround yourself with great people who are passionate about their mission. Have a sense of curiosity, continuous learning, and improvement. Build relationships in technology, in legal and compliance. They're all ultimately stakeholders as well. And think about long-term investments that will play out. And source talent where you can get it. Earlier in my career I thought, well you have to start off in technology to be successful in security.

[00:17:18] Nick Shevelyov: You know, lo and behold, sourcing people from lots of different domains who are interested, passionate, want to learn, show an ability to learn and apply that. You can build yourself a stronger, diverse workforce that's achieving outcomes that you're looking for.

[00:17:35] Brent Maher On What Works In Engaging Business Units With Strategy

[00:17:35] Earl Crane: Take just one component of that strategy, and the way that you engage with business units to connect them in with your cybersecurity strategy. What have you seen that works when you want to engage a business unit to work with you on your strategy?

[00:17:49] Brent Maher: So, building a strategy in a silo then coming out and sort of, tacking it on the wall and telling business partners what they will and won't be doing to support this, is not a go-forward strategy. So, pulling them in into the development and making sure that their purpose is being met as well as yours, or are you in some way supporting their mission? As you're developing the strategy, I think is really important.

[00:18:14] Brent Maher: I think understanding the tempo of how an organization gets things done. So really making sure that the strategy is being developed in consideration of the budget planning process. How the finance department sees capital versus expense.

[00:18:30] Brent Maher: And what kind of projects are you trying to get done to make sure that the spending will be available and your consideratory of the financial strategies of the organization?

[00:18:40] Brent Maher: Definitely you have more, kind of, forward-leaning customer facing entities that really, the sales side of the house that also needs alignment in terms of making sure that their mission is being met through the realization of your strategies.

[00:18:54] You got to perform a stakeholder analysis and understand what that stakeholder landscape looks like, and engage them, I don't want to say often through the process, but you need to engage them while it's being developed to give them an inside view of where you're leaning in and get reads from them, and make adjustments if needed.

[00:19:12] Brent Maher: And I say make adjustments without compromising what your mission is too, you're a stakeholder in your own strategy, and you need to make sure your needs are being met as well.

[00:19:21] Joe Robinson On Not Taking Business Decisions Personally

[00:19:21] Joe Robinson: One of the items that comes to mind as we talk about this topic as well is don't let it become personal. These decisions should not be personal reflections, right? They're not intended to reflect on you personally as a CISO.

[00:19:37] Joe Robinson: Your job is to educate. To make people aware of, and to your point Earl, to let them know what yes could look like. Now, they may choose no based on what that looks like. It's not something you should be internalizing and taking personally. You should be focused on educating, bringing options to the table. Letting people know the situation that they're dealing with from a risk perspective.

[00:20:01] And I would highly suggest you provide your opinion, and your suggestions of what you think that organization should do, but leave it on the table as a set of options.

[00:20:13] Earl Crane: That's some good advice. I hope you've enjoyed listening to our holiday episode.