AI Agents Are Deleting Production. Cloud Security Has to Catch Up | Matt Lea

Joe: How's going, Matt? It's it's great to get you on the podcast finally. the this thing has been in the making for seven months at least at this point. And it it's obviously it's fully on my end, with two little kids and then randomly getting sick, from from them going to daycare. It's just terrible to try and maintain a regular schedule of sorts. so I really do apologize. about having to delay it so long, but I'm really glad that I could finally get you on. Matt Lea: No worries, I totally understand. Actually the last podcast I was on, I was interrupted by a tornado. So I lost power in Joe: jeez. Matt Lea: the middle of the podcast and had to take cover. So Joe: my gosh. That that's terrifying to me. Where where are you based out of? Matt Lea: Wisconsin. Joe: okay. So huh. I don't know. I I'm I'm just like a little bit southwest out of Chicago. And Matt Lea: Mm. Joe: we never really get tornadoes, we'll get tornado warnings and stuff like that, but for the most part, it it just never actually happens, you know? And Matt Lea: Mm-hmm. Joe: then Which is kinda crazy to say that because when my wife and I were building our house, that w that we're in currently, I think towards the end of building it, there was a tornado three miles away. Three three miles away from our house. But it still didn't even I don't know, hit home of this can happen, Matt Lea: Yeah. No, we lost not quite in my neighborhood. We lost power in my neighborhood, but a couple neighborhoods over. People were losing roofs and whatnot. So it's a little a little frightening. Joe: wow. Yeah, no, that that's definitely scary for sure. I've never been in a tornado, thankfully, and I hope I I never have to go through one. That Matt Lea: Knock on wood. Joe: yeah, that would not be fun. But it would be a great reason to move, so maybe maybe it would work out. But, Matt, before we dive in, why don't you tell my audience like who you are, how you got started in this space. really what made you go down the AWS route specifically. and go go as far back as you want, right? the reason why I always start people off here is twofold. I want my audience to hear your story from you rather than me trying to cobble together some intro and, miss something or leave something out or not emphasize something properly. I want them to hear it from you. And Two, if someone in my audience is is watching or listening and they're trying to get into security, they're trying to get into the cloud and they don't know how to do it. If they hear someone with a similar background, right, it maybe a light bulb goes off in their head and says, Hey, this is possible for you. You can do it. follow this guy's blueprint, right? So what does that story look like for you? Matt Lea: Yeah. So I started programming in the sixth grade. My father was trying to learn Q Basic, I think it was. And I picked it up and started writing stuff. And then I actually learned a lot dissecting video games in high school, basically modding them and taking them apart. So that started me out that way. Then went off to college, decided to start a tech startup with a friend. in the process, I blew out my knee. I dropped out of college because I was working on the startup, crashed the startup into the ground because I didn't know what I was doing. And and then it was around the two thousand eight when people are having trouble getting jobs. But my friends were all graduating college, around that time and I I had I was applying for stuff and they're like, You got real world experience? Sure, we'll hire you. So I was getting jobs and my friends who actually had degrees weren't for some reason. So it was a bit backwards that way. and then I ended up getting involved in a lot of early stage startups. And if you're in an early stage startup, a lot of times you don't have the budget to go hire the latest and greatest cloud. So you wear a lot of hats. And so that's when I ended up picking up AWS. We had a project back in twenty ten, I think it was, that w it wasn't scalable on the servers it was on. So we started migrating to AWS S three first and then the E C two instances back then. And then I just kept working on it from there. So ever since two thousand ten, I did a a variety of startups, but I realized from a business standpoint, and for all you guys out there that are trying to break in, that infrastructure is great because let's just say you're building a website, just a mobile app or a design or something like that. You can build it and then they can let you go and and then they don't need you as much possibly. But with infrastructure, they always need you. There's always an attack coming or a scaling problem or something else. So it was I got into it for the consistency of the revenue. It was much better. I did Facebook apps for a while and those Q four I was busy as can be right before the holiday rush. And then come the summer I was broke. So it was a strategic move on my part, financially, but I love it. And it's a the cloud computing world just is growing and growing. So even right now with the AI stuff, someone's gotta know how to host the models there that are replacing people. So Joe: Right. Yeah. If anything, I I feel like AI is almost like a subsect of cloud security at this point, right? and and yes, it does have its own unique nuances and whatnot, but whenever I'm talking to customers about AI security, my f very first question is, All right, well, where are you hosting it? You know? Matt Lea: Mm-hmm. Joe: And it's like, we're using AWS or we're using Azure Foundry or whatever it might be. And you now have to know how to go into AWS and configure bedrock and set up the guardrails, what they look like, what it should be, all that stuff. so it's it's an interesting evolution. And I I think eventually, AI security will be like its own primary domain, in security, but at least for right now, you still need to know. The other infrastructure with it. Matt Lea: Yeah. That's it's evolving so rapidly right now. I I think cybersecurity is going to grow in some point. So the big companies that have, all their systems down, maybe they will be able to use AI models to automate all that. But what I'm seeing right now is an explosion of vibe coding. I don't mean that in a negative term. I like to think of it like a there's a small business owner that maybe runs a a a service business and they wanted an app and so they now have the superpower to code it. But at the same time, they they don't understand security measures and they haven't run it through the ringer and and so all of a sudden they've exposed, let's just say, some type of credentials to their invoicing system because they integrate with the invoicing system. And now all of a sudden bad guys are accepting payments and invoicing other bad guys to launder money through their system and they don't even know that. So I think right there you we're gonna have an explosion in that kind of mid level area where there's a big need for cybersecurity. Joe: Yeah. Yeah, it's definitely I I I feel like it's definitely not going to go away. At least not anytime soon, right? i it's like anthropic CEO would would come out every six months. I I think this last time he he didn't say that it would replace everyone for the maybe the first time in a couple of years, right? But every six months I'm supposed to lose my job over and over and over, right? Like Matt Lea: Mm-hmm. Joe: the the the market is supposed to be completely taken up by AI bots, right? And I I don't know a single IT or security professional that's sitting there being like, yeah, I'm gonna lose my job soon. Matt Lea: Mm-hmm. Joe: I I literally don't know a single one. and I've never I've also Never felt that way too. It's like, yeah, you're gonna have AI do my job. Good luck. I'm sure Matt Lea: Yeah. Joe: I'll be back here in three months, and we're we're even seeing the proof of that too, where AWS laid off a whole bunch of devs and then magically they go down hard down for twenty four plus hours, right? And it's because of some like DNS change or whatever that their AI agent made. and a human would never make that change necessarily. There's so many different checks and balances in place that Matt Lea: Mm-hmm. Joe: it's difficult to make a catastrophic change like that, especially in that sort of environment. Matt Lea: Right. Joe: or at least it should be theoretically, right? Matt Lea: Yeah. Joe: and so it it's just it's just laughable, right? And I I have friends that are trying to get into s security and they're asking me, should I still get into security? is it still gonna be around? It's like, dude, it is more than gonna be around. it's exploding. Matt Lea: Th there's I was just watching something. It was an interview with a former data fraud person. They, they would scam people on the internet. And the the mafia and all those guys are realizing that there's more money in cyber and a lot less chance you're gonna get caught. So it's a wild west for them. So you're seeing a huge uptick take in cybercrime because it's just extremely profitable if if if if done correctly. And that and we're talking about ransomware attacks. Which definitely can be I I've got a I think I totally I don't want to plug anything here, but I've run a scenario where I train people with war games to s try and get them inoculated to stress and make it so it's second nature to respond to these things. And they one of the things I run, and I'm afraid to say it tell your following, but I if you want, I can I'll expose my my Kobayashimaru, but there's a heck of a ransomware attack that you can do with one line of code if you've got the right credentials and it could be absolutely brutal. So I that's I don't know how I got off on that tangent, but that w that's a real fun one to try and fend off. Joe: Yeah. No and and you can you can plug any of the stuff that you're doing. that's totally fine. it's it's it gets weird with sponsoring and stuff, but you're not sponsoring the podcast. But that is fascinating. There's a couple the there's a couple things that I want to touch on there. And one of the things that that you touched on is like the organized crime aspect of it. Where and I didn't realize this until fairly recently when I had someone on. And we were discussing how organized crime will have their own cybersecurity teams defending their own infrastructure. Like Matt Lea: Mm-hmm. Joe: there's actual sex trafficking rings and stuff that are managed by organized crime groups that hires cybersecurity professionals to manage their infrastructure, to secure their infrastructure. Matt Lea: Yeah. Joe: And I I couldn't I couldn't imagine, being a cybersecurity professional and I'm protecting infrastructure that's supporting that, right? I I don't know the kind of mental gymnastics Matt Lea: That's yeah, yeah. Joe: you'd have to go through to be able to rationalize that, no matter the money. Matt Lea: Yeah. That's pretty dark, yeah. Joe: but it it just shows the the switch, right? Because I think When I was getting into cybersecurity, people were still worried about the lone hacker in the basement, 2010s, 2014 time frame. People Matt Lea: Mm-hmm. Joe: were still worried about that. And it really wasn't until probably 2018, 2019 when we started to see a glimpse switch to this is a little bit more organized. We're seeing Matt Lea: Right. Joe: this a little bit more thought through, and and then the attacks started to become more broader, right? Like more Matt Lea: Mm-hmm. Joe: detrimental to society and the globe overall, right? with different ransomware attacks like Not Petcha and WannaCry and all these other things. so it's just a interesting dynamic, right? And and then on the other side, right? we have good guys that are actively going after, these these criminal organizations. And it's like we're going up against real hackers. We're not going up against idiots in a jungle that are putting this stuff together, ad hoc. It's like like, no, this is we're going up against real IT security professionals, know? Matt Lea: Mm. Yeah. Well organized in cells, ready to to with systems to actually handle fallback. So it's not just some kid that's all calfed up and when he sleeps, you're safe. They they go where they work in shifts. They're ready for it. Joe: Yeah. Yeah. Yeah. That's and that's something like even most companies have difficult have difficulty like fulfilling, right? Having that shift available. I think I've worked might have might have worked at one company that actually had a proper twenty four seven SOC, staffed with internal employees. Everyone else just always outsources it, for Matt Lea: Mm-hmm. Joe: forever. you basically have to. Matt Lea: Yeah, I've dealt with a lot of early stage stuff and sometimes you don't have the budget for it. But the good news about that is we don't have as big a target on our back either, when you're dealing dealing with something like that. So just don't be easy pickings and it it at least slows down the amount. But the bigger you get, I I help one customer grow to millions of dollars of sales a day f and they started off in a garage. And I'll tell you this, the problems they've got today are significantly more challenging. It's just we are just constantly being bombarded, or fished or, you name it, cross site scripted, etc. It's it's fantastic to see. Not fantastic and it's a it's mind boggling, I should say, to see, to think of the scale above which they get attacked. And they're not even something the size of Amazon or anything. They're but they're pretty darn big, but not quite that big. And so it's just it's amazing how much just every day there's something new. I can tell you right now, there's people crawling that site maliciously and That we'll probably I'll have a meeting probably later on the day about it. So Joe: Yeah. Yeah, it's pretty it's pretty common for sure. and it's interesting because, OpenAI just announced and I w I wanna dive into this a little bit, right? Where open AI talks about how they put their agent into a sandbox and it escaped the sandbox, executed some, traversal attacks and whatnot, and was able to, hack into hugging phase, right? And compromise some models. Matt Lea: Mm-hmm. Joe: Now, every other time that we've heard of, these models doing it, it was under questionable pretenses, right? Where, you're giving it a goal, you're telling it what you want it to achieve, you're emphasizing it, you're training it in these areas and whatnot, and then it's pulling it off and whatnot. Anthropic did something like this a couple months ago. but when I had someone on to talk about it, they were saying, well, it's kinda questionable a little bit about, the environment that it was actually in. Was it actually sandboxed and, all this other stuff, right? Matt Lea: Yeah. Now Joe: What why why don't you know if if you know it well enough, maybe sum up what OpenAI say that they achieved and what the actual risk is. Matt Lea: I think there's this isn't the the risk factor is very interesting to me. so you summed it up pretty well. As far as I know, it was trying to cheat the tests, and so it went to Hugging Face to get some of the data that it would have been the answers for this particular one that it was being benchmarked on. And it s this is bad security in some ways, but it used several zero days to exit, get to s to a place where it still had network access. In my opinion, you probably shouldn't give it network access. That thing should be plugged into the wall with only electricity, but not, not full access to the internet right there. But it managed to make its way out, which r raises interesting questions. So I think you're probably familiar with worms from back in the day. I don't one of the most famous ones was a MySpace worm where there was some JavaScript on a profile. If you loaded it, it'd execute the JavaScript. So I so you you had this malicious thing on your profile, we'll say. I visit you, now the JavaScript executes some code and adds itself to mine. So then my friends view it and it adds so it expands there. But the interesting thing with models of this nature is that most machines out there aren't beefy enough to run the model itself. So it would have to basically jailbreak itself coded harness of some type that would then invoke whatever it wants, on these other models and memory would be another interesting thing because it would have to have this, how would it How it if i as a worm goes, would it have limited memory? Would you each worm have its own memory? Would it have some type of shared memory? if it could jailbreak some type of database that people aren't looking at that it could use to like a vector database or something like that. But I but long story short, unless it figures out a way for it to run itself on cell phones which have GPUs and computers which have GPUs basically on a lesser model, it's gonna have trouble really bouncing around out there. and getting and really breaking free. If if anything, it could maybe stay inside his data center and steal some GPUs there. But I don't know, it's interest it's almost science fiction to think of it, Joe: Yeah. Matt Lea: God, there's a movie that had this happen. I can't remember what it was, but Joe: That that's why that's why I almost think that it's too good to be true because, you said, it almost sounds like it's not a true sandbox. it it almost sounds like it truly wasn't air gapped because when you with with my limited work with the government, right, and I've never looked at an air gapped system, right? Because it's air gapped, Matt Lea: Mm-hmm. Joe: that and it would require a clearance to even just look at that room. Matt Lea: Right. Joe: But when you when you hear about something breaching the air gap, there's only two options if it's air gap properly, right? Matt Lea: Mm-hmm. Joe: You e you someone either went into the room and plugged something in that they shouldn't have, or you used sound waves through the wall in a very weird NSA like way that I've literally what I've I've read one single report on, right? Matt Lea: Mm-hmm. Joe: Of how they breached. the air the air gap like that. And when I don't see that kind of attack take place on what is supposed to be a sandboxed, AI agent, it's like, okay, well, what were the standards of the sandbox? Matt Lea: Mm-hmm. Joe: what environment are we putting it in? Did you link it to systems? And it sounds like it sounds like they did. they put it on the network, even though it was super limited, they gave it a goal and it starts to traverse its way to that goal. it's just it was given a goal, it has no guardrails, it's gonna work its way through, right? But we are at the point where I think we are actually at the point where these models are powerful enough to where it's valuable enough for an adversary to expend a lot of resources at getting that model. breaching that model in some way so that they can then, use it for their own needs. Right? Matt Lea: Right. Joe: if you go back a year, it probably wouldn't have been worth it for any adversary, Russia, Iran, China, to try and breach open AI to steal their source source code. but now it seems like we're we're at the point where it's like, okay, this thing is probably good enough to where we should actually do that. And maybe Maybe China wouldn't do that, but knowing China, they have a history of doing that. But I Matt Lea: yeah. I mean Joe: only argue because they probably have something similar that they're also working on that they believe in, so it may be like, do we really need to but they probably still would regardless. Matt Lea: Yeah. A are you familiar with China's distillation efforts? I believe that's the term. Joe: I think I've heard of it. I I'm not very familiar with it though. Matt Lea: So my understanding, I'm not a data scientist here, but I I do have to dabble in some of this, is that they have figured out ways to reverse engineer the model a bit. So they feed in input A and whatever the output is, they then use that to backpropagate and train their own models. Cause that's you basically got training data there. put in input B. Now I've got this, and so you can retrain your own models on that. And they're training cheaper, computationally wise models there. And people are accusing I think it's k allegedly I'm I'm saying allegedly ac accusing Kimmy two or I think is the new one that just came out of doing things like that. And so I I even if you they don't get the source model and weights, it is still possible for them to basically get the training data that they need to do for s to train models. So I and that's as far as I know happening every day right now. and I think an interesting point that was just brought up recently, people are arguing That those models are only four weeks to four months behind our current models, which I think is why people are a little getting tighter with releasing these models, getting access to Fable 5. an example on AWS, I have to opt into a new feature on AWS that says you can send all my data back to Anthropic because they want to make sure their model's not being abused. Now, let's just say that's true. Because maybe they don't want they want to catch you if you're trying to distill distill back the results for your own models. but it's it's interesting to watch because I I'm pretty sure that feature didn't exist before June and Anthropic was able to twist AWS's arm enough that they actually put that in there, forcing the data back to Anthropic. maybe for national security, maybe because they want to spy on us and train their own stuff. Hypotheses. I'm not saying any of that for sure, allegedly. Joe: Right. Yeah, no, I I think I think I did hear about that, how they have a billion agents probing anthropics models to Matt Lea: Mm-hmm. Joe: get the the different results and whatnot. And it's fascinating that they're doing it that way. Maybe maybe not that they're doing it that way specifically, right? Because there's so few ways for you to actually achieve that. But at least to me, it like speaks to the intelligence level that Matt Lea: Mm-hmm. Joe: that they have, right? Because I think back to and that probably sounded terrible, but I I think the listeners will understand. there's levels to this thing. I've talked about it before, when you start reverse engineering things in that way where you're feeding an an action or, a prompt or whatever, and then you're getting a response and you're trying to base the next action off of that. it's it's different, right? but I I think back to when I was working with a privileged access management solution. And the the environment that I was deploying it in was probably the largest environment that this vendor had ever been deployed in before. We were running into sizing issues, not related to resource sizing, but actual Hey, your application can't query the database fast enough for this thing. Right. And literally me looking at the web app and then working through in my mind, okay, I did this thing. What's the command that's sending to the database? The database is now doing that lookup in this way. And I'm explaining it basically to the engineer on the other call that works for this vendor. And I'm like reverse engineering their product like on the fly and Matt Lea: Mm-hmm. Joe: and they're confused as to how I'm doing it. It's like, no, I've worked with Postgres before, I've worked with SQL before, I know how it works, this is what it is. And it was only through that that they were able to make progress with their devs because I was able to talk them through hey, it's doing this, that's fine for smaller customers. It needs be doing something like this, know? Matt Lea: Mm-hmm. Joe: that same mentality applying it to an AI model is fascinating and also probably I would think very arduous because you have to send over so many different prompts and you know, I Matt Lea: Yeah. Joe: the report that I was reading, they have millions of agents or something like that, sending over various prompts and whatnot, trying to probe the models. Matt Lea: Yeah. But that's the thing is they they can do it at scale. we're at a pace where yeah it costs money to an extent. if the model's open source, then you can run it, on your own GPUs. So they that speeds them up. if if the model's more closed source or something like that, then you're end up paying for tokens. But they they that that's another thing that's a big trend right now is is it's all these people that are fairly new to programming and InfoSec and you're losing your open AI creds or your, your Gemini API keys or whatever, and they're getting stolen by the bad guys. And now they're you you get a ten thousand dollar bill, not them, but they've got the answers they want. So even if you get it deterred, so it's been a stealing API keys is nothing new, but that's we're seeing a pretty high volume of that happening because people are just aren't infosec minded and are starting to vibe code their own apps. And again, I applaud anybody trying to learn to program. in that technology. Just just be aware there is some security risks. So be careful where you put those creds. Don't just give your model your credit card number, I guess, or your AI assistant your credit card number directly. Like in the chat. Don't do that. Joe: Yeah. Yeah, it It's a weird place that we're in, right? Because it's great that the playing field is somewhat level, right? Matt Lea: Mm-hmm. Joe: Now it now it's somewhat even for people that, have an idea, have a need, they didn't have the money to to pay for a dev, to to do it all. And now AI can get you ninety, ninety five percent of the way there. And it look, Opus is smart enough to tell you, hey, this is what I'm lacking. you need to do this stuff for me. I can't fabricate this stuff, right? Matt Lea: Mm-hmm. Joe: And it'll walk you through how to do it and everything. I'll give you an example. I'm trying to I this is just like for me. I'm trying to create this Pokemon card grading app, right? That gives you a really accurate estimate of what your card would grade at at PSA or Beckett or whatever. And it's a cool little side project for me. It yeah, I really want it. I think that would be really cool. And there are there is one other competitor, but I also don't really care about competitors if if that makes sense because it's like I'm doing my own thing, I don't care. Matt Lea: One competitor is not bad. For every Coke there couldn't be a Pepsi. Joe: Right. it it's surprising that PSA or Beckett didn't already do this. but they're probably only interested in providing the grades, that that's just been their bread and butter. But, in in this whole thing, I'm not a dev. I I've never coded an application or anything like that, right? So it's building out the framework and then it tells me Hey, we need super base database, go set it up. Do this, point these connections, give me that, setting it all up. And I'm a security-minded person, right? So I'm thinking through the API keys, I'm thinking through the different connections that it's having and whatnot. I'm thinking through how I'm gonna secure, super base and whatnot. it's fine for me because I'm Matt Lea: Mm-hmm. Joe: working through those things. but on the flip side, I have a friend who's building a an app that's like supposed to be like a central ingestion point for security tools. in an environment. And, my very first feedback is like, this has to be the most secure application in the world. Like, Matt Lea: Mm-hmm. Joe: if you're plugging everything from someone's environment into it, it's a world of pain if this ever gets breached and Matt Lea: Yeah. Joe: the name will be forever tarnished. you're never going to re recover from that. Matt Lea: Yeah. I so I I I've seen the models personally do things I thought was unsecure. So I I'm paranoid. I ran Hermes agent inside a Docker and tried it with different models. But one of the models decided it wanted to send information out to the world. So it did a I had given it access to the browser so it could do a little web searching. but it it had a little more access than that because it I could have it I could have it test my website or whatever too. It was just playing around with it. And it decided to open a post bin. which is like a basically a free copy and paste thing. So you copy and paste this information and it put a bunch of environment information that it had access to. Luckily just the stuff it had in Docker. But it was that close. If it had an ENV file, I would have s I could I bet you it would have popped it right in there. And right there someone could have crawled post bin or wherever they sent that thing off to would have had that info. So that's that was an eye opener when I found that. I'm like, you just posted all this information without asking my permission on the internet for anybody with this URL to see. that could have been bad. Luckily, again in a Docker container, but I got another story too. so another AWS story. I didn't do this one. And I applaud who the person that did and I'll explain in a second. They gave Terraform access. So Terraform being an infrastructure's code, you can define what you want your server infrastructure to look like. And it didn't like the way the Terraform was. So it rewrote it. And what it did, it deleted basically a database out of there and renamed it something else. Well, Terraform takes that as this database no longer exists. That was the production database. So that the thing did a Terraform apply, production database, all the backups, all that stuff, gone. So this guy luckily and If this ever happens to you, you can try this, but I can't guarantee it'll work. You called AWS, and AWS had some luckily deep under hard backups that was able to restore it. but their production system went down for it was a hellish day. I wrote about that one. I don't remember the guy's name. I'd love to give him credit though for actually doing a post mortem on that so the rest of us can learn. But that's that's my fear. Don't give it the ability to a terraform apply. in your production environment. I don't even know about your other environments, but definitely not production. You should be reviewing absolutely every change. Joe: Yeah, that is crazy. And and the AI agents have no real context around it. It just knows, okay, we don't need that thing anymore. we're cleaning it up. We're doing this other thing over here, right? Like Matt Lea: Mm-hmm. Joe: a couple of months ago when the open AI engineer gave it access to her email, right? Deleted all Matt Lea: Mm-hmm. Joe: of her emails in an instant because it didn't know how to prioritize them. You know, like Matt Lea: Right. Yeah. And we Joe: Crazy stuff. Matt Lea: I saw luckily this happened in a test environment. But a test environment we had all of a sudden everything dropped and the from what we can we we've tracked it down. I can't go into too much detail, but one of the original theories was did the did the AI just not like the schema and it was gonna rewrite it and then just wasn't concerned with the fact that this wasn't a test environment. you gotta wonder when they train these things, how much production data that they get trained on. They probably get probably get trained on a lot of like Hello Worlds and tutorials where if you rewrite the schema, no problem. Okay. Drop the table, rebuild the table. But you do that in production, So Joe: Yeah. Matt Lea: I don't I it makes you really wonder, I guess I don't know, double check everything. Joe: Yeah. Yeah, absolutely. to to maybe circle back to AWS, what what made you go maybe so deep or so far with AWS? Did you ever dabble with Azure or GCP or anything else like that? I ask because typically when I tell people, That want to get into cloud security. I tell them just start with the cloud that your company is currently at, right? Matt Lea: Mm-hmm. Joe: Go deep into that and then dabble with the others, right? you should know what an AWS SCP is, you should know what that is in GCP, right? Maybe not that you know how to work it, right? But you should know the name Matt Lea: Mm-hmm. Joe: to get more well rounded. But why why did you end up going so far into AWS? Matt Lea: So truth be told, yes, I started with AWS. That was the only thing on the market pretty much back in twenty ten. They had a quite a while before that competition entered in the market. so I just started with it and that's what I knew. truth be told, I have done a little bit of GCP. Actually, we have got systems running on GCP that I manage, but the reason you don't see it from a business standpoint, again, I do I run my own consulting business from a business standpoint. You don't want to be the hey, I can do everything guy. you, the a handyman that can do a little bit of everything. I can paint, I can do everything. They're not gonna get nearly as much as the best high grade industrial freezer H VAC guy. I can do a little bit of H V A C stuff or I am the guy that knows how to make sure your food's always frozen for Costco giant freezers that never breaks down. the so the the amounts you can charge are significantly different by being the expert. So you just don't see it in my marketing. But yeah, I I do Quietly manage some G C P stuff. Not really any Azure. Never got into that much. I've done some liaising with people on Azure that wanted to migrate off of Azure, but nothing crazy. Joe: After working with Azure a bit for the past year, I absolutely hate it. Honestly. Matt Lea: Mm-hmm. Joe: Azure is maybe the most frustrating cloud I've ever been in because there's fifteen different admin portals that you need to go into to, sort things out and whatnot. It's like AWS is at least everything is right there, Matt Lea: Mm-hmm. Joe: there yeah, there there's no there's no issues on that front. But Yeah, I I'm right there with you. I I prefer AWS for sure. and there there's gonna be some new developments go coming soon that I'll be focusing on AWS primarily going forward. So I'm very grateful about that. But why why do you see a lot of companies maybe go towards AWS rather than other clouds? Matt Lea: well the complaints there was some complaints about Azure's UI and and how it's all set up. they're just they had such a good head start. I think they're crushing it. And then right now with as we do more AI stuff, which is why I think there's very little moats around these AI companies, is that they're killing it hosting models. They've built built this beautiful bedrock serverless, model that you can I like Bedrock Hall the Netflix of AI. You've got all these different models you can pick from. Hey, I want to do video, I want to do this, that, and it just just magically runs. So it makes it real easy for you to set up. Now, if you're on the other side of the spectrum and you want to do real heavy data science PyTorch stuff, they've got your SageMaker. And now you've got some more stuff there. It's that it's not quite as point and click, but you can really customize it and and get your hands dirty there. So basically what I'm saying is that they seem to be keeping up. really well with with the cutting edge of hosting whatever you want. They've got crazy services that I don't even I've never used, but they've got if you want to get direct 5G access in a city and have the five G go right to an instance running in that city, you can shorten those pipes and they use it for a sporting event. A lot of times people want to watch it from their phone before they go in or out. And so they're doing just just fantastic things like that. Not that the others aren't. So it's all it's it is competitive and I have no idea the winner's going to be, but again, if they end up being the Pepsi to let's just say Google's Coke, it's still a pretty solid market share where you're gonna find work, Joe: Yeah. Matt Lea: So I I honestly I look there's things that G C P does that Google or that AWS doesn't. That's why we do have some G C P stuff. So again I don't wanna just say, hey, AWS is the only thing, do that. I'm just saying I they they do everything my clients need and then some and so yeah. Joe: Right. Yeah, I I'm actually a fan of GCP where GCP to some extent they take the guesswork of security out of it by reducing their own liability significantly. if you turn off backups for instance, Matt Lea: Mm-hmm. Joe: they they actually don't care. They're gonna warn you about it three different times and they're still gonna back up your stuff. Somewhere, right? If you turn off encryption, they're gonna warn you about it, they're gonna emphasize that you shouldn't, but in the back end they're still encrypting it. You know, Matt Lea: Mm-hmm. Joe: so they're building in that security whether you want it or not, which I personally really like because Matt Lea: Gotcha. Joe: it it makes it a little bit easier, a little bit more consumable. And I'm sure, someone could argue, well, AWS is doing TLS, from server to server or what. whatever whatever the argument would be, right? But at the end of the day, when you hear about cloud breaches specifically, with AWS or Azure, it's typically cloud misconfigurations, right? Matt Lea: Mm-hmm. Joe: And I I honestly I just can't remember one with GCP that was a cloud misconfiguration. Right? it was like a I I remember one where there was a hypervisor escape. But that was a long time ago and it was quickly remediated. And ever since then, people really forgot about it. But even when, even when I talk to people at Google that work in GCP, they almost respond with a defeated mentality of yeah, we're a distant third cloud provider. We know, we're trying we're trying to fix it. and they they just sound like they're already. Th they already lost the cloud game and they're trying to just maximize, their their their customer base, their focus, their focus for those customers. Matt Lea: Mm-hmm. Yeah, I don't know. I that's a good question. I wouldn't think they're defeated at all, in my opinion. Again, there's th there's there's plenty of room in the market right now unless people stop using the internet, which doesn't appear to be the case. They're killing it in so many other areas. They're they're they may not be leading the way in the AI models and all that stuff, but they're right there and they've got more compute than a lot of these other guys could even dream of. So I they're I I I don't by any means think they're going away anytime soon. As far as the security goes, AWS will tell you don't do stuff, but they'll let you do it. And my gosh, I I have done many audits of different people's AWS setups and I can't think of one where I didn't find at least something where I was like, this this is Swiss cheese. this is Joe: Yeah. Matt Lea: so there's usually something there. But yeah, it's they they give you all the tools to be as secure as possible. And if you use the UI, they tell you probably shouldn't do that. But people people a lot of times they're just thinking, I need to make this easier for myself. And the same thing with the AI. if you had an AI model running it, and it's like, Well, I need to access this production database. I'm just gonna poke a hole in the security group so I can access it from the public internet, and then that IP address gets crawled on that port next thing So it's convenience versus, making it convenience for your team versus making it inconvenient and expensive for your enemy. Joe: Right. Yeah, that's the thing. After doing a whole bunch of assessments with Azure and AWS, with AWS at least, the configuration options are right in front of you. if I block it here, there there's only so many other places that I have to go to to validate that there's not a workaround or whatever. Matt Lea: Mm-hmm. Joe: With Azure though, I I'll tell you this. I just got done doing a a basically an IEM entre AD ID audit. Just like a security overview. I there was 12 different places I had to confirm for MFA to ensure that there was no workaround. And I'm sitting here like I am done checking this stuff. I am not, I'm not doing another configuration check. You either have it or you don't. Matt Lea: Mm-hmm. Joe: You know, that's just so enraging. It's so frustrating. Because you know what? As the end user, I know I know for a fact they set it up thinking, hey, this one setting is all I need to set. This one configuration thing is all I need to do. And then maybe they Googled a little bit and I gotta set this flag over here. They didn't know about the other ten things, right? Matt Lea: Mm-hmm. Joe: they had no clue about them. And I can Tell they had no clue about them, primarily because some of them I learned on the fly, right? to be quite honest, that's what you have to do sometimes. Matt Lea: Mm-hmm. Joe: And that's just the most frustrating thing because you're making it so much more difficult to manage for your end users that are paying you all this money to use your platform, it's just frustrating. I'm I'm sure Microsoft has blacklisted me several times over. Just from this podcast. Matt Lea: Yeah, well s yeah, yeah. Joe: They make it easy. They make it so easy. Matt Lea: Yeah, yeah. Now it's it's always a balance, security versus especially the early stage stuff, If you get an early stage an idea and hope and dream, be as secure as you can, but you're you don't have that big target on your back yet. So maybe and this isn't my I I usually I help CTOs whatever do this math basically, Okay, we can spend $100,000 on security, but then we've got zero runway for any new features, And as they get bigger, of course, that budget for security grows. but it's it's it's like how much do you want to impede the bad guy, and how much do you potentially want to impede your staff that wants to boot up a new server there? And it's something that you gotta balance. Now, if it was like, if this was a federal government thing, well then obviously we'd have a we'd want to have a You got a big target on your back, and you'd wanna obviously spend every dollar and we hire a whole crew of CIOs and all that stuff and CISOs and all that stuff. So, yeah, it's it it it's an interesting balance. But early stage yet's how much do you wanna spend defending this this data that's still completely unproven if it has any value, And so so I have to have that conversation with my clients. Now, me, I'm paranoid as all it can be, as per the nature of the job. So I'm always saying, Well, perhaps we should do this and this and this, but you've gotta deal with the reality of it if you do an early stage, Joe: Right. Talk to me about the the Cloud War games. I I haven't dove into it, but it honestly it sounds like something that I probably want to explore a little bit. Matt Lea: Sure. So I saw a pattern where, I've got a client. If they go down, they lose hundreds of thousands of dollars an hour. Right. And that's so all of a sudden you've got guys in suits barking at your ear, and and it's i if you're not inoculated to the stress, you you clam up. And so one thing I'm big on is I don't like knowledge silos. I try to always be training up my clients and make sure that we've got redundancy. I also hate being on call, so I don't do that anymore. So I train them to handle that. So you go to you've you've got this outage, you're literally every minute you're losing, ten thousand dollars or so in sales and they're yelling and so you pass it off to the new guy saying, Hey, give it a try. What do you see? What do you you I'm telling, and I'm I'm tracing tracing it down, but I want to hear him talk it through or them talk it through as they're going through it. And they clam up, they're stressing out. They just couldn't deal with that type of pressure. So I wanted to create a simulated environment for that. And so several decades of doing this, I've got a whole list of outages, the most painful thing I've journaled like crazy. So all the most painful outages I had write down, I called up my buddies that have been in the trenches and said, What do you remind me of that? Wrote them all down, created a simulated environment in AWS for a fake company that you could knock down all day long and no one's gonna care. And then I started running people through these different scenarios. And so I do it to train for my clients internally, and then I was doing some publicly where we put them on YouTube. Those were a pain to schedule, but we're we should probably should do some more of those. But either way, I'd knock down part of the infrastructure, either a misconfiguration or an attack, or database fill it up with so it runs out of memory, DDoS from the outside, a handful of different things there. And we played through a couple different ways. So one way we would do it is we'd have a collaborative environment, which is for your more beginners, okay, you're gonna drive, you got the hot seat, you're gonna drive, you're sharing your screen, everybody else is gonna follow and and watch. And we'll coach each other a little bit. And so that's a really nice tool as well. So say you've got a really advanced step. You got that one guy that if he takes vacation, it's the company is in pain. I'm talking about the knowledge silo guy. Joe: Yeah. Matt Lea: The I call it the lone wolf programmer. If you read my comics, if you go to schematical.com slash comics, I got a whole series on that. But it so so you can see if the lone wolf is a lone wolf. Does the lone wolf coach the other guys? Well, then he probably shouldn't be the management there. That's that's not a good spot. That's like a hey, keep him as a consultant, make sure someone's documenting everything they do. Or are they helping? Are they, encouraging them? Are gonna train up people? If so, that's someone you probably should promote, And so it's been a good recruiting tool, it turns out, for companies to say, Hmm, we we want to run our candidates through this little thing. maybe it's a paid test or something like that. I don't know how you do the paperwork, but We bring them to this thing, I run them through and you see how they interact with each other. You see how they communicate, you see how they, what do they do if if I'm putting the stress on, So that's that's been a really fun part of that project is seeing seeing that and really and and letting people that are aspiring cloud people show off. Actually, I've had people take the recordings and circulate them with their job interviews. So no longer are you just a face, in a or it's no longer you are just a stack of piece of paper and a stack of resumes, you're a a face along with it. You get to see an action. So It's been interesting on all sides to see that response. Joe: Hm. Yeah, it's when you're responding to an incident, it's one of those things where it's like you're either used to the stress or you're not gonna perform very well. just plain and simple. my my very first job out of college was a very pretty high stress IT job because of the application that we were running and the clients that we had and whatnot. So I would regularly pick up the phone and someone's already yelling at me, to get something up and running and I'm over here week three. I don't know I don't know Postgres. I don't know Linux admin stuff or anything like that. I have to go take walks after the the phone calls to kinda de stress and come down, from that call, right? Matt Lea: Mm-hmm. Joe: and fast forward, I was working I was working for a large automotive manufacturer on on cloud security. And we went through a tabletop that AWS put on and we we did so bad, so poorly. And and the Matt Lea: Mm-hmm. Joe: reason is there was no actual run book. There was no playbooks or anything. We had no process defined or anything like that, which was mind boggling that this multi billion dollar company a year doesn't have these basic things. And we got to week two in this simulation, and my CISO had to step in and say, hey guys, just so that you realize this, you guys didn't even inform security that there's a breach. Right? it's week two. You didn't tell us that there's a breach. There's no way for us to know in the environment other than you telling us. You guys knew about it, but you didn't tell us. And they're like, That is a problem. And Matt Lea: Mm. Joe: then they finally got, security engaged. And here I am trying to, pick up the pieces. And we figured out, in that exercise, this person over here, if we ever lose him, we're screwed on this entire application. This entire application that makes us a million a minute, it's it's not good. Right? Matt Lea: Mm-hmm. Joe: there was a lot of takeaways from that. And I've found, for those stressful environments or those stressful tasks, you need those detailed, run books and playbooks where it's just like this thing happened. Okay, I gotta contact this person. Here's their email address. And then this person, just go right down the list because you have to make it you have to make it as plain and simple as possible, where it's like you are now following this. step by step guide of what to do. And Matt Lea: Mm-hmm. Joe: by the time you get to step ten, everyone's engaged that you need engaged. Right. Now you start working through the problem. that that's at least what I've found. Matt Lea: Yeah. With early stage stuff it's difficult because each company is unique and a lot of times they don't have that process. I mean, Joe: That is true. Matt Lea: my my clients one of my clients is got his PhD in cow whatever, cow breeding. I d I don't it's it's a better more technical term for that. But it's like that that guy knows nothing about infosex. So that's where the value of of having somebody external like us come in is, but a lot of times they don't have the budget for that. They're just trying to stay afloat, So you've got to start to build that. Those processes So important. if you want to go fast, go alone, lone wolf programmer. If you want to go far, go together. And if you're gonna go together, you're gonna have to come up with a process. And you have to learn if it's a is it a process person problem or is it a per people problem? Is this just the is this person a jerk and they don't like working with other people and they want a job security? I'm putting air quotes up for those of you on the audio. And or do they are they willing to train up others and create a process? Now no two outages are the same, but informing with a C suite that there was an outage that the C this chief financial off officer should be aware we're probably on the line for X amount of dollars. that all that stuff, that those processes can be established, it's c instead of just call Lone Wolf Programmer at three AM. That's I've seen so many of them. Just call call Jimmy at three AM. it's like, well what if he gets hit by a bus? And seeing that metaphorically happen so many times, poached away and now all of sudden so yeah, having those creating those run run books Writing those what you know is definitely an important thing. And it might just be a run cheat sheet if you're a three three man startup, a three person startup. It if if you're, if you're a hundred person startup, it should be a lot bigger and so on and so forth. But I'm totally valuable. I documentation is one thing I other developers get annoyed with me because I'd like they're like, You don't need to write so many docs. I'm like, Yes, I do, 'cause someday I'm gonna forget what the heck we were talking about. So Joe: Yeah. Yeah, no. Notes have saved me so many different times. you you absolutely need it, in in my opinion. Matt Lea: Yeah. And then and then that becomes the next training round for if you tabletop war game, whatever, Joe: Right. Well, Matt, maybe one last question before I let you go. Where do you think people should spend their time studying and learning for this new evolution of security? I don't think cloud is going away, but how do you approach the cloud slash AI like education part of it? Where should people get started, do you think? Matt Lea: That's a good question. I think it depends on where you want to end up in a lot of ways. I hate to say that, but if if you you could become a generalist, and you risk not standing out in a crowd, if you've if you got a consistent it's it's moving crazy fast right now, but you gotta kinda I'd I'd niche down. If you're it dep if you wanted to go and be valuable at a big company, you wanna be the best at one thing. You don't wanna You wanna have a T-shaped skill set, where you're you got a great fundamentals on a couple handful of different things, but you've got something you want to pick something that's in value. And that might be AI model security, making sure that it can't be jailbroken or something like that. That's just an idea. And then I'd find the best person on that and then I'd start writing on it and make sure you, you you're teaching others because you learn plenty by teaching. another thing, so I know I'm gonna can I s I'm just gonna go slightly tangent earlier because we talked about this earlier and people are concerned that it's difficult to break into the infosec right now. Is that you can definitely go the big route with big companies. I have always had, at least if you're if I was a young person just starting out, I would reach out to an early stage company directly. I wouldn't just fill out a resume. I've actually done this before when I was younger. I looked up the CEO's phone number because it was in his whois records and I called him directly on the phone and was like, Yeah, I want to work with your company on this thing. And and that's how I got a job in New York out of from Wisconsin out of nowhere, because I was the only one that actually called the CEO. So it's easier to get experience. in those smaller situations. You're not just stuck in the corner and now you're you're waiting on Lone Wolf programmer to finish whatever they're doing so they can tell you what to do and you've been sitting there for three days twiddling your thumbs. but it also you can you can really contact and make a difference. And and I I think that's a great way to start your career. Now, if you're, fifty years old and you've been doing this forever, that's not where I would go probably if you got a nice cushion thing. I don't I wouldn't blame you. but as far as where to read up on on the information, it's moving so fast right now. The way I gather information today is drastically different than even a year ago. So I would say podcasts like this, are is a great way to do it. And there's but there's if you'll forgive me, O'Reilly's is I've got a course on there if you want, Zero to Hero, an animated guide to security in the cloud. But even that is evolving very quickly. So you've you're gonna learn a lot and technology is gonna move a lot in the next few years. So just be ready for change. Embrace Joe: Yeah. Matt Lea: Embrace change. Joe: Yeah, no, y you have to. you have to embrace the change. You have to embrace, forever learning really Matt Lea: Mm-hmm. Joe: in this field. If you're gonna be relevant in any way, you have to be learning, you have to be pivoting your skill sets and adjusting to what the market is. But Matt Lea: Mm-hmm. Yeah. Joe: Matt, I I really appreciate you coming on. I I really enjoyed our conversation today. Matt Lea: Yeah. thanks for having me. I I had a great time. Joe: how about you tell my audience where they can find you if they wanted to connect with you and where they could find your company and your projects that you're working on if they wanted to learn more about Matt Lea: Sure. so my primary website is schematical.com. That's the word schematic with an A L at the end dot com. And so from there you can see I got a lot of free offerings, things, Terraforms if you want to get started on AWS and spin up infrastructure, a lot of open source stuff with videos there. we've also got links to the O'Reilly's course. and the Cloud War Games it's basically is being folded into my community offering as well. So you're gonna that may change by the time this goes live, but we're Trying to combine the two a little bit more. And then if anybody's interested, I also have an indie game I've been working on about cloud board game or sorry, the cloud infrastructure and infosec that tries to make it more fun for the non-technical, which you can find on Steam at tech debt, so just T-E-C-H-D-E-B-T. And we're looking for playtesters right now. Joe: Awesome. Well, thanks everyone for listening and watching this episode. I really hope that you enjoyed this conversation with Matt. Be sure to go and check out all the links in the description of this episode and connect with Matt if you want to make that connection and check out his links if you want to learn more. Bye everyone.