The Nostalgic Nerds Podcast
The Nostalgic Nerds Podcast, where we take a deep dive into geek culture, tech evolution, and the impact of the past on today’s digital world.
The Nostalgic Nerds Podcast
S2E12 - WarGames Is A Documentary Part 2
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In 1983, NORAD gave a president four minutes to decide whether to end the world. That was the Cold War's gift to the future: the principle that speed matters more than thought. In part two, we pick up where the missiles left off and follow that logic forward. The four-minute window became millisecond cyberattacks, algorithmic trading crashes, and autonomous systems that act before any human can intervene. The battlefield moved from silos to servers, but the core problem is the same one WarGames posed in 1983. The machines are faster than we are. Forty years later, the teenager is gone. The speed isn't. The systems we built to protect us now operate faster than we can supervise them, and nobody's built a failsafe for that.
We'd love to hear from you. Click here to give us ideas on new episodes.
Join Renee and Marc as they discuss tech topics with a view on their nostalgic pasts in tech that help them understand today's challenges and tomorrow's potential.
email us at nostalgicnerdspodcast@gmail.com
Come visit us at https://www.nostalgicnerdspodcast.com/episodes or wherever you get your podcasts.
So, all right, so let's do a time jump from, you know, monochrome screens and, you know, all of that. Forget the 300 baud dial-up modem, and then come up to fiber optic connections running at speeds that, you know, back then would have seemed physically implausible to anyone in 1983. So we'll get rid of the acoustic couplers and rubber suction cups and put that cloud API that accepts authenticated requests from anywhere on the planet. We'll get rid of the rule-based simulation engines of Whopper, which were sophisticated for the era, but, you know, deterministic, with large language models trained on hundreds of billions of parameters across terabytes of text and code. Uh-oh, wait a minute. It's starting to get a little... Uh-oh. It's starting to hit... Close to home here.
Renee:Close to home here.
Marc:Replace the specific bounded attack service of Cold War military networks with the contemporary Internet. And that's to say every programmable device connected to every other programmable device, including critical infrastructure, financial systems, health care networks, your refrigerator, and the supply chains that keep economies running. The surface areas, it's not just larger here. It's a different order of magnitude larger Multiple orders of magnitude larger You don't attack a 1950s appliance, right? You know, the pink enamel stove It doesn't have an attack surface You do attack a smart thermostat Or a smart oven or a refrigerator You don't go after a pinball machine You do attack a cloud service That manages authentication for a hospital network
Renee:Okay, the thing that strikes me when you put it that way is that the basic fear in war games, the fear that a machine might act on a misunderstanding with catastrophic results, it hasn't gone anywhere. It's just been distributed. So in 1983, the anxiety was localized to a single system with a single catastrophic failure mode. But the machine misunderstands a simulation as reality and initiates a launch sequence. One machine, one mistake, one very large explosion, but it's one, right? The contemporary version of that anxiety is, you know, it spreads across thousands of interconnected systems, each of which might be making smaller mistakes that individually might look manageable, but that compounded and interacted in ways that nobody predicted and nobody was fully monitoring. Like, I always thought, like, if you were an adversary of the United States and you had cyber capability, like, rather than, like, take down the grid, why not just turn everybody's heat up the hundred and just let us all sweat it out? Like, we couldn't stay in our own homes because we couldn't control our own thermostats anymore, right? Like, that to me is the stuff that I don't think we think enough about, right? I don't think we think enough about.
Marc:Don't turn the grid off. Turn everything that uses the grid on. On. And turn it on at the highest possible draw it could possibly take.
Renee:And then the grid will fail all by itself.
Marc:Exactly. Exactly. Like there are other failure modes, right? And I think that's the thing that. There's so much attack surface, you know, and even even benign systems. What was it? We were talking about this when there was the big AWS cloud outage a few months back. Right. And people couldn't people couldn't inflate their beds and they could remember and they couldn't use their little water filter machines and stuff like that. Like the cloud was gone. I can't get my water.
Renee:Yes. Right. I can't even get to Amazon to let them know through a chat thing that I need help because I can't get them. I got no internet. It's all – you guys, you don't even know how bad it's going to be when something happens like that.
Marc:It's just going to be all. Yeah, it's bad. It's bad. Let's talk about sort of attack process. Okay. Yeah. So here's the traditional cyber kill chain, which is the framework security researchers use to describe the stages of a cyber attack. And there are seven phases. Reconnaissance, which is gathering information about the target enumeration, right? Weaponization, which is developing the actual attack payload. Delivery, which is getting that payload to the target. Exploitation, which is using a vulnerability to gain initial access. Installation, which is establishing persistence in the compromised system. Command and control, which is maintaining communication with the compromised system. And finally, actions on objectives, which is doing whatever the attacker actually came to do. Historically, every single one of those phases required human beings making deliberate decisions, real choices. Someone chose the target and conducted research. Someone wrote the exploit. Someone sent the phishing email. Someone monitors the command and control channel. The whole chain moved at human speed, even though it could be automated. You know, but there's still a human that had to go from step to step to step, which meant that that defense also had time to operate at something, you know, approaching that human speed.
Renee:Yeah, now all those phases are increasingly automated, which changes the tempo of the whole interaction in ways that I don't think governance structures and, you know, my favorite thing is governance. Your favorite thing. Right. And so governance structures, they can't they can't keep up with that. Like like how if you're a chief information security officer, can you watch that attack chain happen now? Like it used to be like, hey, there's someone in the network. Oh, no way. Let's isolate them. Like, they're in and out faster than you'll even know they're there. And they'll have exfiltrated anything they wanted and took it with them, right? And so, like, good luck. You'll find it on the dark web, right? Like, I think, yeah, I don't think governance structures have fully absorbed what this all means yet.
Marc:Yeah. Well, and that speed of attack, you know, and compression of the chain, right? The automation and compression of the speed means that your defense has to move faster than that. And that's, you know, it completely changes how this whole thing plays out. So Reconnaissance now runs across massive data sets, you know, in minutes using automated tools that can map an organization's entire external attack surface faster than any human analyst could. So I won't say which three-letter acronym organization told me all about enumeration. Enumeration but enumeration could used to be you know something that might take days weeks even months and it would be things like Okay, pick a target, any target, GSK, Glasgow, SmithKline, right? Okay, what's the first step that I would do? I would go to all their public websites, look at all their domains, and then start doing reverse structure, like reverse lookups to see what domains they owned. And then, you know, who were on the contact list for the domains? Look up all the Whois records. Look at all of the different things, like all of this sort of research on the, their public network persona. Then I would start looking at email, you know, you know, the, whatever the email domain is and look at all the different people and how do they structure their email? You know, like all of these things that are public information that would take days, weeks, months, depending upon, you know, how, how you wanted to do that. I knew, well, again, I won't say where, but A guy that was teaching us how to do some of this enumeration, his favorite target was pharmaceuticals. And so he knew how to get satellite imagery to figure out, you know, oh, here's the, this is the type of plant that this pharmaceutical company is building. And it always, you know, I could tell you what type of drugs they were using by the size of the plant. I could tell you what kinds of, you know, like how big the production was going to be. Like all sorts of weird, crazy stuff. And he could see it from satellite imagery. And like, again, that would take weeks or months potentially to pull out all together. Now, minutes, minutes. And far, far beyond all of that. So much information. So anyways, that reconnaissance part is huge and it collapsed down into minutes. Weaponization is increasingly scaffolded through open source models and frameworks. What's the metasploit, right? Metasploit was a big deal, you know, was like eight years ago now. And now it's like metasploit on steroids. You know, exploit code can adapt and debug itself in near real time now. Phishing campaigns can now be personalized. at scale using language models. Okay, and let me tell you, with the language models getting better, the crappy grammar and spelling errors are starting to go away, right? You know, they're trained on specific individuals, writing style, professional relationships, personal interests, all of it's scraped from LinkedIn, social media accounts, public records.
Renee:Okay, that. That right there. Every time you put your resume on LinkedIn and you say, I'm currently employed here. Here's all the equipment I'm certified in. Here's everything we use. You're just handing it to hackers. Now they know what your infrastructure looks like and they don't even have to try. Like at least to make them work for it, right?
Marc:Yeah, I know. When I was at the big bank, the whole time I was at the big bank, I didn't put that I worked at that bank on the LinkedIn until after I left.
Renee:Yeah, I mean, it's a huge amount of data.
Marc:I had thousands, about thousands of people in my organization. I had well beyond a half a billion dollars in spend every year. I know I was a target because I was targeted. So like you just can't, you can't make yourself an easy target. Anyways, the, you know, if you, the delivery and exploitation phases, you know, benefit all from the same automation and it's just getting better. The defensive side mirrors this exactly, which is good, but also not good. Autonomous detection systems, automated response playbooks, AI driven triage and prioritization. Both sides of the engagement are increasingly operating at agent speed, machine speed. Human oversight is lagging completely behind the temple of the interaction. The kill chain used to be a linear sequence with human decision points at every stage. Now it's just a feedback loop where the loop can complete faster than a human can actually read a status report.
Renee:So in 1983, the fear was that the machine might misinterpret a simulation as reality and initiate a catastrophic sequence of events before a human could intervene. And in 2026, the question we should be asking is what happens when machines are interacting with machines across distributed systems at speed? Where human intervention isn't just slow, it's structurally impossible given the architecture because speed doesn't change efficiency. Speed changes risk and it changes the nature of what oversight even means, right?
Marc:We haven't seen one in a while, but, you know, remember the flash crashes, right? You know, these algorithmic trading systems go. Oh, yeah, they just blow up. They do their thing, and you have massive swings, hundreds of points up and down with flash crash. That stuff happens before a human can do anything, before regulator governance. It's like, oh, we're going to freeze the market. It's like, well, it already happened. It's done. Do we freeze the market now? What do you do? It's really scary. So anyways, all right. So here's a question. Do we stop being afraid of ai because it stopped looking like a missile and started looking like a chatbot
Renee:Okay it's a good one okay yes and i think it's because we're inherently dumb like i just think it's funny that human nature is because you know you know. Missiles are cinematic, right? They have smoke trails. They leave clear, you know, visual grammar of danger. And we've had decades of film and television to help us internalize that. When something, you know, looks like a missile, our threat detection instincts activate appropriately. But chatbots, dude, they're polite. They say things like, happy to help. You're so smart, Renee. Let me think about that for a minute. And then that reduction in friction, it's reducing the cognitive signals that would normally make you say, I shouldn't trust this. And maybe I shouldn't give it that much power over me, right? We are evolutionary wired to be afraid of fire. We are not in any meaningful sense at all wired to be afraid of autocomplete. But the structural power is still there. The capability is still there. It's just wearing much better branding. And you know what? It's because we're stupid human beings. You're being polite. Like, Renee, can I have $50,000? Sure. Like, Renee, give me your $50,000. That seems like a threat. But ask me nicely. And sure, right? Like, yeah, we fall for it every time. And I think they build the chatbots for exactly that reason.
Marc:Yeah, of course. Of course. I think, you know, that's definitely the way to use your example of the $50,000, you know, dollars, like if you were at the ATM, you know, or whatever, in a bank or, you know, or who knows what somebody was held you at gunpoint and said, you know, give me all your money in your account. Like, of course, you're, you're, you know, you know, that that person intends to do bad, you know, whether or not you give them the money is a different sort of, it's a different thing. If somebody, you know, that you know and you trusted and you had built a rapport with said, Renee, I really need you to do this. Yeah, invest in my,
Renee:Invest with me. Can I, like, give me $50,000. That's a different thing.
Marc:At the very least, you would say, oh, well, what do you need that for? Or can we talk about that? Or, like, tell me what you're trying to do. You're not, at gunpoint, you're going to run, you know. Right. At this sort of polite UX process with AI, you know, it'd be like, oh, well, I don't know. Maybe. Should I give you $50,000? Am I a partner now?
Renee:Like, let's talk about it. Like, yeah, yeah, yeah, yeah.
Marc:Yeah. So that UX of contemporary AI systems is specifically optimized to feel collaborative rather than autonomous. The interface design communicates. Oh, I, like, I needed to do something. I'm looking at my tractor. I have some electrical problems with my tractor. So I used ChatGPT to help me with it. Because Claude was no good with the tractor, you know, with tractors. I don't know why. But, but. It's too higher order thinking, Claude. It probably is.
Renee:He's got other things to think about.
Marc:It was an electrical issue. It turns out there are some more common electrical issues. And I couldn't uncover that information using Google search without getting inundated with stupid sponsor crap. So, so I get this and, you know, I get to the end of the conversation and it's like, oh, would you like me to look up this? Or would you like me to explain common issues? Or would you like to, I'm thinking, gosh, man, it's just like, like, you know, it wants, it wants to keep me engaged all the time and stuff. It's like, I instantly deleted the chat. But, you know, that interface, it's designed to communicate. You know it's a tool you're wielding rather than you know rather than something that is kind of abstracted here. I'm a system with significant capabilities operating under normal direction. Those are different things with different governance implications. The UX doesn't distinguish between them. A language model that can generate functional exploit code presents itself with the same conversational warmth as a language model that's helping you write a birthday card.
Renee:And that's the riff, right? We stopped being afraid of it when it learned to say please. When it uses my name, I'm all like, aw. Like, no, that's manipulative crap, right? In War Games, Escalation had a very legible visual vocabulary. Blinking lights, DEF CON alerts, big board with lines showing missile trajectories. The aesthetic was designed to communicate danger. And it worked, right? It worked. The audience understood viscerally that that situation was deteriorating fast. And the contemporary version of Escalation, a synthetic voice that passes an identity verification check and authorizes a fraudulent transaction, an autonomous system that detects anonymous behavior, and an infrastructure network that initiates a defensive response that's inadvertently cascading into a second system and then a third system, and then model-generated script that exploits a misconfigured cloud storage bucket, which happened during the 2016 election. And that filtrates data before the logging system can even register the access request. None of that looks like a missile. None of it has visual grammar that activates our threat detection instincts. There's a no DEF CON countdown. There's just a series of API calls that nobody can see, nobody understands, completed successfully, and somewhere downstream that damage is already done.
Marc:Renee, would you like me to finish this metasploit package for you? Right.
Renee:Want me not to explain it to you?
Marc:Yeah.
Renee:Yes. Yes. I would like you not to explain it to me.
Marc:Would you like me to deliver this attack package now? Oh, yeah. Sure. Go ahead.
Renee:Sure. Because, like, I'm the typical dumb human. Like, if it said in the dictionary, it said typical dumb human, it would be like my face.
Marc:No, no. So, you know, the absence, though, of visible consequence at the point of action is one of the genuinely novel features of this risk landscape that we're living with. So with physical systems, cause and effect are often proximate in time and space. You actually see what happened. With distributed automated systems, the casual chain can be long and direct and difficult to reconstruct after the fact. Attribution becomes complicated. Accountability is diffused, right? And the speed at which these chains can complete means that by the time anyone has a clear picture of what happened, the window for intervention is long since closed, right? When we talk about the flash crashes, boom, boom, boom, gone.
Renee:Yeah, and that's why it takes everybody so long to give an attribute to a hack. Like it'll be like a half a year later, like it was the North Koreans. Like, well, what took so long? Like, you have no idea. They went through they went through like U.S. universities and then they pinged off of Canadian stuff and then they pinged over to Europe and then they finally come down and attack Sony. Like like it really is like that. I mean, they're trying everybody's trying to hide where where it originates from. It takes a long time to figure that stuff out. So, yeah, it might have taken them like three days to package the exploit and send it out. It will take us half a year to be able to definitively say where it came from. Yeah, that's the world we live in.
Marc:Yeah. From a systems design perspective, the core problem here is that what you might call control latency. So governance structures, meaning policies, review processes, oversight mechanisms, and accountability frameworks that we use to manage risk in complex organizations. They're all designed with the assumption that the systems they govern move roughly human speed. A quarterly board review. Is the board irrelevant in the era of AI? You know, if you've got quarterly board meetings, what are they going to contribute at the speed of AI? A change management process requires human sign-off at each stage. You know, it makes sense if the stages of a change take days or weeks to complete, right? When you layer autonomous tools onto those systems, tools that can initiate and execute and complete significant actions in milliseconds, right? You have a mismatch between the temple of action and the temple of oversight. Right. It's not just a gap. It's it's it's an incompatible structure. You're not going to get anywhere with that.
Renee:OK, and here's where I think War Games was genuinely prophetic in a way that went beyond its immediate Cold War context. The lesson at the end of the film was one that Whopper arrives at running through every possible nuclear scenario and finding no path to victory is that some systems should not be optimized for autonomous execution. I mean, that's where Whopper gets to in the end. Right. Like, sometimes you just shouldn't play. Like, this is a dumb game. Like, why are we doing this? And the value of human involvement isn't efficiency. It's not that humans are any better at making decisions than machines. I mean, quite frankly, MIT did a study of that. It was like, humans make bad decisions by themselves. Machines make bad decisions by themselves. But machines and humans make exceptionally good decisions. It's kind of weird, right?
Marc:They're good partners, yeah.
Renee:Right? We're really good at that. The value of human involvement is the fact that we do slow it down. We pause, right? The only winning move is not to play isn't nihilism. It's a design principle, right? Again, this is a design movie. It's not a Cold War movie. It's an argument for building interruption into systems rather than optimizing it out. We call that now human in the loop, right? That's what we call it now.
Marc:Yeah. Well, I mean, you just think about, we talked about this last time, right? We talked about Tamagotchis and stuff, like context. You know, the amount of context that a human can hold is decades. Yeah. The amount of context that, you know, these models can hold is lines.
Renee:Exactly seven and a half minutes before it says, I'm going to compress all the context. And then I'll be back to you in three minutes and I'll remember none of it. Like, that's exactly how these, once they compress and they're like, what do you remember now? Who are you? Like, that's, yes.
Marc:It's not that bad. Come on.
Renee:Almost that bad sometimes.
Marc:But the challenge is, you know, interrupting, you know, slowing things down, changing the pace. It has a cost, right? The competitive pressure in most domains runs directly against governance pressure. How many times, like, I hear this all the time. LinkedIn pundits throw up all over the EU. The EU's got too much regulation, too much governance. We can't innovate. It makes everything slow. I hate that. I hate that line of talk, but whatever. Organizations that build in more oversight operate more slowly than organizations that don't, at least in the short term. And in markets where speed is a competitive advantage, The incentive structure pushes systematically toward removing the friction, removing governance, right, that oversight requires. It's the same tension that produced financial crisis before adequate regulatory frameworks were in place. Do we have adequate regulatory frameworks for financial systems, right, Rene? Just smile just smile it's the same tension that produced industrial safety disasters before mandatory inspection regimes right how many figures were lost before you know certain things had to be implemented the question you know is whether we learn from those patterns before after a significant failure event focuses collective attention on the problem
Renee:Well, OK, so you so when the steam engine was first built, there was no regulation. So people would just build them. People would build them in the garages. Right. And then they would put them onto boats. And if you lived by a river and you were looking for a fun date, you would just go down to the river and watch them explode because they always exploded. They never they didn't have any gauges on them to show they were too hot and they would just blow up. They were like it was so crazy at some point. And it was causing so many fires that finally someone says we have to standardize. You can't just build this crap everywhere. And we're tired of fishing dead bodies out of the river on date night. Like we're done. We're done. Right. But how many people had to die before we did that? Right. So we build oversight at human speed. We layered intelligence on systems that move faster than the meetings. Right. Like, let's sit down and talk about this change. Well, we could, except that the cloud already made the change. So now we're talking about something that already, and thank God it went okay, right?
Marc:Retroactive change records.
Renee:Yeah, change management, right? I love that. Yeah, yeah. All you're really doing is making sure the database matches whatever the hell you just did. And the response to that can't be moving the meeting faster because at some point the meeting can't go fast enough. The response has to be architectural. It has to be about designing systems that have interruption built into them and that have human oversight built into that structure rather than bolted onto the exterior as a compliance theater, right? It's the same basic principle we apply to aircraft design and pharmaceutical approval and nuclear power plant operation. The more consequential the system, the more deliberately you design it for the possibility of failure, and the more explicitly you build in the mechanism for catching it before it gets compromised. I mean, can you imagine if we treated nuclear power plants like they did the steam engine?
Marc:Okay well they've all blown up and so maybe
Renee:We should do something about that like that's that's crazy talk.
Marc:That's just crazy i mean i mean you know springfield nuclear plant did hire homer just saying talking
Renee:About compliance theater like there it is right there.
Marc:I mean he's got one button to push if something goes wrong and even then he messes it up yeah i mean this is this brings in a good point right like how how should you design Like if everything is moving so fast, how do you manage that? And, you know, there's not a great answer, but you have to design for interruption. You have to design for oversight. You have to design for places to pause. And they're not, it's like, that's not a default setting in our contemporary AI system architecture. You know, they require intentional choices that run against the optimization pressures. I love how the chatbots just kind of flow, you know, they just like, they like to, you know, create the flow. And we're moving away from, you know, like chat interactions, you know, and now completely autonomous agents built, you know, using these LLMs, you know, but those, but now you're taking out the human interaction of the chat process and just letting the thing flow. And, you know, that's not, that's not, that's not okay. You have to build that pause into it. But it's challenging to implement this technically and politically, right? The business is pushing to go faster. People don't care about risk and compliance as much as they care about revenue. I won't out her because I really do love her, but there's a woman that's a longtime tech board member and early employee at some of these early internet companies. And, you know, she said to me once, Mark, revenue hides a lot of sins.
Renee:It does.
Marc:And it does. It totally does. You know, and it's tough to have the political will to implement something like a pause in an automated, you know, whatever, AI chain. And it's, you know, it's difficult to get that process.
Renee:So here's what I struggle with. Like ever since I did my very first audit, like I, so I used to be an IT auditor for a while and I did the very first one and I thought, oh my God, this is the dumbest job ever. Like, why are we doing this? Why are we not just collecting the evidence, running it through like some kind of application that tells us whether it's good or bad and like call it a day. But we didn't really have the technology to pull that off. But now that we have API connectors to ERP platforms and to IT tools and everything else, like, well, why don't we just go get it? Like.
Marc:I don't have
Renee:To sit here and talk to you I don't have to go talk to the finance people I don't just go get it Like just give me access to that database I'll just go get it And now platforms are coming online Audit platforms that actually do that So now I'm at the point where I'm like.
Marc:Yeah, but I don't have an AI agent.
Renee:So why can't I just go get that and test it? And then it's tested. And then if there's a material weakness, then it can tell somebody, right? And if it passes, then nobody has to look at it because it passed it, right? And I think, and at that point, I struggle with, okay, I move the auditor out of the job of collecting the information, organizing the information, evaluating the information and signing off on it, right? I've removed so much of that process that now I have an auditor sitting in a chair saying, I agree that's the right data. I agree that's the right formula. I agree with that outcome. That one's good. Like, that's it. That's what an auditor does now, right? Right. And on one hand, I think, yes. But on the other hand, I think the institutional knowledge that that person. Brought to that having done all that work. Oh, right. Except I'll go back to the rest of the business is running so fast that if that's how slow we're working, then we're really not helping at all anymore. Right. So like I think like there are ways where this is like in my own life right here today, I can tell you, yeah, we're running into it right now. Like right now, I want to automate the entire audit like business. I just don't want it to be a thing anymore, except that I need a human in that loop and I need that human to be consequential. So, yeah, I'm struggling with it, dude. Yeah, I get it.
Marc:I get it. But I think the issue that I see is that the speed at which the agents act and then the speed at which you're able to respond, if you do need to respond, if you do need to pull a switch, if you do need to pause, they're completely different timescales. So that mismatch is really tough. And I think in your audit use case, Hopefully, in theory, like if you could automate a bunch of this stuff, then an auditor themselves should have more time to actually do the job that they should be doing, right?
Renee:Yeah. And I think they can pivot to strategy, right? We should be thinking about, like they are smart people who are high order thinkers, right? And they're usually CPAs. Like, how are they not, you know, talking about, like, other stuff with people? Yeah. So I feel like the dumb work of audit should be automated. But I do need a human in the loop. There's not a single regulator on Earth that would be like, oh, yeah, turn that over to an algorithm. You're going to be good. Right? Yeah. It's not going to work.
Marc:Human accountability.
Renee:A machine can't be accountable. Yeah. That's it.
Marc:A machine can't be accountable.
Renee:I need someone to blame. I can't blame it on the algorithm. Right? Okay.
Marc:So here's where. Yeah. Go ahead. Go ahead. Go ahead. I was going to say, you know, we constantly talk about, you know, algorithms and models making decisions. But, like, I can see that, yes, we allow algorithms and models to make decisions. But do they actually make decisions? I don't know. No, not really. No, no, they're not smart. No. They can't make a decision. They can't make a decision.
Renee:No, I made that decision. I said, if it meets this threshold, then this happens. It's coded. But as it's going out and doing crazy things, there's got to be a way to say, here's who's accountable to that, right?
Marc:I still need people to be accountable. Of course.
Renee:Okay, so here's where the film stops being a metaphor and starts being a design document. Because everything we've been talking about in the context of cybersecurity and enterprise AI governance applies... With considerably more force and considerably more urgency when the system in question isn't deciding whether to approve a loan or flag a phishing email. It's deciding whether to engage a target. Whopper was fiction in 1983. The systems that it's describing are not fiction in 2026. They are deployed, they're operational, and the governance questions that we're raising are no longer hypothetical.
Marc:Yeah. Yeah, so let's just talk about what autonomous systems actually look like on the contemporary battlefield, because public understanding of this is shaped almost entirely by science fiction, and it's worth being precise on this, right? Yeah. Yeah, Terminator, right?
Renee:Right, yeah. Yeah, that's not what it's like.
Marc:That's not what it's like. The category is usually called laws, lethal autonomous weapon systems, and it covers a whole spectrum. At one end, you have systems that are autonomous in a narrow and bounded sense, a missile that uses onboard sensors to guide itself to a target that a human designated, laser-guided systems, smart bombs, whatever. At the other end, you have systems capable of selecting and engaging targets without a human in the decision loop at all. Most of what's currently deployed sits somewhere in the middle, but the trajectory is clearly toward the far end of that spectrum, and it's moving faster than the policy framework's designed to constrain it.
Renee:The drone wars gave everyone a specific mental image of what the autonomous battlefield technology looks like. And that it's already outdated, right? The operational reality in current conflicts has moved well past the model. So remember how you used to think it was a kid sitting at a TV screen with his joystick, like looking for the thing and it pushes the button? And I mean, that's how we remember that. But drone swarms. Like if you've ever been to, like here, like you can't have fireworks in California, right? Because it causes wildfires. So instead they do a lot of drone shows. I love it. Yeah, right, where the drones go. Those are fully autonomous drones that are flying next to each other and a whole thing being run by a computer, right? There's dozens or hundreds of low-cost autonomous units. They're all coordinating with each other. And what they're doing on the battlefield is they're trying to overwhelm defenses, and they've been deployed in actual combat operations. These systems can't be operated by a human at the end of, like, there can't be 500 of you flying right next to each other. Like, all right, we got this. We got this, right? Like, they're fully autonomous, and there's, you know, autonomy isn't a design choice anymore. It's an operational requirement. You either give the swarm autonomous decision-making or you don't use the swarms, right? And so, like, we're well beyond the idea that this is all theoretical. Like, if you, like, and the swarms are doing interesting things. Like, they don't all have bombs on them or guns on them. Sometimes they have Wi-Fi jammers so no one can, like, or microwave jammers so no one can communicate with anyone else as long as that swarm is around. Like, that's what they're doing with that stuff. And it's real. It's real on the battlefield today.
Marc:Yeah, well, I mean, weapons are heavy, you know?
Renee:Yeah, right, right.
Marc:You know, an electronic jammer doesn't have to be as heavy as, you know, a bomb, so... And I've seen, you know, if you want to see some, not, not the swarming stuff, but low cost drone usage, there's, you know, there's a whole subreddit on Reddit with the Ukraine stuff where they've hacked, they've hacked these, you know, drones and then they hack them up to like literally drop a grenade, you know, like a normal field grenade. And, you know, it's not autonomous. It's not like what we're talking about here, but like, that's, that's like the garage stuff, you know? So, if you think about what the capability is, you go all the way over to the other side of the spectrum, out of the garage and into the lab, like, the capabilities are massive. Yeah. So, the speed argument here is the same one we had about the cyber kill chain, though. But the stakes on this are completely different. You know, people's lives are, you know, at stake here. And cyber, well, you know, cybersecurity has, you know, there could be, you know, life. Yeah. True. True, but I think it's a different thing on the battlefield. In cybersecurity, the argument for autonomous response is that human analysts can't process threat data fast enough to respond at the tempo of machine speed attacks. Okay, valid argument. Real potential operational problems there. On the battlefield, the argument for autonomous engagement is that the tempo of modern combat, particularly in electronic warfare environments, where communications can be jammed or degraded, means that waiting for human authorization creates an asymmetric disadvantage. An adversary whose systems can identify and engage a target in milliseconds can defeat an adversary whose systems have to route that decision through a human operator every time. The military logic is coherent on this, right? Humanitarian and legal implications are what keep international law scholars up at night.
Renee:Well, international humanitarian law requires that any use of lethal force distinguished between combatants and civilians be proportionate to the military objective and take feasible precautions to minimize civilian harm. Those legal standards that were written around the assumption of a human decision-maker, right, who can exercise judgment in context. Whether an autonomous system can meet that standard is genuinely contested. I seriously was contesting it just the other day, and not just philosophically. It's contested in international courts, in UN working groups, in defense ministries. No one has resolved it. And in the meantime, the systems are deployed, the conflicts are ongoing, and the decisions are being made at machine speed in environments that are far more ambiguous than any training set ever anticipated.
Marc:Yeah, the AI targeting problem is where this gets sort of technically specific on this. You know, contemporary military AI systems use computer vision and machine learning models to identify targets, classify threats, and recommend or initiate engagement. They're not doing the traffic grid.
Renee:Right, no, they're not. They're not.
Marc:That would be very scary if all of a sudden the Google image grid started showing up with, you know.
Renee:Right, the caption was like.
Marc:Yeah. Show all the civilians in the image.
Renee:Oh, see. I don't need my email that bad, you jerks. Yeah, that's what I would be saying.
Marc:Yeah. So these models are trained on data sets of labeled images and sensor data. The fundamental limitation of any machine learning model applies here with lethal consequences. Okay, let me tell you this little side story on machine learning. I think it's a relatively famous example. But there was, you know, there's a model that was trying to determine the difference between wolves and huskies, I think it was, you know, model would say wolf, husky, husky, wolf, wolf, wolf, wolf, wolf, husky, husky, whatever. And then it gets to something and it says the wrong thing. And I think it wasn't either a wolf or a husky, but it was something else. It was like a rabbit with snow. And it turns out that the model was like, oh, well, if there's snow, it must be this animal. And if there's not snow, it must be this animal. So it wasn't trained at all on the animal itself, but the presence of snow in the images. So, it really matters what the image data set is. Right. Oh, my God. Yeah. That's why, like, you know, we say these models have lethal consequences. Like, if you don't train the model correctly, like, it might be that the presence of, you know, that's a civilian versus that's not a civilian is, you know, that they're carrying a bag.
Renee:Dude, I wore camouflage pants today. Like, that's all it could be, right? And I have nothing to do with, I'm not armed. I don't have anything to do with anything. But because I'm wearing camo today, I'm an official combatant.
Marc:Yeah. Right? Yeah. Bad news, right? A model trained to identify a specific type of military vehicle will have a confidence score attached to every classification it makes. In a control test environment, that confidence score, it's meaningful. In a contested, degraded, electronically jammed combat environment with partial sensor data, and adversaries who are actively trying to deceive the system, that confidence score is a guess rather than a number attached to it. And the number feels authoritative, right? It's confident because the AI, the machine learning model says, yeah, but we know that a guess is still just a guess.
Renee:And that's when Whopper stops being Cold War artifact and becomes a live case study. Whopper's failure mode wasn't that the system was malicious or broken. The problem is that its model of the situation was wrong. It had no mechanism for recognizing its model was wrong, and no one had built in an adequate interruption point into the architecture. Every autonomous battlefield system deployed today carries a version of that same risk. Let me tell you something really quick, because we've been talking about this a lot, the AI governance weirdos on LinkedIn. There's a bunch of us that are blunt, and they're like, we're fed up with this. And so it sent me down a rabbit hole of, with, you know, Anthropic saying, I don't, I don't want to do this. You can't use our platform to do this. And there's a very good reason why it's because, you know.
Marc:Air Force tested this stuff.
Renee:And what they found out was it's wrong 40 percent of the time in targeting. Like it's wrong. It's just flat out wrong. Right. And so for drop a thousand bombs, 400 of them are going to hit the wrong thing. That doesn't seem like the right thing to do. That model's not good enough yet to be used in. We're not talking about two percent failure rate or four percent failure or even eight. I mean, maybe you could say that that's world class, but like at the rate that it's failing, like like you really have to think about that stuff. Right. It is a different version of that risk. And there's a there's a way different like thing there. And and the human in the loop. So we've gone from having 2000 soldiers trying to figure out what to do with all this stuff to 20. Like and while that may be faster, that level of oversight probably isn't adequate. yeah.
Marc:Yeah oh man it's like what even to say like we don't have to say
Renee:Anything i'm just saying that like that that right there is like a reason to pause and say you know what we should we should really think about this this is not and it's one thing to say i'm gonna let it give me a target list of a thousand and now i'm gonna let humans figure out if those are okay because i think there There is a level of understanding that a human being has that, you know, and and a conscience and, you know, an understanding ethics and understanding of international law and military rules of conflict that aren't built into that model at all. Yeah. Not at all. Right. And so, yeah, there is. I mean, if that's if that's the only reason to slow slow down, that's the right reason for sure.
Marc:There's there's also escalation dynamics, you know, in play here. And that goes beyond individual targeting decisions. And this one maps almost perfectly onto, you know, War Games' central plot. When both sides of a conflict deploy autonomous systems operating at machine speed, You get the potential for what strategists call an escalation spiral that no human operator initiated or intended. This is the flash crash scenario. System A detects an action by system B and classifies it as a threat. System A responds autonomously. System B detects system A's response and classifies it as an escalation. System B responds autonomously at a higher threshold. The cycle completes in seconds. The humans on both sides are watching a dashboard that's updating faster than they can interpret it. And by the time anyone with decision authority has a clear picture of what happened, the situation has already moved several steps beyond what any human could have authorized. Flash crashes in financial markets, the same principle, trillion-dollar swings in asset value in minutes. On a battlefield, the equivalent, it doesn't reverse when the exchange closes.
Renee:Yeah, and here's what I think is the most uncomfortable parallel to war games. The one the film was actually warning about, that we somehow failed to absorb as a civilization, the scenario that almost ends the world in the film is not initiated by a decision to go to war. It's initiated by a misclassification. A teenager trying to play a game is interpreted by an autonomous system as a Soviet first strike because the system has no way to ask, wait, does this actually make sense given everything else I know about the current geopolitical situation? Nope.
Marc:Pattern matches, it classifies,
Renee:It followed by a cascade of logically coherent but contextually catastrophic automated responses that, you know, is not a 1983 problem. That's a problem we're actively building at scale on real battlefields with real weapons in 2026. The film told us exactly what could go wrong. And you know what? We watched it and said, great movie. And then we We built the thing anyway. We built the thing anyway.
Marc:I mean, that's not what we always do. Which we always do.
Renee:That's what this whole podcast has always been about. We know better.
Marc:We did it anyway. We're going to do it anyway. Yeah. Yeah. To be fair, though, to the people building, you know, to be building this stuff, if the pressure to build it is real. Unilateral restraint in a multi-party competitive environment is not a stable equilibrium. If one major military power deploys autonomous systems that provide a decisive operational advantage, the others follow or accept strategic disadvantage. It's not cynicism. That's how arms competition works. It's, you know, I think about this a lot, you know, in a historical context, right? The crossbow versus the longbow, you know, tanks versus anti-tank, you know, helicopters versus tanks, different types of planes. Like each one, each generation is designed to, you know, defend or attack the previous generation, you know, attack or defense structure. It's just... It just never ends. It just never ends. Anyways, but the answer to that dynamic historically has been arms of control frameworks, treaties. You can't do this type of thing. You can't do this type of thing. And international agreements that create coordinated restraint because unilateral restraint is not rational. We have those frameworks for chemical weapons and biological weapons and you know, to some degree, nuclear weapons. We do not have them for autonomous weapon systems today yet. Probably, I would think, you know, for a long time. Conversations are happening, which is good. Frameworks are not in place and the systems are already deployed.
Renee:Which brings us back to the only winning move, not as a platitude, right, as a genuine design and policy question. Whopper arrived at the conclusion by running every scenario to completion and finding that none of them produced an outcome that would be called winning by any meaningful definition, right? The machine had to exhaust every possibility before it could recognize that the frame itself was wrong. We don't have the luxury of running every single scenario to completion on real battlefields before we decide that the governance framework, what that should look like, and why the current framework might actually be wrong. Yeah. Yeah, I mean, at the same time, right? I mean, this is progress. It's progress, dude. And here's what ultimately really upsets me in the long run is like, Like when we were using like drones and we were saying like, I'm going to launch you. You know, I gave you a coordinate. You know where to go. You know how to get there. You have GPS. You're going to know when you're there. And when you're there, you do what you do. You drop that bomb or whatever. That eventually turned into the GPS in my car. Right? Like that's what ultimately upsets me. So like one day, all this autonomous stuff on the battlefield will one day find its way into my kitchen somehow or find my way into it. And it becomes, like, a huge part of how we live our lives. And, you know, and that, if you just want to talk, like, what really disappoints me is that one day it just, you know, gets patented for the rest of us. And we all think, what an amazing world we live in. But, you know, it started somewhere else.
Marc:That's the whole chain of the, you know, military-industrial complex, right? All of that money turns into consumer products.
Renee:On this happy note, let's wrap it up, shall we? War Games was never really about a kid with a modem, even though a kid with a modem was the vessel the story needed to make it legible to a 1983 audience, right? It was about a machine optimizing toward an outcome with a perfect internal logic and zero contextual judgment. It was about the gap between the system it was designed to do and the people who built it, what they actually wanted. It was about the specific and durable danger of delegating consequential decisions to systems that are very good at following their instructions and not equipped to question whether the instructions are right. That fear didn't go away when the Cold War ended. It didn't go away when modems got faster. It just stopped wearing a bomber jacket and it started wearing a conversational interface with a friendly tone and a helpful disposition. Renee, you're up late. What do you want to talk about? That's what Claude said to me the other day. It was like 2 o'clock in the morning. Hey, Renee, you're up late. What do you want to talk about? I was just nothing.
Marc:Like, it was so weird to me.
Renee:I just, I just closed it. It was weird.
Marc:Yeah. I mean, okay. So because it's me, I'm going to have to say like.
Renee:Of course.
Marc:Like a pump the brakes on intelligence. Right. So, so we use this word intelligence. I want to be extra careful all the time. You know, when we apply it to these types of systems, you know, what we're, what we're talking about is generative pattern completion running at enormous scale. The systems are extraordinarily capable of producing outputs that look like reasoning and sound like understanding, but the underlying process is fundamentally different from what we mean when we talk about a human exercising judgment. Whopper didn't understand that it was about to end the world. It was processing all the inputs and generating outputs consistent with its training. So these contemporary large-scale AI systems are more capable versions of that same basic operation. They don't understand what they're doing any more than Whopper did. They're just doing it across a much larger surface area with a much larger, you know, data set, right? And much faster, more convincing presentation, confidence. And, you know, it's not an argument against using them. It's an argument for being precise about what they are and what they aren't. It's so, oh my gosh, I think this is the big problem. Like, would we even see the deployment of so much of the AI stuff in the battlefield if it wasn't, like, easy to use chat GPT, you know? Like, I won't name names right, but, like, let's just say, you know, particularly in the States, leadership for the Department of Defense— They're easily influenced about how, you know, this cool thing that I can experience and understand. Why don't I have that with all of my soldiers? And, like, but they should know better.
Renee:Yeah, and I think that every time we say it's smart, it's getting smarter. The more GPUs we use, the smarter it gets. That's not smart. That's not smart. It's inherently stupid. and that you would think that it's smart just means that it knows a tiny bit more than you do, right? Like that doesn't mean anything. Like again, like let's go back to what a probabilistic model really is. All it is is a guess. It's guessing. It's saying, you ask this word. Usually this word comes after it. Usually that word comes after it. I'll look at all three of these words. Well, usually this word comes after it. And you're using that on the battlefield to target stuff. Like if you really understood what a large language model was, you wouldn't let it plan a vacation for you because really it's not smart. And just because we keep doubling the size of the data centers and we can get larger and larger surfaces of information, like we ran out of information a long time ago. Like we're a critical mass at this point. We can't create new information long and like fast enough to teach that model anything better. And here's what's crazy. Like back in 2024 is when they hit the wall. So it used to be give me twice as much compute power, the model will get twice as good. Twice as much compute power, twice as good. The last time they did it, twice as much compute power, it only got 5% better. Like we've hit a wall a thousand times over. This is as good as it's going to get for a while. And here's what I'll say about that, right? It's still guessing. And if your bank regulator wouldn't accept that, I'm not sure why four-star generals do.
Marc:Exactly. Like this is the thing that you're trying to be nuts. Like Like the models that are being used to train image recognition models and, you know, threat models and that. Like, that's a person, you know, that it's targeting. That's a, you know, that's a school, that's a hospital. It could be a military target. Absolutely, that's true. But, like, it's a guess.
Renee:It's a guess.
Marc:It's a guess based on data, yes. But, like, we shouldn't outsource accountability to the model, you know? Yeah. Like, that's an ethical problem.
Renee:Oh, and here's what I think is terrible. We don't, actually. So we don't do that on the battlefield. Do you know who we outsource it to? That kid who has 30 seconds to decide whether this is right or not, and he says, okay. So, you know, that I think is what gets me in the long run. That it's not, you know, no one else is being held accountable except for that person in the chair where there used to be 2,000 of you and now there's 20. Like, you're responsible for that. You're responsible. And you've got 20 minutes to figure out if that's the right thing to do.
Marc:I totally agree with you. And I think that's why it's unfair to hold that person accountable.
Renee:Yeah, I agree.
Marc:So, you know, it's not fair to hold that 22-year-old kid accountable. It's just not.
Renee:I agree. It's not. Yeah. So, okay, so listen, you said it before, design for interruption, design for oversight, design for pause. Because escalation doesn't have to look like a missile trajectory on a big board in a war room. Sometimes it looks like a prompt completing successfully, right? Sometimes it looks like an automated response that was technically correct. And sometimes it looks exactly like everything working is intended right up until the moment it doesn't. The machine doesn't know the difference. The machine is not smart. We have to be. We have to know the difference and we have to be smart. Yes, that's what I, I think that's where we leave it. It's not smart. We have to be.
Marc:Yeah. Yeah. All right. Look, kind of heavy.
Renee:You have a lot of editing to do, dude.
Marc:Don't do this one on video.
Renee:You've got a ton to edit. I know.
Marc:It'd be too hard to do the video on this. I'll have to split it up because this is two hours of stuff here almost. So look, if you once war dialed a random phone number and felt, you know, your heart rate increase when something answered or it went, you know, we get that. We understand completely.
Renee:And if you've spent any part of your professional life trying to explain to a boardroom why governance frameworks designed in 2010 are not adequate for AI systems operating in 2026, this episode was for you specifically, and we hope it helps.
Marc:Yeah, the kid in the bedroom is still there, by the way. You know, he, she, they got a laptop and a cloud account and access to open source models that could do things the 1983 version of him or her or them, you know, could not have imagined. The infrastructure is still fragile in the places it matters most. And the machines are still very good at following their instructions. Deterministic outcomes still work, Renee. They still work. I know it's hard to believe. We can do things without AI. I'm just saying.
Renee:I'm all for deterministic layers in AI. I'm all for this stuff.
Marc:Yes, yes.
Renee:Right? I love seeing deterministic layers in AI. We need it. Like, that keeps it from hallucinating. It keeps it from wandering off the ranch. Like, yes, yes, yes, yes, yes, yes.
Marc:My favorite that says, make no mistakes. Right?
Renee:Because it's so good at that.
Marc:It's so good at that. Okay. Okay. Sure, Mark.
Renee:Done. Done. This is the Nostalgic Nerds, where 1980 hacker mythology meets the autonomous infrastructure it accidentally predicted. Thanks for tuning in. Subscribe, follow, share this with someone who remembers the sound of a dial-up handshake and now spends their days writing AI governance policy. I feel their pain every day. The through line is shorter than it looks, so we'll be back next week with more childhood tech and the adult problems it quietly set in motion. Thanks, Mark.
Marc:Thank you. Glass is warm beneath my hands Warmer than it should be
