When Will GenAI's "Big One" Happen?

I've come to appreciate having Claude as my little assistant, delegating tasks to it like a junior engineer. Writing code can be damn tedious. I do find myself getting too comfortable sometimes with accepting code that Claude wrote. It's a constant battle to stay vigilant in the face of convenience.

With business leaders being constantly bombarded with GenAI marketing, especially leaders of businesses with software engineers on their payroll, I do wonder if they don't quite understand the downsides of running arbitrary executable instructions on live servers that no human who works for them wrote or maybe even understands.

As software engineers, we're in a tough position where colleagues that "go faster" by using GenAI more aggressively are often rewarded by leaders. The downsides of getting that pull request merged a few hours quicker are diffuse and left for a later, mostly hypothetical reckoning.

This may be the only time I've ever quoted a LinkedIn post on this blog, but this one from Eric Wise appeared in my feed this week, and it stuck in my head:

When I lived in Portland, Oregon I was disturbed when locals told me about the "Big One", a mega-earthquake that was all but guaranteed to strike the Pacific Northwest at some unknown future date. According to everyone, it was a matter of when not if it would happen.

I've been thinking about when GenAI's "Big One" will hit. This is what Eric describes in his post above: the inevitable catastrophic moment when some vibe-code in production in a Very Important system causes something very, very bad to happen, very publicly, and no one knows how to fix it. Just like the earthquake version of the "Big One", I do believe that GenAI's "Big One" is all but guaranteed to happen. It's not a matter of if but when.

I've already seen with my own eyes bugs caused in production systems traced back to a GenAI-coded pull request that made it to real users, and the "author" didn't understand how it worked. These were relatively low-blast-radius bugs in the cases I've seen, but when will a bug like this cause an airplane to fall out of the sky, or a multinational bank to lose billions of its customers' money in a day?

Even lowering the stakes a bit, what will happen as the slop piles up in production software systems all over the world, at an ever-increasing rate? This stuff is not slowing down, code is merging into trunk faster than the world has ever seen, and only getting faster every day.

We used to joke about how retired COBOL programmers were getting hired back as consultants at insane fees to patch up the mainframe systems they worked on in the 1970s that still run the world of finance. What about all the code shipping each week in 2026 that no human understands? We're creating obscure systems at a rate the COBOL heroes of yore could never conceive. 

GenAI's "Big One" is certainly coming, but the "Little Ones" are happening every day.