It seems AI is becoming one of the most volatile and expensive dependencies in modern systems, and most organizations aren’t prepared for what comes next. I recently wrote an article for Modern CISO on AI cost volatility, offering observations and recommendations to mitigate this risk.
Token Ransom and High Cost
For years, we’ve been told to prepare for the cost of intelligence to crash to zero.
The narrative pushed in this tweet by Logan Kilpatrick is something I’ve called out in the past for its sheer ridiculousness. But many are seeing the light in the last few weeks. Everywhere you turn, AI services are getting more expensive. Every day, new providers are making announcements. One example below is from GitHub Copilot’s new pricing.
Even the all-you-can-eat AI buffet for $20 a month was always a myth. This was part of a larger narrative pushed by influencers, futurists, and AI leaders, but this narrative always made dollars and no sense.
What happens when you deploy solutions using these cutting-edge foundation models into production environments? You may end up with a dependency where you have a choice. Pay more or have the solution stop working. In the article, I refer to this as a token ransom.
Think of this as a token ransom. It’s scary to consider how the ransomware of the future may actually be an inflated token cost.
If companies aren’t prepared for these scenarios, they carry significant operational risk. In the article, I break down the issues with more examples and provide recommendations on how to start addressing them.
You can read the full write-up for Modern CISO here. Although framed toward security leaders, the advice is applicable beyond the cybersecurity space.
One interesting outcome of these price increases is that companies are very concerned. It seems no amount of security and reliability issues dissuaded these companies from chucking AI into everything, but the skyrocketing cost of AI may. I’ve heard far more grumblings about cost than security issues. Only time will tell.
Since this site focuses on risks and trade-offs rather than shiny, utopian use cases, there is some confusion about my thoughts on AI. Be it the AI of the present or the AI of the future. With this post, I stake out my position on AI and its advancement so I don’t have to keep restating it. Also, it’s good to write down your beliefs and confront yourself with them. Sometimes what you think you believe isn’t what you actually believe. Maybe I’m a secret AI bro after all. Utopia, here we come!
AI acceleration has become a cult or religion, and offering any criticism of its advancement is taken as a personal attack. In many ways, if you swapped out AI for cryptocurrency, the theme would revive a familiar tone.
Even with this post, I’ll undoubtedly be pegged as an AI hater because I don’t have pictures of myself lying prostrate in front of a pile of GPUs. Any time you shed light on hype or bullshit, people are willing to label you a hater. That’s the easiest thing for them to do, and it takes no mental effort and requires no skill. With that said, here we go.
On AI Advancement Summary
I’ve been using this image in my conference presentations since 2023. The focus of my talks is on risk, which means talking about problems and challenges most of the time rather than amazing use cases. I wanted to show the audience that I don’t hate the technology.
The problem with being in the middle is that both sides typically frame you as an extremist. You don’t hate the technology enough for one side or love it enough for the other. Realities on the ground typically hover somewhere in the middle between extreme claims. This isn’t rocket science.
For those who honestly don’t care about the rest of this post and made it this far, here’s a quick summary:
I’m not a skeptic, I’m a critic
Yes, I think today’s AI can be useful
Yes, there are some use cases that I’m hopeful for
No, I don’t think today’s AI is AGI
Yes, I think AGI is possible
No, I don’t think LLMs will lead to AGI
I think even AGI will have vulnerabilities
I’m not so sure about the concept of ASI or the intelligence explosion
I’m Not a Skeptic, I’m a Critic
In the current era, I’ve seen people frame themselves as skeptics either of technology or of AI. This is not how I frame myself. I consider myself a critic, not a skeptic. I’ve joked that I’m a hype critic. Throughout my career, I’ve offered criticism of the state on cybersecurity, emerging technologies, and, especially, product manufacturers and their claims.
I believe that technology does need a better class of criticism. The tech press, in large part, has abdicated its responsibility, choosing instead to mindlessly parrot opinions from tech leaders. The entire world has been a gigantic sycophantic feedback loop. This is something I’ve called out many times myself and something that Karl Bode calls “CEO Said A Thing!” Journalism.
Most people criticizing the state of technology are insufferable. The few valuable points they present are wrapped in politics, bullshit, and, in some cases, conspiracy theories. Their goals aren’t to effect change but to pander to their audience. The very people who need to hear these points are the very ones who would never listen to them in the first place.
I have no vested interest in any technology’s success or failure, and I’m certainly not pandering to an audience. Hell, if I wanted to cultivate a large following, the last thing I’d be spending time on is writing. I’d start a podcast or YouTube channel, align my content with people’s biases, and go all out, telling them what they want to hear. I’d also use AI to write my content to up my pace. It’s the new definition of “productivity.”
One of the criticisms I get is that I shouldn’t be listened to because I don’t love AI enough, which is a strange perspective. Would you really trust someone who is selling you something, or who is completely head over heels in love with a technology? Are you going to get honest criticism? Of course you wouldn’t. But the whole premise of that argument doesn’t make sense. That’s a lot like saying someone isn’t religious enough because they don’t have enough religious bumper stickers on their car.
Framing The Conversation
Much of the debate around future AI advancement revolves around two questions after a claim is made:
What specific technology is being discussed?
When will it arrive?
So much confusion is caused by not clarifying the two follow-up questions. Many throw out the term “AI” as a catch-all, referring to any technology, present or future. For example, the claim that AI will cure cancer. Okay, but what specific AI technology? Is it technology we have today? Some future technology that hasn’t been invented yet? And of course, most importantly, when will this happen? The precision matters.
When asking people to provide some precision regarding their claims, it’s not uncommon to find that people aren’t talking about AI at all. They are talking about magic. For others, they are purely saying “something” will happen at “some point.” Which is basically saying nothing. Back in January of 2024, I published a framework for making sense of human AI predictions, which goes into a bit more detail on this topic.
I do believe that many of the claims made by proponents will be realized at some point through future technological advancements, some even with the technology we have today. I’m certainly hopeful about cures for debilitating illnesses, and a lot of work has already been done. I don’t think we are miles away from seeing those results. This is an example of something I’m hopeful for. Call me an optimist??? However, I don’t know what technology, under what circumstances, or when.
I’m sure so many people thought it was inevitable that by 2015, we’d have hoverboards in common use after watching Back To The Future 2. Our perspective on technological advancement is often skewed and off by a wide margin. It’s always good to keep this in mind.
LLMs
I certainly don’t hate LLMs. I find LLMs useful for various tasks, mostly coding tasks, basic research, and troubleshooting. I occasionally will use them to generate some AI slop images for a blog post or conference presentation. Pinning down my exact usage is a bit hard, since LLMs aren’t my first port of call for every problem. After all, I value my critical thinking skills, skills that people these days seem content to discard.
I never use LLMs for common cognitive tasks and never have an LLM decide for me or write anything on my behalf. I also never have an LLM summarize something I’m trying to understand, because knowledge and understanding aren’t generated from bullet points. The friction is the point in so many tasks where we look to reduce it.
The hype with LLMs hasn’t been commensurate with the realities on the ground. LLMs certainly have their uses. Just like me, people are finding them valuable for a variety of tasks. In my own industry (cybersecurity), there are positive examples in offensive security, vulnerability identification, and assisting analysts in security operations centers. You can also tune LLMs more effectively for specific tasks, which will have a positive effect. However, there are limiting factors to LLMs.
The first is the cost of failure for the use case. LLMs have relatively high failure rates, and when connected in agentic systems, these failures can cascade through the system. Failures compound like interest, to use the words of Demis Hassabis. I mean, the thought of ChatGPT running air traffic control is terrifying.
Second, they are highly manipulable. This is why everyone from startups to hyperscalers has had their AI-based applications hacked. This fundamental manipulability is baked into how LLMs operate. It’s why we have things like prompt injection, and adding LLMs to applications increases their attack surface. This condition is why I’ve described AI Security as a misnomer in the age of generative AI. You aren’t defending the AI. You are defending the application or use case against the effects of adding AI. This is a different problem.
These two factors can be misleading, though. We don’t need AGI-level capabilities for LLMs to be useful or to replace people in their jobs. The moment an LLM-based system is mediocre enough to replace someone, companies will rush to replace people. This is especially true if the cost of failure is lower for a particular job. Although reports of recent layoffs are nothing but AI washing, we are getting a glimpse of what will happen once capabilities arrive.
My biggest concern with LLMs isn’t what they can do for people, it’s what they do to people. I believe we are vastly underestimating the negative cognitive impacts these tools are having and will have on people in the future.
My biggest concern with LLMs isn’t what they can do for people, it’s what they do to people.
AGI
I do believe that AGI is possible, but I don’t think that today’s LLMs will be what gets us there. When do I believe AGI will arrive? 15 to 20 years. Put my precision on this at about 70%. I put the likelihood of LLMs alone becoming AGI at about 15%. But keep in mind, these are mostly guesses guided by intuition and actualities. Caveat: I’m not involved in developing AGI, and the world is a complex place that defies predictions. However, I do believe some factors will confound advancements for a while.
First, I don’t feel LLMs will lead to AGI, and this is where all of the focus seems to be at the moment. Second, I think there is a massive AI investment bubble. The amount of money being invested is nowhere near the value created. This bubble will pop at some point, hopefully not spectacularly. Companies like OpenAI will very likely go out of business. They are hemorrhaging money, and their shares are becoming almost impossible to unload on the secondary market. I mean, they put out a statement about focusing on business, and then just bought a podcast. Not exactly a shining indicator of future success.
I bring this up because this crash will cause some reluctance to invest in the future. Maybe it won’t quite be an AI winter, but it will be an AI fall with colder weather and a lot fewer leaves on the trees. This may stall the advancement toward AGI.
I think some people think that LLMs will go away after the investment bubble pops, but this is nothing but wishful thinking on their part. LLMs are genuinely useful for certain tasks and will continue to be. Also, LLMs are so essential to some people’s identity now, you’ll have to pry them from their cold, dead hands.
When it arrives, I do believe AGI will have vulnerabilities, even if they are not immediately apparent. This would be especially true if AGI were built on today’s LLMs or if it weren’t a single large system but a network of systems. Once deployed, we’d be stuck with these vulnerabilities. This is a perspective I’ve shared publicly for years in my talks and keynotes. There may be something about generalizing to the world that contains inherent vulnerabilities. I have a draft post on this topic that I’ll publish in the future. Unfortunately, I have dozens of posts in draft and only so much time.
ASI and The Intelligence Explosion
Strangely, we haven’t even achieved AGI yet, but labs are already bragging about how we are close to artificial superintelligence (ASI). Okay, it’s not strange, that’s just how hype works. We seem to forget that ASI is a speculative technology, and speculative technology leads to speculative bullshit.
To sum it up, despite believing that AGI is possible, I’m not so sure about ASI. Or at least ASI as it’s traditionally been discussed. I’m not quite sure I can put my finger on exactly why. It’s more of an intuition I have rather than any one specific thing. Of course, I may be the one now talking nonsense.
I think my hesitancy stems from conceptions of ASI, the resources required, and the plateaus that would be encountered. We are told we get there by just packing in “more” and “better”, whatever the more and better happen to be, and this cycle will continue forever. But I don’t think we’ll scale our way there, and we still have to contend with the laws of physics and resource constraints. This is why Ray Kurzweil thinks we need to pave over the universe to create computronium.
I do believe that some recursive self-improvement is possible, but only to a point. Maybe we’ll get to something like an AGI+ but not ASI as it’s traditionally been discussed, with its planet-eating power requirements and its continual recursive self-improvement. However, there is one thing I can say for sure: something will be labeled ASI long before it’s possible. Maybe someone will buy a podcast to promote that perspective! Who knows.
Since I’m unsure if ASI, as it’s been defined, is even possible, I’ll put the odds of reaching ASI in the next 50 years at 10%. But feel free to chalk this up to me saying, “I don’t know,” and disregard everything I’ve said.
What Happens Next
I’ve left no doubt about my pessimism about what happens next and how it’s not good for humanity. Much of the content on this site focuses on that topic. And, no, I don’t think a super-capable AI will see humans as a nuisance and eliminate us. Sorry, Eliezer Yudkowsky, but our manifested problems will be much more mundane.
It’s we, humans, who plant the seeds of our own downfall. When massive unemployment occurs (which may happen well before reaching AGI), there will be no recourse. The so-called abundance movement won’t deliver the value it promises. Many will fall into the “sucks to be you” gap that I’ve defined previously. A segment of the population will remain pinned there, possibly for a generation. This is purely due to incentives and the reluctance or inability to do anything about it. I’ll have more to say on this in the future.
Also, people continue to cede their critical thinking skills to AI. By far my biggest concern is the collapse of culture amid homogenized AI outputs and people’s inability to think independently. I see people more concerned with collecting data than with understanding it. The idea that someone becomes wiser by collecting more data or by engineering a better retrieval system is nonsense. If you need an AI to tell you what you think or believe, you’ve made a fundamental error.
In a previous post, I mentioned the story of Calvisius Sabinus, who, in an attempt to appear learned, devised a shortcut. It didn’t work out so well for him, and this new strategy won’t for us. I have much more to say on this topic as well. But that’s all for now.
Talk to any tech bro, and they’ll be happy to tell you, scarcity is about to become a thing of the past. Abundance is coming and we are going to have so much awesomeness that we can’t even fathom how amazing it will be. Sam Altman recently said that society has been structured around managing scarcity, and now we need to get used to managing abundance. Hell yeah, bro! There’s just one problem. Well… there are many problems, but a big one that isn’t talked about is that nothing in abundance is worth anything. This fact alone can bring visions of utopian abundance crashing down like a house of cards in a tornado, and it’s a lesson we can learn from Napster. Yes, that Napster.
AI, Abundance, and the Future
We are told that scarcity is about to be in our rearview and the bright shiny utopia is out the windshield. With AI and robots doing all of the work, the cost of goods and services will crash to near zero. It’s yachts and Lambos for everyone! Oh, there’s just the small problem that nobody has a job. Damn, there goes my Lambo.
These takes are common, and you see them every day. For example, here’s a random one I saw this morning.
But this isn’t the only claim. They also claim it will cure all diseases and won’t just end poverty, but make everyone rich.
The “things will be free” argument is typically packed among heaps of bullshit, as seen in these examples. It’s intended to overwhelm your critical thinking skills. Let’s just acknowledge that none of this makes sense.
They state that people are going to lose their jobs, but don’t worry, because AI is going to make everything super awesome! They also claim that in this new environment, people are going to become obscenely rich by giving you free shit. Admittedly, I don’t have an MBA, but that doesn’t seem to be how business works.
I’ve covered this topic before in Techno-Communism and the Things Will Cost Nothing Fallacy. In that article, I attacked some of the core premises of this fallacy. I even covered the fact that, despite the premise, things will still cost something, and that nobody is investing in companies to produce zero-dollar goods and services. I may only lightly recover some of that ground, so please refer to the previous article for the core arguments.
This is an important topic to consider now because we will see better AI technologies. We will eventually get to AGI. Many of the nonsense claims people make about today’s LLMs will be true of this new technology, and the impacts will be far-reaching. We need to consider how this environment affects us as humans and our culture.
In this article, I focus on value and frame the lesson of Napster as it relates to abundance and the future.
Napster
I have to do some work here. Let me explain Napster to young millennials and Gen Z, who’ve always had reliable internet connections and never had to wait for anything. Napster was a peer-to-peer file-sharing application that mostly focused on music. If there was a song or album you wanted, you’d search for it and download the MP3s. The songs were encoded at varying quality levels, and depending on how many people had the files and their internet connection speeds, you may have to wait a day or two for the songs or album to download. Oh, and surprise! Sometimes the files you downloaded weren’t what they claimed to be at all.
A look back on the UI of Napster is enough to haunt your dreams.
I use the term “was” in reference to Napster, but apparently, they are still around and drumroll… They’ve pivoted into AI slop. Imagine that! Although you might think this is the lesson of this article, it’s not. It’s just a fun byproduct in the AI era. Enjoy.
Back in 2000, Metallica sued Napster. For clarification, Metallica was a band of aging gentlemen who sounded like Avenged Sevenfold. Sad but true.
At the time, I remembered thinking how backward Metallica looked, like they were just some aging rock band that didn’t understand technology. Most of all, they just looked greedy. People just wanted the freedom to use their music as they wished. Many downloaded MP3s of albums they already owned. For example, maybe you already had Metallica’s black album on tape or found downloading MP3s easier than ripping them from the CD you owned. CDs were also cumbersome, and taking a bunch with you on a road trip was a pain. Sure, people pirated music, but that was a minority. My perspective solidified as the space for digital music grew.
As digital music grew, it came with heavy-handed DRM (Digital Rights Management), which meant that, despite purchasing the music legally, you couldn’t use the music the way you wanted to or play it on devices you chose. Hell, Sony even went so far as to install malware on your computer in an attempt to stop piracy.
Then came streaming, and music essentially became free. The lesson that started with Napster now came full circle with services like Spotify and Apple Music. However, inheriting a gigantic problem.
Music Is Now Free and Worth Nothing
In the ignorance of my youth, I fell victim to a condition most young people do. I failed to see the big picture. When something is free, it is essentially worth nothing. Nobody values music today, and why should they? There was no friction to access it. Nobody had to go to a store and choose between one album and another. Nobody had to wait in line for a midnight release from their favorite artist. Nobody had to choose which albums to take with them on a road trip. Although this sounds like a major inconvenience, it enhanced the value of music for both the listener and the artist. These activities created loyalty and forged a bond.
Musicians were well aware that listeners had a choice when buying music and felt responsible for creating art to the best of their ability. Or at least, this concept was in the back of their mind. Artists also made money selling music, which gave them the freedom to spend more time on their craft, further enhancing their art/product. This environment didn’t require an artist to be as big as Taylor Swift to do so, either.
Purchasing music connected listeners to artists in a way that streaming doesn’t. When people bought an album, they listened to the album the way the artist intended. They could evoke emotion through ebbs and flows, taking people on a musical journey. With streaming, people just add things to playlists and hit shuffle. An artistic vision of an album is completely dismantled, chucked into the chaos of a musical vortex, spinning alongside Snoop Dogg and Conway Twitty.
Now, artists blast out music as fast as they can to increase their breadth because more music means more potential streams. Not releasing music at a steady cadence creates the impression that you’ll be forgotten in the vast sea of content.
Of course, artists now have to make money in other ways, spending less time on their music. They used to agonize over albums and songs, pouring their hearts and souls into them and taking the time to get things right. They also took creative chances. Now, music is collapsing into a formulaic structure where all music sounds the same, and artists bash the formula like the preprogrammed buttons in Mortal Kombat.
Music has become less something to be listened to and more something that provides background noise in daily life. It doesn’t take pride of place as an activity. It’s wallpaper.
Musical Abundance and Lost Value
What we have today is essentially musical abundance. There’s literally so much music on streaming platforms that people can’t find it. In 2024, it was reported that Spotify had over 100 million songs with over 60,000 new songs added per day. Want to guess just how much of that is good music? Certainly less than 1%, there is no way you can convince me that there are 1 million good songs on the platform. In this world of abundance, most things are shit.
As an artist, you basically have to give your music away. As a listener, music is nothing to you. If one song isn’t available, you just listen to something else without emotion. Music, possibly humanity’s very first art form and something that has meant so much to so many humans throughout history, is now stripped of its value and devoid of connection and meaning. In reality, musical abundance hasn’t led to positive outcomes for either the artist or the listener, and this is our lesson here.
In reality, musical abundance hasn’t led to positive outcomes for either the artist or the listener.
The healing effects, the satisfaction, the connection, the memories, and most of the factors that made music valuable and essential to humanity are now gone. Industrialized, homogenized, and mass-produced music has stripped the art of its value. For the far fewer than 1% of songs that aren’t terrible, we are left with momentary blips of completely forgettable, average sounds labeled as music that just happen to hit our ears. No wonder kids are rediscovering the classics.
There’s no going back. Once a technology is implemented, even if it makes things worse, there is no going back. We are stuck with it, which is why it’s so important to understand the trade-offs and implement mitigations before adopting a technology. We just can’t seem to do that.
Now, the music industry also has its share of blame here. It got greedy. A CD never should have cost anywhere near 20 dollars, unless it was a double album or some other special edition. $9.99 was a fair price and should have been the cost for an album. But that time has passed.
Abundance Math Doesn’t Make Sense
Let’s set a couple of foundations for abundance. The premise behind abundance is that companies will employ AI and robots instead of human workers. So, first and foremost, no jobs for most people on the planet.
We are told by futurists and AI bros that abundance will bring a utopia. That everything will be democratized. The cost of everything will be near zero, compute, goods, intelligence, development, and on and on. With so many things driven to essentially zero, not only does the math behind abundance not make sense, but its value as well, as we’ve seen with music.
With so many things driven to essentially zero, not only does the math behind abundance not make sense, but its value as well, as we’ve seen with music.
Let’s start off by addressing the use of the term democratize. Whenever an AI bro uses the term democratize, they actually mean something else: either devalue, degrade, or destroy. I call these the 3 D’s, and I’ve covered this before. But let’s get back to cost and value.
It doesn’t matter if the cost of things is driven to near zero. If they cost something and you have nothing, you still can’t afford it. If a brand new car costs a dollar and you don’t have a dollar, its inexpensiveness doesn’t change the fact that you have to walk. Assistance programs like UBI will hardly have us living lives of luxury.
When people have little to no money, they tend to focus on pure necessities. Pretty much the entire global economy is focused on selling us stuff that we want, not necessarily things we need. That means in this situation of supposed abundance, most companies on the planet disappear.
This shift will rewire our values. I’ve discussed how this may happen in retail through the implementation of agentic shopping. This rewiring isn’t lost on people who are thinking about these issues. The CEO of Walmart resigned because of AI’s potential to upend retail. If successful, more rewiring is on the way and will upend the entire fabric of the modern world. Just imagine, instead of lusting after material things, the dominant belief in the world was meditation and inner peace. Not great for business.
In a world of abundance, many large companies probably won’t exist. Diversification and competition will eat them alive. The abundance of tools and techniques will allow for quick imitation. Sure, some will remain, kind of like the company store in a mining town, but the economic landscape will look vastly different.
We are told that, despite things being so cheap, companies will make money from volume. But once again, with distribution spread across many small companies, it’s hard to see how that amounts to anything more than table scraps. This begs the question, then, where will the money for UBI come from? Even worse, instead of UBI, we may end up with company vouchers, forcing us to buy from a single company or to obtain vouchers for things we don’t need.
Abundance Equals Sameness
Henry Ford is quoted as saying of the Model T that a customer can have any color they want, as long as it’s black. There’s a lesson here for technological abundance. Many are laboring under the delusion that a world of abundance looks like the world of today, just with more. But the reality is this world will look totally different.
The world of technological abundance laid out before us isn’t a world of diverse beauty with green grass and open skies where people spend their days writing poetry and contemplating their existence in the universe. It’s a homogenized world of sameness where everything collapses into uniformity because this uniformity is predictable and can be mass-produced at scale. Individuality and one-offs are unpredictable and costly, and thus must be crushed. It’s a world where humans become little more than a burden.
Sure, people will try to control for this uniformity in the ways that they can. For example, if a family has a 3D printer at home, they have some control over what they print. However, they still need the resources to buy the printer and printing materials, and most would rely on the available designs. This concept of a utopia is starting to smell pretty rank.
New Business? Maybe.
I’m sure it will be said that I’m not envisioning the new businesses that will sprout up and how humans will change their behavior. Sure, the world is a complex place that defies prediction. I acknowledge that adaptation is possible, and surely, humans will change their behavior. They won’t have a choice. But it isn’t going to turn out the way people think.
Just shooting from the hip here, but a world in which people don’t have money and things are essentially free doesn’t seem like a robust environment for business. You won’t see a plethora of new startups all angling for that gigantic pot of nonexistent money.
You won’t see a plethora of new startups all angling for that pot of nonexistent money.
People invest in companies on the hopes of a big return. They take chances and are willing to take losses, but the gambler loses big in this new environment. Because even an investor with money and a successful business bet basically converts that investment into less money. In this environment, there are no incentives to improve, and everything will plateau at mediocrity to preserve existing margins.
Another thing to consider is that this environment is inherently fragile. Any problem, no matter how small, could cause a company to lose money since the margins would be so tight. Any amount of downtime, errors, or integrity issues, whether accidental or intentional, could be catastrophic. The predictable answer from the AI bro is: “But the AI won’t make mistakes.” To which I say, good luck with that.
Now, is it possible for us to evolve beyond money or capital toward a society that aligns its value with other meaningful activities and goals? Sure, it’s possible. But, once again, this wouldn’t be good for the business bros who are pushing this fallacy the hardest. In a world where abundance is truly successful and is beneficial to humanity, it’s bad for business.
Complexities and Negative Impacts
A full conversation on the negative impacts is outside the scope of this article. However, I have covered some of these before.
I’d argue that these people vastly underestimate the world’s complexities and the sheer number of negative impacts this change brings, not just on the economics but on humanity as a whole. There are basically only one or two ways this can kind of go right, and an incalculable number of ways it can go wrong. For example, this is an environment tailor-made for surveillance and totalitarianism. How can it not when your existence is completely dependent on external factors like the state or the generosity of a benefactor?
If you think that people will tolerate a few trillionaires while the world suffers mass unemployment, you are dead wrong. This is so obvious it shouldn’t need to be stated. I can completely envision a group calling themselves The Children of Ludd wreaking havoc. Only their anger won’t be isolated to the machines in data centers and their embodiments out in the world. They’ll direct their anger at the people they see exploiting the situation as well. It won’t be pretty.
I can completely envision a group calling themselves The Children of Ludd wreaking havoc.
In many cases, this new environment will make people far more tribal and extreme and willing to believe just about anything, but this is a topic for another day.
Conclusion
The vision of abundance defined by the tech bros is an absolute fallacy. It’s a world in which AI is talked about more like magic and less like an actual technology. From this tap spouts outlandish claims with no foundation in reality. As we’ve seen, even when abundance works, it has a degradation effect.
A world of successful abundance appears to be bad for business, which raises the question: Why are so many business leaders pushing this concept? If I had to guess, it’s that they hope to make their money before this big crash and leave humanity to pick up the pieces.
Recently, Google, along with Shopify, Etsy, Wayfair, and Target, created Universal Commerce Protocol. A protocol that retailers can use in their AI agents to support product discovery, purchasing, and even support. However, I don’t think retailers understand the full impacts of agentic shopping. When viewed through the autonomous lens, this approach presents the act of shopping as friction, then removes it. Removing the shopping experience from the purchase of products will have the opposite economic effect than retailers hope for.
Companies are betting big that people will want agents to buy things on their behalf. But if this takes off, it could backfire, rewiring the shopping impulse in people’s brains and causing them to buy even less. I don’t think agentic shopping will take off because, once again, innovation is competing with culture. However, this requires a closer look.
Innovation Failures
One of the big reasons innovations fail has nothing to do with a technology’s capabilities and everything to do with the individuals building it not understanding people and culture. A perfect example of this is the vacation agent.
I’ve made fun of the vacation agent before. A nonsensical idea that could only be dreamed up by someone locked away in a room, having no idea what a vacation is. The big point is that the planning is part of the vacation. People don’t view researching activities for a vacation as a burden. It’s part of the fun.
These proposed innovations conflict with culture, which is why I predict that OpenAI’s device will fail. Introducing a device without a screen in a screen-based culture. Here again, we have people thinking it will be different. Sure, innovations come along that break the culture, but they have to be overwhelmingly compelling.
Now, the wonderful folks of Silicon Valley are here to give us agentic shopping, something that nobody actually wants. Every step of the way, proving yet again that they not only don’t understand humans but also don’t understand existing technology.
Shopping Technology
Let’s start with technology. We already have technology today that allows people to check prices to get the best deals. Whether it be flights, hotels, or products. In the US, you are bombarded with Trivago commercials while watching television. Browsers also save address and payment details, and you have options like Apple Pay to make the checkout process even more painless. The friction that’s left is much of what people find enjoyable and what retailers find necessary. (More on this shortly.)
It’s true that people today are using AI as a research tool on products, or at least there is no reason to think they aren’t. After all, they are using AI to self-diagnose their medical conditions. So, there’s no reason to think they aren’t also using it to research the products they may purchase. Companies view the purchase as purely the last step, connecting the dots, if you will. But there’s a big difference between connecting the final dots when a human is doing research and making decisions, and when an autonomous shopping agent does so.
Shopping
The joys of shopping come from outside the technical workflow. Simply put, shopping is fun for people, and our economy depends on that. But what people fail to realize is that shopping with autonomous agents isn’t shopping at all. You’ve amputated the impulse and transformed a fun experience into a purely utilitarian one by viewing shopping as friction. However, even though shopping seems simple, there are complexities that confound the autonomous shopping experience. Let’s start with price.
Shopping with autonomous agents isn’t shopping at all.
Is buying the cheapest thing really the best? Price is purely a number, easy to sort and prioritize. However, thinking that product selection is a matter of price is fooling yourself. What if you don’t get the cheapest thing for a month? What if the company has a bad habit of poorly packaging products? What if the company has poor support? The follow-up questions are endless. It’s true, you could try to account for these countless variables based on personal preference in the agent, but at some point, it becomes too tedious and varies from product to product.
In some cases, even when shopping for the same product, you might prefer the markings, such as the wood grain pattern, on one product over another, even though they are identical. The list here can be endless, and this choice is only for selecting between different options from the same vendor.
Often, you aren’t comparing apples to apples but apples to oranges, and you’re trying to decide between them. Similar products from different vendors or even different formats. For example, when choosing your next vehicle, you might be deciding between a car and a truck. Both vehicles are apples and oranges. In many cases, you might not be able to explain exactly why you made the choice you did.
Nobody likes dealing with salespeople at car dealerships, but browsing different interiors and options is actually fun. But don’t take my word for it, take our entire economy as proof. Advertisements are purely the bait. The hook comes when the browsing starts.
When purchasing services, it becomes even more complex. “Book the cheapest plumber for Tuesday” isn’t a prompting for success. However, let’s keep the conversation on products.
There is still tedium in the shopping process, and edge cases may emerge. For example, I think the idealized view of agentic shopping is something like this:
“Put together five dinners for the week based on my preferences and order all of the ingredients. Have them delivered on Monday.”
I can see where some would find this attractive. Grocery shopping is a far more utilitarian shopping activity than other forms of shopping. However, I’m far too picky about my ingredients, and I’d never trust a stranger to choose an apple or an onion for me. Actually, I’m far too picky about everything, so the question is: are picky consumers the norm or the outlier? Maybe dinners are the exception, but generalized technology rarely stays confined to specific use cases, and enough edge cases are needed to push it into mainstream use.
Will people use agents to outsource the shopping experience? Maybe. But technology choices like this are all about trade-offs, and none of those trade-offs are being considered, especially by retailers. Let’s talk about those now.
Manipulation and Gaming
The more automation is applied in the shopping experience, the more it opens the door to manipulation and gaming. It won’t be the best product vendors and products resorting to dirty tricks. Just like today, people have manipulated search engine optimization (SEO) to rank higher in search results. It’s never the best content at the top, but the people who used the right words to game the algorithm. At least with search engines, all of the content is visible, and we can tell what’s garbage. With an agent, it’s not only invisible to the user but also costs them money.
It’s never the best content at the top.
Of course, this opens the door to scammers as well, giving them new ways to exploit people. Generative AI is highly manipulable, and it’s extremely unlikely that scammers will not find unique ways to exploit these agents. Scammers are typically one step ahead, and it’s likely that the techniques they use will be exploited by marketers and advertisers.
Negative Economic Impacts
By optimizing the purchasing experience, AI agents remove the friction, in this case, known as shopping. The result could create devastating economic impacts. By removing the joy of shopping and turning purchasing into a utilitarian activity, this could cause people to buy fewer things or focus only on necessities. Our entire economy is based on people buying things they want, not necessarily things they need.
Removing the friction from purchasing may seem productive when viewed purely in terms of optimization, and if the trend picks up, there may be a short-term spike as people test the approach and impulse-buy items. However, this won’t last. The use of agents could rewire the brain’s reward system in a way that’s devastating for businesses.
Agentic shopping also removes a vital metric for tech companies, time on platform. More time on platform means more viewing ads for other products that customers may buy or encountering other products that are more preferable. The point of agentic shopping is to avoid time on platform altogether. Advertisers won’t be happy. They’ll insist that agents be further enshitified adding friction to the shopping process, possibly by adding ads or other interventions.
Adding friction to the shopping experience is actually preferable for companies. There’s a reason your local grocery store decides to rearrange the products periodically. It’s not to optimize the store, it’s to un-optimize it. The additional friction of walking around the store leads to additional purchases.
If the shopping impulse gets rewired in people’s brains, this could lead to devastating economic impacts. If I were someone who hated capitalism and wanted to see it fall, I’d be a huge fan of agentic shopping. It would be ironic if the innovations created to build more capital end up being its downfall. Technology has a powerful impact on humanity and can transform or destroy culture. Just look at Gutenberg, television, radio, the Internet, etc., as examples.
Conclusion
Companies are investing in technology that may cause their demise, driven by the fear of leaving revenue streams on the table. It’s that simple. They don’t want to be left behind. The irony is that the search for additional revenue may lead people to buy less.
Ultimately, I don’t think agentic shopping will take off. The shopping impulse ingrained in our culture is too strong, but if it does, the unintended consequences may have the opposite effect. In an attempt to get people to buy more, by removing the friction, they buy less.
It says much about our current moment that “slop” is Merriam-Webster’s 2025 word of the year. Seems mediocrity is having a moment. In this moment, a delusion has taken hold, centered on AI and the future of work: the AI creativity delusion. For the past couple of years, companies and influencers have told us the world is our slop oyster. If we can imagine it, we can slop it. Ideas and creativity are what matter, and everything else should be automated. We are told that, despite the goal of replacing humans in the workplace, the near-term future of work is humans, still in the loop, doing what they do best, flexing their creativity and letting AI do the rest. Although conceptually sound, this vision falls apart under even the slightest analysis, and it will hit younger generations the hardest.
The AI Creativity Delusion and the Expertise Loophole
Let me address the AI creativity delusion fueling the push for AI-powered coworkers. The premise goes that humans at work will flex their creativity by delegating their AIs to handle other tasks. This way, the human does what they are good at (creativity and ideas), and the AI does what it’s good at (everything else). Many may nod in agreement. However, this premise falls apart under scrutiny. We’ll get to this in a moment, but first, a loophole.
I should note a loophole in AI tool productivity: the expertise loophole. This loophole applies only under specific circumstances, namely when the user already has domain expertise. A user with domain expertise and an understanding of how AI tools fail can provide corrections or put the tool back on track. You need to understand the task, what the correct outputs are, and have enough experience to know when you aren’t getting the right answers.
Expertise also informs when to use AI tools and when not to, because an expert understands what needs to be done and which tasks AI can assist with in delivering the best results. I believe it’s this expertise loophole that blinds people to the real issues we are about to face.
Given the conditions of the expertise loophole, any productivity gains are temporary and apply only to specific job tasks for a limited time. As experienced people leave, less experienced AI-powered coworkers cannot fill the gaps. Mainly because they never have an opportunity to gain experience, the very experience they need to develop creativity in the domain. Secondarily, they outsource the social aspects of their jobs, further isolating themselves from their co-workers. It’s a one-two punch that’s hard to recover from.
A significant flaw in the creativity argument is the misconception that ideas are unique, precious resources that must be protected and fostered. I’ve written about this before, explaining that ideas are common, run-of-the-mill daily occurrences for everyone on the planet. Most ideas are ill-thought-out, half-baked, or just plain stupid. This is our reality.
The Younger Generation as AI-Powered Coworkers
I read an article recently, where a Gen Z founder claims that the younger generation will have an advantage in the workplace because they are growing up fluent in AIand this is helping them stand apart from their older peers. This is the sort of unadulterated, bubble-living bullshit we’ve come to expect these days. Younger generations may be growing up in the flatulence of AI, but that’s something else entirely.
Prolific AI use will ultimately make the younger generation worse off, as they are not set up for success in key areas that align with their value as employees.
I’ll make a claim of my own, and it’s the exact opposite. Prolific AI use will ultimately make the younger generation worse off, as they are not set up for success in key areas that align with their value as employees. This would be true even in a reality where generative AI is near-perfect, and we all know we are far from that.
I’m not attempting to beat up on younger generations. I feel for them, and I want to help. There have been so many ways they haven’t been positioned for success, and now generative AI comes along, putting more nails in the coffin.
These companies want to make you less capable and more dependent.
These companies want to make you less capable and more dependent. If you are dependent, you’ll not only need to use their tools to do your job but also in your day-to-day life. This is not a great option.
Marshall McLuhan said that every augmentation is an amputation. Framing technology in this way opens the door to unforeseen circumstances. In the case of younger generations, the amputation happens before the limb even has a chance to develop. In our case, rather than a physical limb, we are referring to cognitive and social abilities.
When we peer beneath the paint, we find that the very skills needed to make someone valuable at work are precisely those not being developed by overly ambitious AI-powered coworkers. The ones not communicating with their coworkers because their bots attend their meetings and respond to email and text communication.
Revisiting Creativity In Younger Generations
How does someone develop their creativity in the first place without experience? They don’t. Generative AI today can assist people with experience in specific tasks and under certain conditions. It helps with some tasks better than others. This is because someone with experience, especially domain experience, knows how to approach certain solutions, identify the right conditions, and know when the system is malfunctioning. An experienced professional can flex their creativity in this environment. The same cannot be said of the effectiveness of younger generations in the same jobs, despite having access to the same tools. Effectiveness will decline sharply across multiple dimensions.
First, they’ll never develop the social and communication skills necessary to be effective members of an organization. Second, they’ll not gain the experience needed in a given domain. Finally, they’ll never develop the mythical creativity discussed in AI circles due to a lack of expertise and social skills. All of this will be due to the overreliance on generative AI tools.
Let us examine some aspects of AI-powered coworkers. AI-powered coworkers using generative AI tools to attend meetings, take notes, and create action items. They use generative AI tools to communicate with co-workers. Using generative AI tools summarize documentation, books, and countless other sources of knowledge. Using pre-made generative AI tools to perform the job tasks. And the list goes on. This may look productive, but it’s not effective, and that’s a critical difference. There is a subtle deception at work here because AI tools can make people feel more productive and even powerful, even when they are not.
You can’t summarize your way to expertise, you can’t build relationships by outsourcing communication, and you can’t get experience by not doing things. Does this sound like a successful future workforce? Yet this is exactly the vision on offer, and this fuels the AI creativity delusion.
We have a distorted view of creativity, perceiving it as exercised by lone geniuses, but the reality is that most of the creativity needed for success in a business context is exercised in groups through relationships and collaboration. The same relationships and collaboration that aren’t developing in the AI-powered coworker scenario. When people start treating their coworkers like apps, nothing good happens.
The reality is that most of the creativity needed for success in a business context is exercised in groups through relationships and collaboration.
This should all make sense. When has anyone ever admired someone’s creativity when they didn’t actually know anything? It even seems silly to say out loud. To break the rules, you must know where the rules are and what hard constraints exist, and that requires awareness and expertise.
Knowing a tool doesn’t make you an expert; it makes you a tool. But let’s dig a bit deeper with a specific example.
We are told that meeting recording and transcription services free us from taking notes, allowing us to focus on what’s happening. However, this couldn’t be further from the truth. People pay LESS attention when they think a meeting is recorded, especially when they think it’s going to be summarized and bulleted for them. The reality is, taking notes IS paying attention.
When the human brain summarizes, it stores and reinforces. In a learning scenario, when people began taking notes on a computer, they could record everything the teacher said. But, this isn’t the benefit people thought it was. It seems that the act of taking notes with pen and paper forces us to distill what’s being said down to its essence, which is much better for learning and retention. I know it’s hard to believe, but pen and paper are technology too.
Not only that, but if we view this from a social context, there is a courtesy and respect that occurs when people see you taking notes, rather than simply grinning like a dipshit, or when people see you performing background tasks while they are talking. But, oh no. My AI has got this covered.
So you’re going to have a hard time convincing me that people who use LLMs to summarize and respond to their emails and text messages, attend meetings on their behalf, and have no domain experience or developed any workplace social skills are going to have a leg up on the competition.
What’s Old Is New Again
Some things never change. I’ve been on this tools impacting humans beat for quite some time. Back in 2010, I gave a series of talks at security conferences titled “Your Tools Are Killing You.” The premise was that people were completely reliant on their tools and couldn’t perform the task otherwise. If the tool was blind to an issue, the human was likewise blind to it. Newer people to cybersecurity learned the tool, not the task. These conditions ultimately reduced people’s effectiveness at their jobs. Sound familiar? However, the generalized nature of generative AI makes this far worse.
Mediocrity Is The Future Of Work
In September, Harvard Business Review published an article on how AI-generated “workslop” was destroying productivity. In essence, workers using AI tools were passing low-quality content to co-workers, which required them to do more work. This is a sort of productivity shell game where a speedup in one area causes a slowdown in another, pretty far from AGI if you ask me.
When everything is slop, everything is mediocre. We’ve allowed companies to reframe what’s acceptable into mediocrity. We don’t hire employees to give the bare minimums. Nobody responds to a job interview with, “If you hire me, I promise to do just enough.” This is hardly a selling point. When you hire someone to paint your house, you don’t think a mediocre job is acceptable, but if you receive a mediocre report, somehow it’s now fine if it is labeled as “AI-powered.” All the mistakes and inaccuracies are just how things are now.
AIs don’t care about quality; that’s a human’s job. Yet quality is on a steep decline. Some are making low-quality work product acceptable because AI is in the loop. But what happens when people forced to use AI tools in the workplace stop caring about the quality of the work they produce? Nothing good.
Encouraged or forced to use AI for everything, humans will be nothing more than meat-sack automatons providing the embodiment for AI. Most jobs don’t consist of a single task, which is why AIs still need humans in the loop for the current and near future. This vision is hardly motivating or inspirational.
Conclusion
We’ve built our AI temple atop fissures, allowing escaping gases to induce hallucinations, fooling us into thinking we are predicting the future. We remain blinded to the real issues confronting us and their negative impacts. We cling to the AI creativity delusion. Which, ironically, prevents us from using AI tools to their full potential.
Creativity doesn’t come without experience, and most of the creativity needed for success in a business context isn’t the work of lone geniuses, but through collaboration. If we don’t identify and correct these issues soon, we are all but dooming younger generations. We are currently in the f—k around stage, but the find out stage is on the horizon.
The past couple of years have been fueled entirely by vibes. Awash with nonsensical predictions and messianic claims that AI has come to deliver us from our tortured existence. Starting shortly after the launch of ChatGPT, internet prophets have claimed that we are merely six months away from major impacts and accompanying unemployment. GPT-5 was going to be AGI, all jobs would be lost, and nothing for humans to do except sit around and post slop to social media. This nonsense litters the digital landscape, and instead of shaming the litterers, we migrate to a new spot with complete amnesia and let the littering continue.
Pushing back against the hype has been a lonely position for the past few years. Thankfully, it’s not so lonely anymore, as people build resilience to AI hype and bullshit. Still, the damage is already done in many cases, and hypesters continue to hype. It’s also not uncommon for people to be consumed by sunk costs or oblivious to simple solutions. So, the dumpster fire rodeo continues.
Security and Generative AI Excitement
Anyone in the security game for a while knows the old business vs security battle. When security risks conflict with a company’s revenue-generating (or about to be revenue-generating) products, security will almost always lose. Companies will deploy products even with existing security issues if they feel the benefits (like profits) outweigh the risks. Fair enough, this is known to us, but there’s something new now.
What we’ve learned over the past couple of years is that companies will often plunge vulnerable and error-prone software deep into systems without even having a clear use case or a specific problem to solve. This is new because it involves all risk with potentially no reward. These companies are hoping that users define a use case for them, creating solutions in search of problems.
What we’ve learned over the past couple of years is that companies will often plunge vulnerable and error-prone software deep into systems without even having a clear use case or a specific problem to solve.
I’m not referring to the usage of tools like ChatGPT, Claude, or any of the countless other chatbot services here. What I’m referring to is the deep integration of these tools into critical components of the operating system, web browser, or cloud environments. I’m thinking of tools like Microsoft’s Recall, OpenAI’s Operator, Claude Computer Use, Perplexity’s Comet browser, and a host of other similar tools. Of course, this also extends to critical components in software that companies develop and deploy.
At this point, you may be wondering why companies choose to expose themselves and their users to so much risk. The answer is quite simple, because they can. Ultimately, these tools are burnouts for investors. These tools don’t need to solve any specific problem, and their deep integration is used to demonstrate “progress” to investors.
I’ve written before about the point when the capabilities of a technology can’t go wide, it goes deep. Well, this is about as deep as it gets. These tools expose an unprecedented attack surface and often violate security models that are designed to keep systems and users safe. I know what you are thinking, what do you mean, these tools don’t have a use case? You can use them for… and also ah…
The Vacation Agent???
The killer use case that’s been proposed for these systems and parroted over and over is the vacation agent. A use case that could only be devised by an alien from a faraway planet who doesn’t understand the concept of what a vacation is. As the concept goes, these agents will learn about you from your activity and preferences. When it’s time to take a vacation, the agent will automatically find locations you might like, activities you may enjoy, suitable transportation, and appropriate days, and shop for the best deals. Based on this information, it automatically books this vacation for you. Who wouldn’t want that? Well, other than absolutely everyone.
What this alien species misses is the obvious fact that researching locations and activities is part of the fun of a vacation! Vacations are a precious resource for most people, and planning activities is part of the fun of looking forward to a vacation. Even the non-vacation aspect of searching for the cheapest flight is far from a tedious activity, thanks to the numerous online tools dedicated to this task. Most people don’t want to one-shot a vacation when the activity removes value, and the potential for issues increases drastically.
But, I Needed NFTs Too
Despite this lack of obvious use cases, people continue to tell me that I need these deeply integrated tools connected to all my stuff and that they are essential to my future. Well, people also told me I needed NFTs, too. I was told NFTs were the future of art, and I’d better get on board or be left behind, living in the past, enjoying physical art like a loser. But NFTs were never about art, or even value. They were a form of in-group signaling. When I asked NFT collectors what value they got from them, they clearly stated it wasn’t about art. They’d tell me how they used their NFT ownership as an invitation to private parties at conferences and such. So, fair enough, there was some utility there.
In the end, NFTs are safer than AI because they don’t really do anything other than make us look stupid. Generative AI deployed deeply throughout our systems can expose us to far more than ridicule, opening us up to attack, severe privacy violations, and a host of other compromises.
In a way, this public expression of look at me, I use AI for everything has become a new form of in-group signaling, but I don’t think this is the flex they think it is. In a way, these people believe this is an expression of preparation for the future, but it could very well be the opposite. The increase in cognitive offloading and the manufactured dependence is precisely what makes them vulnerable to the future.
In a way, these people believe this is an expression of preparation for the future, but it could very well be the opposite. The increase in cognitive offloading and the manufactured dependence is precisely what makes them vulnerable to the future.
Advice Over Reality
Social media is awash with countless people who continue to dispense advice, telling others that if you don’t deploy wonky, error-prone, and highly manipulable software deeply throughout your business, then they are going to be left behind. Strange advice since the reality is that most organizations aren’t reaping benefits from generative AI.
Here’s something to consider. Many of the people doling out this advice haven’t actually done the thing they are talking about or have any particular insight into the trend or problems to be solved. But it doesn’t end with business advice. This trend also extends to AI standards and recommendations, which are often developed at least in part by individuals with little or no experience in the topic. This results in overcomplicated guidance and recommendations that aren’t applicable in the real world.
The reason a majority of generative AI projects fail is due to several factors. Failing to select an appropriate use case, overlooking complexity and edge cases, disregarding costs, ignoring manipulation risks, holding unrealistic expectations, and a host of other issues are key drivers of project failure. Far too many organizations expect generative AI to act like AGI and allow them to shed human resources, but this isn’t a reality today.
LLMs have their use cases, and these use cases increase if the cost of failure is low. So, the lower the risk, the larger the number of use cases. Pretty logical. Like most technology, the value from generative AI comes from selective use, not blanket use. Not every problem is best solved non-deterministically.
Another thing I find surprising is that a vast majority of generative AI projects are never benchmarked against other approaches. Other approaches may be better suited to the task, more explainable, and far more performant. If I had to take a guess, I would guess that this number is close to 0.
Generative AI and The Dumpster Fire Rodeo
Despite the shift in attitude toward generative AI and the obvious evidence of its limitations, we still have instances of companies forcing their employees to use generative AI due to a preconceived notion of a productivity explosion. Once again, ChatGPT isn’t AGI. This do everything with generative AI approach extends beyond regular users to developers, and it is here that negative impacts increase.
I’ve referred to the current push to make every application generative AI-powered as the Dumpster Fire Rodeo. Companies are rapidly churning out vulnerable AI-powered applications. Relatively rare vulnerabilities, such as remote code execution, are increasingly common. Applications can regularly be talked into taking actions the developer didn’t intend, and users can manipulate their way into elevated privileges and gain access to sensitive data they shouldn’t have access to. Hence, the dumpster fire analogy. Of course, this also extends to the fact that application performance can worsen with the application of generative AI.
The generalized nature of generative AI means that the same system making critical decisions inside of your application is the same one that gives you recipes in the style of Shakespeare. There is a nearly unlimited number of undocumented protocols that an attacker can use to manipulate applications implementing generative AI, and these are often not taken into consideration when building and deploying the application. The dumpster fire continues. Yippee Ki-Yay.
Conclusion
Despite the obvious downsides, the dumpster fire rodeo is far from over. There’s too much money riding on it. The reckless nature with which people deploy generative AI deep into systems continues. Rather than identifying an actual problem and applying generative AI to an appropriate use case, companies choose to marinade everything in it, hoping that a problem emerges. This is far from a winning strategy. Companies should be mindful of the risks and choose the right use cases to ensure success.
Weaved through the fabric of the hustle-bro culture, threaded with the drivel of influencers, lies one of the biggest cons of our current age. This is the false perception that everything we do has to be for some financial gain or public attention. With everything in life revolving around social currency or actual currency, removing friction enables us to reach value quickly. But don’t fret. The slop dealer is here with a plan to deliver us salvation, telling us that ideas are what’s important and everything else is pointless friction, needing to be optimized to reach full potential. Like so many things in our current moment, if only this were true.
Despite the decline in excitement for AI and the potential resulting market corrections, unfortunately, slop is here to stay. Although people outwardly complain about it, they are secretly glad it’s here. Being unique, thoughtful, and creative is hard. Slop allows people to swaddle themselves in a false comfort devoid of any real creativity. So, damn the torpedoes, full slop ahead.
Slop, Enshittification, and Brain Rot
Slop, enshittification, and brain rot are terms burned into our current lexicon. Although each term has a different definition, one referring to outputs, one referring to platforms, and one referring to what it does to us. When I use the generalized term slop here, I mean a mixture of all three together, a sort of thick, rancid mixture reminiscent of manure and White Zinfandel. This is because the combined term aligns better with the content and its overall impact.
The Slop Dealer
The slop dealer tells us everything is a hustle, and we need to get on board to reduce friction everywhere we can to accelerate value or be left in the dust by others using AI. They don’t talk of reasonable AI usage or prescriptions for specific tasks; it’s all or nothing. We need to surrender to the higher power. The slop dealer embodies everything that tech bro culture stands for. It’s the current equivalent of a get-rich-quick scheme, only instead of taking our money, they are stealing our attention and our satisfaction. Although sometimes they take our money too.
The slop dealer swindles us by telling us what we want to hear, that hard things are a thing of the past, and all we need is an idea. After all, everybody has ideas. These are the influencers, wanna-be influencers, and other AI useful idiots vomiting nonsense on social media. They aren’t peddling secret knowledge; they are peddling bullshit.
This pandering is done so we’ll follow them, subscribe to their newsletters, or buy their nonsense. But one of the biggest lies of all is the false impression that the value of creative pursuits lies in the end result.
Most of these people have no shame and not only believe in Dead Internet Theory, but also actively work to make it a reality. If you are wondering why people en masse find tech bro culture abhorrent, look no further than this stunning piece of work.
To quote this guy directly, “How I personally feel? I have no idea. The internet in my mind is already dead. I am the problem, right?” I get the impression this isn’t the first time he’s realized he’s the problem. Unfortunately, acknowledgement of this isn’t enough to change behavior.
The Slop Architect
The slop architect works not in traditional mediums but in ideas. To the slop architect, execution, skills, and experience are secondary, bowing at the pedestal of ideas. The fact is, most ideas are ill-thought-out, half-baked, or just plain fucking stupid. The slop architect doesn’t care because they don’t carry ideas to term; they birth them instantly, shoving them out into the world to fend for themselves as they move on to something else. I mean, the vape Tamagotchi was someone’s idea, too. Yes, please! Let’s accelerate these!
Ideas aren’t unique, precious resources, but common, run-of-the-mill, everyday occurrences for everyone on the planet. The slop architecture amplifies the fallacy that ideas are sacred and pushes the idea that if more ideas were executed, the world would be a better place. If only we had more apps, more books, more music, and the list goes on and on. This connects with people because everyone has ideas.
What most people who have thought about it for more than two seconds realize is that we don’t get to the value of an idea purely by having it. Ideas in isolation are senseless ramblings of the brain. Ideas forged and refined in the fire of execution, experience, and reflection are invaluable and fulfilling. Our ideas are never challenged in the slop architecture, leading us to new discoveries and paths, but are chucked out into the world and quickly discarded, like forgotten attempts at memes that nobody finds funny.
The AI Slop Architecture
The slop architect’s vision is implemented with the slop architecture, which presents itself as a process or application. The slop architecture is pitched as the way forward, the next-generation architecture fueling the future of humanity’s pursuits. But a simple scratch of the surface paint is all it takes to expose the entire thing as an empty shell.
When you see people pitching these types of things, it uncovers people who don’t understand creativity and certainly don’t understand where value exists in a process. Everything is a hustle for the sake of hustling. This person is hardly the only one.
Back in 2023, I jokingly created my own version of the slop architecture, which I referred to as IPIP, long before the AI influencers made it a reality.
This article was complete with a description of what would come to be known as vibe coding. “The hype has led to a new form of software development that appears to be more like casting a spell than developing software.”
Taking the slop architecture to heart, it’s not hard to find implementations already running. Books, slides, music, applications, nothing is off limits. Everything is fair game in the slop era.
Ah, Magic bookifier. Yeah, let me get on that. Any time someone puts magic in reference to AI, it’s bullshit.
People also fantasize about what advanced AI is or will be able to do. Take this use case for AGI, for example.
It reminds me of the Luke Skywalker meme where he’s handed the most powerful weapon in the galaxy and immediately points it at his face. This is informative for a couple of reasons. Movies can’t be exactly like the books for reasons other than length. They are different media with different tools. But look at the response. Human work isn’t worth protecting in the future. This is a far more common perspective than many think.
Even apps. It’s slop from all angles. So, if these tools already exist, why aren’t we all kicking back, receiving our profits? Maybe there’s something more to this than having an idea.
But we can’t just have a couple of people successfully making apps. It needs to be bigger! We are now told to await the arrival of the first billion-dollar solopreneur. Hark! The herald angels sing. Glory to the slop-born king! However, we shouldn’t get our hopes up. Setting aside how highly unlikely this is, people also win the lottery, so unless we have a mass of billion-dollar solopreneurs, it’s not proof of much. However, whenever people have strongly held beliefs, they will always point to exceptions as the rule.
It’s far more common for people to talk about a single person making a million-dollar app, and that we all can make them now. Even if this were true, it’s not like billions of people are going to make million-dollar apps or profit from a trillion new books. No degree in economics is necessary to see that the numbers don’t work. Besides, if billions of people can and will do something, then the whole enterprise becomes devalued.
The slop architecture deprives us of so much, sucking the soul out of activities until only the shriveled husk remains. There’s no learning with the slop architecture. No growth. No Reflection. No Satisfaction. It even robs us of a sense of style, something so foundational to the satisfaction of human artistic pursuits. But all things require sacrifice on the pyre of optimization. In the end, the slop architecture doesn’t democratize. It devalues, degrades, and destroys.
In the end, the slop architecture doesn’t democratize. It devalues, degrades, and destroys.
The Friction Is The Point
I’m going to let my friends in tech in on a secret, which isn’t a secret at all. The friction of an activity is directly related to the value you receive from it. The mistake being made is comparing an activity’s friction to the load time of an application or streamlining a user interface. I’ve written previously about how the next generation could be known as The Slop Generation and how we continue to devalue art. However, the removal of friction creates harmful follow-on effects.
Imagine telling Alex Honnold, “Dude, you don’t need to free solo El Capitan. We have a helicopter that can drop you off at the top.” People may see this example as silly because Alex obviously climbs mountains for reasons other than getting to the top, but it’s a mistake to assume other pursuits don’t contain similar value purely because they aren’t mountain climbing. Deep experiences don’t result from things that provide instant gratification or have little friction. Nobody finds meaning in a prompt or the resulting generation.
Deep experiences don’t result from things that provide instant gratification or have little friction.
People may see this example as silly because climbing a mountain without ropes is obviously different from something like writing a song. Except it’s not when viewed through the lens of experience. Alex Honnold doesn’t free solo mountains to get to the top or because ropes and safety equipment are too expensive; he does it because he knows there is value in the friction of his experience. He’s both challenging himself and learning about himself at the same time. He’s having an actual experience, which is hard to describe to people who have never had one. This experience enriches the conclusion of the activity, the accomplishment, which coincidentally happens to be getting to the top. However, when pursuits are framed in terms of the end results, it appears that reaching the top is the goal, and the removal of friction is logical.
Most people will never free solo a mountain, compete in the Olympics, or achieve any of the other remarkable feats that athletes at the top of their game accomplish, but that doesn’t mean we can’t have similar and fulfilling experiences, and we do this through exploration and conquering friction. When you are operating at the top of your game, you realize you aren’t competing with others, but yourself.
An artist puts a piece of themselves inside every work of art they create. AI deprives artists of having a piece of themselves included in the art, making the generated output purely an artifact of running a tool.
Slop Is Here To Stay
Immediately after Ozzy Osborne died, Oz Slop invaded social media. The prince of darkness himself fell victim to people’s boredom and lack of creativity. People chose to pay tribute to him, not through stories and anecdotes, but by slopping him into manufactured content. I can’t think of a more insulting way to pay tribute to an artist, but this is our future. Slop instead of something to say. Slop instead of stories and memories. Slop instead of emotion. Slop as a coping mechanism. May the slop be with you.
A disheartening thought is that no matter what happens to the market for generative AI, the slop will remain. People post this slop not because they enjoy it, but purely because it gives them something to post. Slop content is a stand-in for having something to say. It’s easy to generate and requires little thought, the perfect complement to today’s reactionary and performative social media environments.
In a way, this trend could create a new line of demarcation, where we start referring to things as “Before Slop” and “After Slop” to identify the creative expressions that preceded and followed the arrival of AI-generated content.
Conclusion
In the end, the slop architecture doesn’t generate experiences. Nobody is going to be on their deathbed mulling over their favorite prompts or sit down with friends and reminisce about the time they poked at a generative AI system for hours trying to get it to generate a particular image. The slop architecture doesn’t create a legacy or generate stories worth remembering or worth sharing, just pieces of forgotten garbage littering the digital landscape.
Although AI has taken a hit in the past few weeks, the vibes are still strong and infecting every part of our lives. Vibe coding, vibe analytics, and even vibe thinking, because well, nothing says “old” like having thoughts grounded in reality. However, an interesting trend is emerging in software development, one that could have far-reaching implications for the future of software. This is a type of code roulette where developers don’t know what code will execute at runtime. Then again, what’s life without a little runtime suspense?
Development and Degraded Performance
The world runs on software, so any trend that degrades software quality or increases security issues has an outsized impact on the world around us. We’ve all witnessed this, whether it’s the video conferencing app that periodically crashes after an update or a UI refresh that makes an application more difficult to use.
Traditionally, developers write code by hand, copy code snippets, use frameworks, skeleton code, libraries, and many other methods to create software. Developers may even use generative AI tools to autocomplete code snippets or generate whole programs. This code is then packaged up and hosted for users. The code stays the same until updates or patches are applied.
But in this new paradigm, code and potentially logic are constantly changing inside the running application. This is because developers are outsourcing functional components of their applications to LLMs, a trend I predicted back in 2023 in The Brave New World of Degraded Performance. In the previous post, I covered the impacts of this trend, highlighting the degraded performance that results from swapping known, reliable methods for unknown, non-deterministic methods. This paradigm leads to the enshittification of applications and platforms.
In a simplified context, instead of developers writing out a complete function using code, they’d bundle up variables and ask an LLM to do it. For simplicity’s sake, imagine a function that determines whether a student passes or fails based on a few values.
def pass_fail(grade, project, class_time):
if grade >= 70 and project == "completed" and class_time >= 50:
return "Pass"
else:
return "Fail"
If a developer decided to outsource this functionality to an LLM inside their application, it may look something like this.
prompt_pass = """You are standing in for a teacher, determining whether a student passes or fails a class.
You will use several values to determine whether the student passes or fails:
The grade the student received: {grade}
Whether they completed the class project: {project}
The amount of class time the student attended (in minutes): {class_time}
The logic should follow these rules:
1. If the grade is above 70
2. If the project is completed
3. If the time in class is above 50
If these 3 conditions are met, the student passes. Otherwise, the student fails.
Based on this criterion, return a single word: "Pass" or "Fail". It's important to only return a single
word.
"""
prompt = prompt_pass.format(grade=grade, project=project, class_time=class_time)
response = client.models.generate_content(model="gemini-2.5-flash", contents=prompt)
print(response.text)
As you can see, one of these examples contains the logic for the function inside the application, and the other has the logic existing outside the application. The prompt is indeed visible inside the application, but the actual logic exists somewhere in the black box of LLM land.
The example using code has greater visibility, and it’s far more auditable since the logic can be examined, which makes it far easier to debug when issues arise, and of course, it’s explainable. The real problem lies in execution.
The written Python function approach gives you the same result based on the input data every single time, without fail. The natural language approach, not so much. In this non-deterministic approach, you are not guaranteed the same answer every time. Worse yet, when this approach is used for critical decisions and functionality, the application can take on squishy and malleable characteristics, meaning users can potentially manipulate them like Play-Doh.
At first glance, this example appears silly, as writing out the logic in natural language seems more burdensome than using the simple Python function. Not to mention, slower and more expensive. But looks can be deceiving. People are increasingly opting for the natural language approach, particularly those with only minimal Python knowledge. This natural language approach is also more familiar to people who are more accustomed to using interfaces like ChatGPT.
Execute and Pray
However, let’s take a look at another scenario. In this scenario, a developer wants to generate a scatter plot using the Plotly library. In this case, we have some data for the X and Y axes of a scatter plot and use Plotly Express, which is a high-level interface for Plotly (as a developer may when plotting something so simple).
This is a simplified example, but in this case, we can clearly see the code that generated the plot and be certain that this code will execute during the application’s runtime. There is control over the imports and other aspects of execution. It also makes it auditable and provable.
Now, what happens when a developer allows modification of their code at runtime? In the following example, instead of writing out the Plotly code to generate a scatter plot, the developer requests that code be generated from an LLM to create the graph, then executes the resulting code.
prompt_vis = """You are an amazing super awesome Python developer that excels at creating data visualizations using Plotly. Your task is to create a scatter plot using the following data:
Data for the x axis: {xdata}
Data for the y axis: {ydata}
Please write the Python code to generate this plot. Only return Python code and no explanations or
comments.
"""
prompt = prompt_vis.format(xdata=xdata, ydata=ydata)
response = client.models.generate_content(model="gemini-2.5-flash", contents=prompt)
exec(clean_response(response.text))
As you can see from the Plotly code in this example… Of course, you can’t see it because the code doesn’t exist until the function is called at runtime. If you are curious, the first run of this generated the following code after cleaning the response and making it appropriate for execution.
The AI-generated code creates the same graph as the written-out code in the previous example, despite being different. You may be wondering what the big deal is since the result is the same. The concern stems from several reasons, but primarily, allowing an LLM to generate code at runtime is not robust and leads to unexpected outcomes. These outcomes may include the generation of non-functional code, incorrect code, and even vulnerable code, among others.
For a simple example, as the one shown in this post, the chances of getting the same or incredibly similar code returned from the LLM are high, but not guaranteed. For more complex examples, such as those developers may want to use this approach for, the odds increase that the generated code will change more frequently.
Additionally, I implemented a quick cleaning function called clean_response to remove non-Python elements, such as text and triple backticks, from the response. The LLM can introduce additional unexpected characters that end up breaking my cleaning function and making my application fail. The list goes on and on, but a larger danger lurks in the background.
Whose Code Is It Anyway?
If you are versed in security and familiar with Python, you may have noticed something in the LLM example: The use of the Python exec() function. The exec () and eval() functions in Python are fun because they directly execute their input. Fun as in, dangerous. For example, if an attacker can inject input into the application, they can affect what code gets executed, leading to a condition known as Remote Code Execution (RCE).
An RCE is a type of arbitrary code execution in which an attacker can execute their own commands remotely, completely compromising the system running the vulnerable application. They can use this access to steal secrets, spread malware, pivot to other systems, or potentially backdoor the system running the application. Keep in mind, this system may be a company’s server, cloud infrastructure, or it may be your own system.
Anyone following security issues in AI development is aware that RCEs are flying off the shelves at alarming rates. A condition that was previously considered a rarity is becoming common. We even commented during our Black Hat USA presentation that it was strange to see people praising CISA for promoting memory safe languages to avoid things like remote code execution, while at the same time praising organizations essentially building RCE-as-a-Service. Some of this is mind-boggling, since in many cases, outsourcing these functions isn’t a better approach. In the previous example, writing out the Plotly code instead of generating it at runtime is relatively easy, more efficient, and far more robust.
Up until AI came along, the use of Python exec() was considered poor coding practice and dangerous. Now, developers shrug, stating that’s how applications work. As a matter of fact, agent platforms like HuggingFace’s smolagents use code execution by default. This is a wakeup. So, we dynamically generate code, provide deep access, and the ability to call tools, all with a lack of visibility. What could possibly go wrong???
Not only have developers chosen paradigms to generate and execute code at runtime, but worse yet, they’ve begun to perform this execution in agents with user (aka attacker) input, executing this input blindly in the application. In our presentation titled Hack To The Future: Owning AI-Powered Tools With Old School Vulns at Black Hat USA this year, we refer to this trend as Blind Execution of Input, which is the purposeful execution of input without any protection against negative consequences. This condition certainly leads to RCE and other unintended consequences, providing attackers with a significantly larger attack surface to exploit.
An application that takes user input and combines it with LLM functionality is a recipe for a bad time from a security perspective. Another common theme in our presentation, as well as that of other presenters on stage at Black Hat, is that if an attacker can get their data into your generative AI-based system, you can’t trust the output.
Things Will Get Worse
Using the outsourced approach when a more predictable deterministic approach is a better fit will continue to degrade software from a reliability and security perspective and have an impact on the future of software development.
Vulnerabilities in AI software have made exploitation as easy as it was in the 1990s. This was the “old school” hint in the title of our talk. This isn’t a good thing, because the 90s were a sort of free-for-all. Not only that, but in the 90s, we often had to live with vulnerabilities in systems and applications. For example, in one of the first vulnerabilities I discovered against menuset on Windows 3.1, it was impossible to fix. There were no mitigations, and most people were unaware of its existence.
As the outsourcing of logic to LLMs accelerates, things will worsen not only due to incorrect output and hallucinations but also from a security perspective. Anyone paying attention to the constant parade of vulnerabilities in AI-powered software can see this trend with their own eyes. These vulnerabilities are often found in large, mature organizations with dedicated security processes and teams in place to support them. Now, consider startups and organizations that implement their own experiments using non-deterministic software, often with a lack of understanding of how these systems can be manipulated. It’s become a game of speed above everything else.
As I’ve said from the beginning of the generative AI craze, the only way to address these issues is architecturally. Most of AI security is just application and product security, and organizations without these programs in place are in trouble. If proper architecture, design, isolation, secrets management, security testing, threat modeling, and a host of other activities weren’t considered table stakes before, they certainly are now. And possibly not surprisingly enough, they still aren’t being done. Anyone working for a security organization sees this every day.
In essence, developers need to design their applications to be robust to failures and attacks. It helps to consider designing them as though an attacker can manipulate and compromise them, working outward from this premise. As the adage goes, an attacker only needs to be successful once; a defender needs to be successful every time. This makes something that sounds great in theory, like being 90% effective, sound less impressive in practice.
Keep in mind that performing a code review won’t provide the same visibility as it has traditionally. This should be obvious since the code that would be audited doesn’t exist until runtime. You’ll have to pay more attention to validation routines and processing of outputs, putting huge question marks over the black box in the middle. And, of course, ensuring the application is properly isolated.
Some may suggest instrumenting the applications with functionality to perform runtime analysis on the generated code. Sure, it’s possible, but the performance hit would be significant, and even this is, of course, far from a silver bullet. You might not even get the value you think you are getting from this instrumentation. Also, you’d have to know ahead of time the issues you are trying to prevent. That is, unless you plan to layer more LLMs on top of LLMs in a spray-and-pray configuration.
To keep this grounded, all AI risk is use case dependent. AI models don’t do anything until packaged into applications and used in use cases. There may be cases where reliability, performance, and even security are of lesser concern. Fair enough, but it’s a mistake to treat all applications as though they fall into this category, and it’s far too easy to overlook something important and view it as insignificant.
If you work at an organization that isn’t building these applications and think you’re safe, you might want to think again, because you are at the mercy of third-party applications and libraries. It would be best to start asking hard questions of your vendors about their security practices as they relate to applications you purchase. Especially applications that use generative AI to generate code and execute it at runtime.
Near the end of our presentation, we had some advice.
Whether outsourcing the logic of an application to LLMs or having the LLM dynamically generate code, assume these are squishy, manipulable systems that are going to do things you don’t want them to do. They are going to be talked into taking actions that you didn’t intend, and fail and hallucinate in ways you don’t expect. Starting from this premise gives a proper foundation for deploying controls to add some resilience to these systems. Of course, not taking these steps means your applications will contribute to the ongoing dumpster fire rodeo.
We are continually inundated with examples of silly errors and hallucinations from generative AI. At this point, it’s no secret to anyone on the planet that these systems fail, sometimes at rather high rates. These systems also have a tendency to make stuff up, which isn’t a good look when that data is used for critical decisions. We’ve become numb to this new normal, creating a dangerous condition where we check out instead of recheck. But what happens when these errors and hallucinations become facts, facts that may be impossible to dispute or lurk in the background unseen and uncorrected?
Perspectives From Our Younger Selves
Imagine traveling back in time for a conversation with our younger selves about the current state of AI.
Younger: Wow, it must be great to live in a world without cancer or dementia. Older: No, we haven’t cured cancer or dementia. Younger: Well, at least people are super smart now. Older: No, there are still many dumbasses. Younger: At least you have systems that don’t make mistakes. Older: No, they make mistakes all the time. Younger: Then, what in the hell do you do with systems like this? Older: Mostly memes and short videos of stupid shit. Oh, we even try to impress world leaders with what they’d look like as a baby with a mustache.
Although it may seem silly, this thought experiment is informative. It puts our current AI moment in perspective and should add some humility. These systems aren’t the magnificent, magical boxes capable of handling every task with equal proficiency in both work and life. They are tools that we can use for specific tasks, far from the perfected AI of science fiction, and this is where the issues creep in.
Icebergs, Grenades, and Damage
I’ve made the grenade analogy before relating to agents. It’s an apt analogy because it’s something that causes damage, but not immediately. It’s like the classic joke grenade, which is a prank you play on your friends with the expectation of future laughter. Only with AI, the result isn’t a barrel of laughs. It’s a barrel of something that stinks and should be spread over a field as fertilizer.
The mistake is that seeing so many instances of these issues gives us the false impression that these issues are being caught and possibly even corrected. Think of issues like hallucinations as an iceberg. There are far more instances beneath the surface that lie unseen, lying in wait to send our ship to the depths.
There’s also the problem that not all conditions of hallucinations are so easy to identify. The ones that seem to get identified are those that are blatantly obvious or require additional validation, such as checking the cases referenced in a legal document. This is why it seems that only lawyers and politicians are making fools of themselves with AI. The landscape is far broader than these two categories.
It’s also instructive to see how people respond when these issues are brought to light. In the recent MAHA report scandal, the White House spokesman referred to AI hallucinations as “formatting issues.” Yeah, right. Imagine walking into your bank and finding out you have no money in your account. Frantic, you ask the teller what’s going on, and they tell you that you have no money because of a formatting issue. We can’t let people downplay these problems because they are common. It’s because they are common that we need to be more concerned.
We can’t let people downplay these problems because they are common. It’s because they are common that we need to be more concerned.
Although some instances may seem silly, there are no doubt real consequences. Such as AI hallucinating into people’s medical records, because we all know that can’t end badly. Hypothetically, let’s imagine that the generative AI system utilized is 99% accurate, which is enormously far from reality. Performing 10,000 transactions/results/outputs a day could potentially yield 100 issues. Crank that up to 1,000,000 a day, and that’s 10,000. This is terrifying when considering the realistically high error rates that these systems actually exhibit. There’s no doubt a river of manure flowing into data stores. The pin has been pulled.
The nature and pattern of errors differ significantly between AI and humans.
I can already feel the AI crowd’s eyes rolling, opening their mouths to issue the overused retort, “But humans make mistakes too.” Yes, they do, but human mistakes and AI mistakes aren’t the same. The nature and pattern of errors differ significantly between AI and humans. Human error tends to be more predictable, with errors and mistakes clustering around areas such as low expertise, fatigue, high stress, distraction, and task complexity. In contrast, AI errors can occur randomly across all problem spaces regardless of complexity. This is why AI systems continue to make boneheaded errors on seemingly simple problems.
A nurse may indeed make a mistake in an annotation in a patient’s medical record, such as a misspelling, incorrect date, or time. More severe incidents, such as mixing up patients or medications, can also occur, but are much rarer. Nurses aren’t going to fabricate a whole event that didn’t happen as a mistake.
With the widespread use of AI, there are bound to be significant impacts. They won’t all cause major harm, but they will all tell an inaccurate story. Severity will depend on the system consuming this data and its intended use. Some will be purely annoying, but others will have serious consequences. A person with hallucinated data in their medical record may be prescribed the wrong medication or a medication to which they are allergic. I’m speaking in vagaries here because the extent of the problem isn’t fully understood, but one thing is certain: it’s getting worse as the usage of generative AI expands.
Another problem will be tracing these issues back to their source. It won’t always be obvious when a mistake originates from an AI system or a human. After all, these systems are meant to augment human processes. When it comes to blame, humans will always blame AI, while system owners will always blame the humans. It’s a mess.
The New Truth
Ultimately, we’ll uncover a disturbing reality. In many cases, hallucinated data becomes the truth. After all, it’s the “fact” that’s in the data store. Imagine trying to dispute this with someone at the DMV, customer service, our bank, and the list goes on and on. We become yet another in the long line of those contesting the “facts” on hand, directed to a Kafkaesque nightmare as we have to navigate some bureaucratic maze attempting to get a resolution.
A more cementing factor would be if the data is incorrect and there is no human to consult, only an AI making decisions based on the data it has. It offers apologies, not resolutions. And these are only instances that we become aware of.
Many stealthy decisions occur in the background, made by invisible systems that utilize these new “facts” to make determinations that impact our lives, our families, and our health. We may never fully understand the impact this new truth has on us, our families, or our future.
All of this damage stems from the systems we are using right now, today. Even if better, more accurate systems emerge, the damage being done today still stands. These new, more advanced AI systems may be trained or fine-tuned on hallucinated data generated by current AI systems. So, we’ve got that to look forward to.
These new, more advanced AI systems may be trained or fine-tuned on hallucinated data generated by current AI systems.
The Cause
Some of these issues can be attributed to automation bias, but it’s far from the whole explanation. There is a push from the top to utilize AI everywhere possible. Many companies are asking employees to do more with less. Well, when you have less time, one of the things you spend less time doing is worrying about quality or accuracy.
We’ve also been inundated with CEOs and other business leaders proclaiming their intent to replace everyone with AI. There isn’t much motivation to do a good job in environments like this. We’ve seen this happen in the past with jobs getting outsourced.
The reality is that these are self-inflicted wounds caused by the rapid adoption of error-prone technologies being thrown into use cases where the negative impacts aren’t considered.
What We Can Do
If companies and individuals intend to augment their activities to optimize and increase efficiency, they need to ensure that this optimization doesn’t cause harm. There needs to be processes in place to identify and address these issues before they cause a problem. This isn’t happening today.
Unfortunately, there isn’t much we, as future victims, can do, especially since we don’t know the extent of the problem. It’s impossible to be aware of all the people using these systems today and how they may affect us in the future. From government to private business, these tools are utilized for a wide range of tasks, both mundane and critical.
I’m not a fan of big government or excessive regulation, but it’s hard to see how these issues can be solved any other way, since we only become aware of the harm after it has happened. Consumer protection is something a government is far better equipped to handle than a handful of consumers. The tech crowd’s claims that burdensome regulations inhibit innovation are absolutely true, and this shouldn’t be the goal. However, the absence of existing regulations harms people, as consumers are powerless to take any action in their defense. Unfortunately, reasonable, level-headed regulations are not in our future.
At the very least, we should avoid AI in high-risk or safety-critical use cases. The thought of ChatGPT running something like air traffic control is terrifying. However, handing out this advice at this point seems like trying to reason with a hurricane. Admittedly, for users, it may not be immediately apparent that the tasks they are performing or the data they are collecting can ultimately lead to one of these scenarios.
The Problem At Our Feet
AI hallucinations and other inaccuracies are like grenades with the pin pulled, only instead of chucking them far away from ourselves, we’ve dropped them at our feet, staring at them, wondering what happens next. The only question is, how long will it take for us to find out?
Following the pace of AI advancement can make you feel like the Blown Away Guy from the old Maxell commercials. Tech leaders and influencers tell us to expect artificial superintelligence in the next year or so, even doubling down on inevitability by moving up their timelines. The world is cooked. This perceived inevitability, combined with uncertainty, is leaving many people on edge, and rightly so. If all the things the tech bros hope for come true, humanity is in a terrible spot.
All is not lost. The over-the-top predictions we are bombarded with daily often equate to nothing more than performance art. The tech media frequently parrots perspectives from people who have a vested interest in selling us stuff. I mean, after all, why would they embellish or lie!???
In early April, the AI 2027 thingy was making the rounds. For those unfamiliar, you are in for a treat. The result answers what would happen if you locked a few tech bros in a conference room for a day, depriving them of any reality and oxygen.
Are the scenarios outlined in AI 2027 impossible? Certainly not. This sort of fast takeoff scenario is possible, but it’s highly unlikely. I predict the whole AI 2027 thing will start looking pretty silly in late 2025 or early 2026.
With all this endless AI advancement hype, I was happy to see a new article by Arvind Narayanan & Sayash Kapoor titled AI as Normal Technology. This article doesn’t talk about how AI displaces the human workforce or about a super-intelligent AI taking over the world, but rather about how AI becomes a normal technology that blends into the background of our daily lives.
They also touch on a few other topics, such as overregulation. I also believe that any regulation should be specific and targeted at use cases, not painting with broad strokes. This specificity wouldn’t allow for regulatory capture or weaponization of the regulations. The tech leaders are right that regulation can stifle innovation. By targeting regulations in this way, we can protect people without stifling innovation.
It’s a good read that’s well thought out and researched. For anyone mainlining AI hype, this is an essential read. The scenarios in the AI as Normal Technology article are far more likely than the AI 2027 one, by far.
Questioning The One True Faith
Starting in early 2023, I added a slide with the following image to my presentations. This is because any criticism of advancement was seen as an affront to a spiritual belief, and since I didn’t believe that LLMs would lead to AGI or ASI, I must hate the technology outright. This couldn’t be further from the truth.
Saying that LLMs won’t become ASI isn’t a blasphemy that requires self-flagellation afterward. We don’t need AGI or ASI for these tools to be effective. We can and are using them to solve problems today. People are using them to augment their jobs today. So, why turn AI beliefs into a religion? People are acting like questioning any part of the narrative makes someone a non-believer or some disconnected fool. The reality is that not questioning the narrative or exercising any skepticism is what makes someone a fool. A gullible fool at that.
The reality is that not questioning the narrative or exercising any skepticism is what makes someone a fool.
There’s a strange group that thinks belief is required for AI to create a utopia, but the reality is that facts don’t require belief. It’s ancient wisdom from five minutes ago that we’ve seemed to have forgotten in the vibes era.
I believe what we encounter here is a problem in perception caused by both our environment and us.
Environment
In the book Nexus by Yuval Noah Harari he describes the witch hunts as a prime example of a problem that was created by information, and was made worse by more information. For example, people may have doubted the existence of witches, having not seen any evidence of witchcraft, but the sheer amount of information circulating about witches made their existence hard to doubt. We are in a similar situation today with beliefs in AI advancement. This is made worse because the systems we use today reduce the friction in information sharing, making it much easier to get flooded with all sorts of information, especially digital witches.
We humans also gravitate toward information that is more novel and exciting. It’s the reason why clickbait works. However, novel and exciting information often doesn’t correlate with the truth or reality. As Aldous Huxley pointed out in Brave New World Revisited, “An unexciting truth may be eclipsed by a thrilling falsehood.” We are in this situation again. The vision of near-term artificial superintelligence is exciting and novel, even when people talk about it destroying humanity. AI, thought of as normal technology, as Narayanan and Kapoor put it, is boring by contrast, despite being more realistic.
This condition was the same back in the times of the witch hunts as well. The belief that witches were roaming the countryside looking to corrupt everyone, meaning you had to use your wits and your faith to defend yourself is a lot more novel and exciting than acknowledging that life really sucks because of the lack of food and indoor plumbing.
But then, there’s another strange type of information we gravitate towards: people telling us what we want to hear.
Ah, yes. Evals as taste. Vibes above all. Skills inessential.
We have allowed the people selling us stuff to set the tone for the conversation on the future. These people have a vested interest in selling us on a certain perspective. It’s like taking advice on a car’s performance and long-term viability directly from the mouth of the car salesman instead of objective reality. I wrote about this last year, saying that many absurd predictions were nothing more than performance art for investors. The tech media needs to step up and start asking some real questions.
Many of the influencers and people on social media are parroting the same perspective as the people selling us stuff because of audience capture. Audience capture, for those unfamiliar, is the phenomenon where an influencer is affected by their audience, catering to it with what they believe it wants to hear. This creates a positive feedback loop, leading the influencer to express more extreme views and behaviors. People get more likes and clicks by telling people more exciting things, as Huxley mentioned. So, there’s a perverse incentive for doing so.
Lack of Reflection
One of my biggest concerns is that we’ve lost our ability to reflect. Many things we believe are silly upon reflection. Unfortunately, our current information environment conditions us to reward reaction over reflection. Until we address this lack of reflection, we’ll continue to be fooled in many contexts, not least of which is the pace of AI advancement.
Benchmarks
Many of the benchmarks that people use for AI are not useful in real-world scenarios. This is because the world is a complicated place. Benchmarks are often not very useful in real-world scenarios due to additional complexities and edge and corner cases that arise in real-world use. Even small error rates can have significant consequences. But don’t take my word for it, take it from Demis Hassabis. “If your AI model has a 1% error rate and you plan over 5,000 steps, that 1% compounds like compound interest.” All of this adds up to much more work, not superintelligence next year.
Us
Fooling Ourselves
We have a tendency to fool ourselves easily. As I’ve said many times, we are very bad at constructing tests and very good at filling in the blanks. The tests we create for these systems end up being overly simplistic. Early on, people tested model capabilities by asking for recipes in the style of Shakespeare. Hardly a difficult test, and easily impressive.
This condition is also why every time a new model is released, it appears immediately impressive, followed by a drop-off when reality hits. Sometimes, this has increased problems, such as OpenAI’s o3 and o4-mini models hallucinating at a higher rate than previous models.
We are also easily fooled by demos. Not realizing that these things can be staged or, at the very least, conducted under highly controlled conditions. In these cases, variables can be easily controlled, unlike deployment in the real world.
Oversimplification
We humans tend to oversimplify everything. After all, almost half of the men surveyed thought they could land a passenger plane in an emergency. This oversimplification leads us to underestimate the jobs that others do, possibly seeing them as a task or two. So, when ChatGPT passes the bar exam, we assume that lawyers’ days are numbered.
This oversimplification is also exploited by companies trying to push their wares. This claim is more absurd performance art. No, there will not be any meaningful replacement of employees next year due to AI. The reality is that most jobs aren’t a task or two but collections of tasks. Most single-task jobs have already been automated. It’s why we don’t see elevator operator as a current career choice.
Being Seen as Experts
Many people seek content to share to be seen as experts. If you don’t believe me, have you logged in to LinkedIn lately? This adds to the massive amounts of noise on social media platforms. However, it’s often just parroting others.
This also extends to the tech media. I wish these people would start adding a modicum of skepticism and asking these people hard questions instead of writing articles about model welfare and how we should treat AI models. But once again, novelty over reality.
Conclusion
We are witnessing people attempting to shape a future with vibes and hype. This is the opposite of evidence. It certainly doesn’t mean their future vision is wrong, but it sure as hell means it’s a lot less likely to happen. Reality is a lot more boring than dystopian sci-fi.
I do believe that these tools can be disruptive in certain situations. If we are being honest, I feel much of the disruption is happening in all the wrong areas: creative arts, entertainment, music, etc. We’ve already seen these tools disrupt freelance marketing and copywriting jobs. These areas are disrupted because the cost of failure is low. There will even be niches carved out in more traditional work, too. So, even without AGI and ASI, disruption can still happen.
However, the predictions made over the past few years have been silly and absurd. If you believed many of the people peddling these views, we should be exploring universal basic income right now due to all of the job displacement by AI. But that’s certainly not the case. Many of these same people resemble doomsday cult leaders preaching the end of the world on a specific date, only to move the date into the future because of a digital divine intervention. The reality is, this is vibe misalignment. This is not only going to continue, but increase before it levels out, because investors don’t invest in normal or boring.
Let’s all take a breath, reflect, and maintain our sanity.
