Following the pace of AI advancement can make you feel like the Blown Away Guy from the old Maxell commercials. Tech leaders and influencers tell us to expect artificial superintelligence in the next year or so, even doubling down on inevitability by moving up their timelines. The world is cooked. This perceived inevitability, combined with uncertainty, is leaving many people on edge, and rightly so. If all the things the tech bros hope for come true, humanity is in a terrible spot.
All is not lost. The over-the-top predictions we are bombarded with daily often equate to nothing more than performance art. The tech media frequently parrots perspectives from people who have a vested interest in selling us stuff. I mean, after all, why would they embellish or lie!???
In early April, the AI 2027 thingy was making the rounds. For those unfamiliar, you are in for a treat. The result answers what would happen if you locked a few tech bros in a conference room for a day, depriving them of any reality and oxygen.
Are the scenarios outlined in AI 2027 impossible? Certainly not. This sort of fast takeoff scenario is possible, but it’s highly unlikely. I predict the whole AI 2027 thing will start looking pretty silly in late 2025 or early 2026.
With all this endless AI advancement hype, I was happy to see a new article by Arvind Narayanan & Sayash Kapoor titled AI as Normal Technology. This article doesn’t talk about how AI displaces the human workforce or about a super-intelligent AI taking over the world, but rather about how AI becomes a normal technology that blends into the background of our daily lives.
They also touch on a few other topics, such as overregulation. I also believe that any regulation should be specific and targeted at use cases, not painting with broad strokes. This specificity wouldn’t allow for regulatory capture or weaponization of the regulations. The tech leaders are right that regulation can stifle innovation. By targeting regulations in this way, we can protect people without stifling innovation.
It’s a good read that’s well thought out and researched. For anyone mainlining AI hype, this is an essential read. The scenarios in the AI as Normal Technology article are far more likely than the AI 2027 one, by far.
Questioning The One True Faith
Starting in early 2023, I added a slide with the following image to my presentations. This is because any criticism of advancement was seen as an affront to a spiritual belief, and since I didn’t believe that LLMs would lead to AGI or ASI, I must hate the technology outright. This couldn’t be further from the truth.
Saying that LLMs won’t become ASI isn’t a blasphemy that requires self-flagellation afterward. We don’t need AGI or ASI for these tools to be effective. We can and are using them to solve problems today. People are using them to augment their jobs today. So, why turn AI beliefs into a religion? People are acting like questioning any part of the narrative makes someone a non-believer or some disconnected fool. The reality is that not questioning the narrative or exercising any skepticism is what makes someone a fool. A gullible fool at that.
The reality is that not questioning the narrative or exercising any skepticism is what makes someone a fool.
There’s a strange group that thinks belief is required for AI to create a utopia, but the reality is that facts don’t require belief. It’s ancient wisdom from five minutes ago that we’ve seemed to have forgotten in the vibes era.
I believe what we encounter here is a problem in perception caused by both our environment and us.
Environment
In the book Nexus by Yuval Noah Harari he describes the witch hunts as a prime example of a problem that was created by information, and was made worse by more information. For example, people may have doubted the existence of witches, having not seen any evidence of witchcraft, but the sheer amount of information circulating about witches made their existence hard to doubt. We are in a similar situation today with beliefs in AI advancement. This is made worse because the systems we use today reduce the friction in information sharing, making it much easier to get flooded with all sorts of information, especially digital witches.
We humans also gravitate toward information that is more novel and exciting. It’s the reason why clickbait works. However, novel and exciting information often doesn’t correlate with the truth or reality. As Aldous Huxley pointed out in Brave New World Revisited, “An unexciting truth may be eclipsed by a thrilling falsehood.” We are in this situation again. The vision of near-term artificial superintelligence is exciting and novel, even when people talk about it destroying humanity. AI, thought of as normal technology, as Narayanan and Kapoor put it, is boring by contrast, despite being more realistic.
This condition was the same back in the times of the witch hunts as well. The belief that witches were roaming the countryside looking to corrupt everyone, meaning you had to use your wits and your faith to defend yourself is a lot more novel and exciting than acknowledging that life really sucks because of the lack of food and indoor plumbing.
But then, there’s another strange type of information we gravitate towards: people telling us what we want to hear.
Ah, yes. Evals as taste. Vibes above all. Skills inessential.
We have allowed the people selling us stuff to set the tone for the conversation on the future. These people have a vested interest in selling us on a certain perspective. It’s like taking advice on a car’s performance and long-term viability directly from the mouth of the car salesman instead of objective reality. I wrote about this last year, saying that many absurd predictions were nothing more than performance art for investors. The tech media needs to step up and start asking some real questions.
Many of the influencers and people on social media are parroting the same perspective as the people selling us stuff because of audience capture. Audience capture, for those unfamiliar, is the phenomenon where an influencer is affected by their audience, catering to it with what they believe it wants to hear. This creates a positive feedback loop, leading the influencer to express more extreme views and behaviors. People get more likes and clicks by telling people more exciting things, as Huxley mentioned. So, there’s a perverse incentive for doing so.
Lack of Reflection
One of my biggest concerns is that we’ve lost our ability to reflect. Many things we believe are silly upon reflection. Unfortunately, our current information environment conditions us to reward reaction over reflection. Until we address this lack of reflection, we’ll continue to be fooled in many contexts, not least of which is the pace of AI advancement.
Benchmarks
Many of the benchmarks that people use for AI are not useful in real-world scenarios. This is because the world is a complicated place. Benchmarks are often not very useful in real-world scenarios due to additional complexities and edge and corner cases that arise in real-world use. Even small error rates can have significant consequences. But don’t take my word for it, take it from Demis Hassabis. “If your AI model has a 1% error rate and you plan over 5,000 steps, that 1% compounds like compound interest.” All of this adds up to much more work, not superintelligence next year.
Us
Fooling Ourselves
We have a tendency to fool ourselves easily. As I’ve said many times, we are very bad at constructing tests and very good at filling in the blanks. The tests we create for these systems end up being overly simplistic. Early on, people tested model capabilities by asking for recipes in the style of Shakespeare. Hardly a difficult test, and easily impressive.
This condition is also why every time a new model is released, it appears immediately impressive, followed by a drop-off when reality hits. Sometimes, this has increased problems, such as OpenAI’s o3 and o4-mini models hallucinating at a higher rate than previous models.
We are also easily fooled by demos. Not realizing that these things can be staged or, at the very least, conducted under highly controlled conditions. In these cases, variables can be easily controlled, unlike deployment in the real world.
Oversimplification
We humans tend to oversimplify everything. After all, almost half of the men surveyed thought they could land a passenger plane in an emergency. This oversimplification leads us to underestimate the jobs that others do, possibly seeing them as a task or two. So, when ChatGPT passes the bar exam, we assume that lawyers’ days are numbered.
This oversimplification is also exploited by companies trying to push their wares. This claim is more absurd performance art. No, there will not be any meaningful replacement of employees next year due to AI. The reality is that most jobs aren’t a task or two but collections of tasks. Most single-task jobs have already been automated. It’s why we don’t see elevator operator as a current career choice.
Being Seen as Experts
Many people seek content to share to be seen as experts. If you don’t believe me, have you logged in to LinkedIn lately? This adds to the massive amounts of noise on social media platforms. However, it’s often just parroting others.
This also extends to the tech media. I wish these people would start adding a modicum of skepticism and asking these people hard questions instead of writing articles about model welfare and how we should treat AI models. But once again, novelty over reality.
Conclusion
We are witnessing people attempting to shape a future with vibes and hype. This is the opposite of evidence. It certainly doesn’t mean their future vision is wrong, but it sure as hell means it’s a lot less likely to happen. Reality is a lot more boring than dystopian sci-fi.
I do believe that these tools can be disruptive in certain situations. If we are being honest, I feel much of the disruption is happening in all the wrong areas: creative arts, entertainment, music, etc. We’ve already seen these tools disrupt freelance marketing and copywriting jobs. These areas are disrupted because the cost of failure is low. There will even be niches carved out in more traditional work, too. So, even without AGI and ASI, disruption can still happen.
However, the predictions made over the past few years have been silly and absurd. If you believed many of the people peddling these views, we should be exploring universal basic income right now due to all of the job displacement by AI. But that’s certainly not the case. Many of these same people resemble doomsday cult leaders preaching the end of the world on a specific date, only to move the date into the future because of a digital divine intervention. The reality is, this is vibe misalignment. This is not only going to continue, but increase before it levels out, because investors don’t invest in normal or boring.
Let’s all take a breath, reflect, and maintain our sanity.
By now, you’ve no doubt heard of the term vibe coding. It’s become the favorite talking point from influencers and the tech media, which, even in 2025, can’t seem to muster a modicum of skepticism. But, if you’ve ever wondered what it was like to play Russian Roulette in reverse, loading all the chambers but one, spinning the cylinder, and having a go, you’re in for a treat. Welcome to the world of vibe coding and YOLO mode, two things that go together like nitroglycerin and roller coasters. So, of course, it’s become one of the hottest topics right now, and it has all of the bros super psyched.
For those out of the loop, vibe coding is “Where you fully give in to the vibes, embrace exponentials, and forget that the code even exists.” You get to this state by talking with a computer and letting it generate all the code while you kick back and enjoy a cold one. YOLO mode is the spirit animal of vibe coding. It’s where you blindly accept all the code suggestions generated by the AI tool and push the code to see what happens. Neat. It’s interesting to note that YOLO mode in video games means if you die once, you are dead. No respawning.
Vibe Coding = Code Slop
Before we probe the issues with vibe coding, let’s take a step back and look at what vibe coding is. This practice shouldn’t be confused with a developer using an AI coding tool to assist with tasks or gain productivity. You know, the intended use of many of these tools.
Vibe coding is a delusional dream state in which people use these tools as if they are in the future instead of the present. The fact is, these tools aren’t reliable enough or mature enough to be used this way. It’s a lot like having an illness and getting a bottle of pills from the pharmacy marked with clear directions and immediately slamming the entire bottle because the instructions are for losers who don’t know how to hustle.
Vibe coding is a delusional dream state in which people use these tools as if they are in the future instead of the present.
The reality is that vibe coding generates slop that happens to be code instead of media categories, meaning that the negative consequences extend far beyond other categories of slop. This distinction is essential when evaluating the potential value of vibe coding.
Vibe Coding Pitch
The pitch of vibe coding is that literally, anyone can become an instant millionaire developing apps. You’ll be too busy making money to worry about how your code works or its security. There are no boundaries or barriers. All you need is an idea (more on this in a future post.) Like so many things, it’s technically true but practically false. And also, you aren’t wrong if you are beginning to get crypto bro vibes.
Much of the same logic employed by conspiracy theorists is at work here. If one person gains some success vibe coding, then it must be possible for anyone to do that. Technically true, but practically false. This is like thinking millions of new rock stars will be minted because people can publish their songs on Apple Music.
It’s about tradeoffs. We don’t say that being a conspiracy theorist is a good thing since some conspiracy theories turn out to be true. That’s because the negative impacts of conspiracy thinking outweigh the potential benefits. The same applies here.
There are other flaws with their logic. For example, people don’t consume apps the same way they do media like music, video, or photos. People can listen to hundreds of songs every day and not repeat a single one, but this consumption strategy doesn’t apply to applications. People these days often consume media passively. For example, people often don’t listen to music when they are listening to music; it’s purely background noise for other tasks. Applications usually can’t be consumed passively and require active interaction. This would make consuming many different apps irritating to users.
There are already 1.9 million apps on the App Store alone. Are we really hurting for apps? Do we need 100 million apps to compete with the number of songs on Apple Music? Of course not, but that doesn’t mean we won’t get them anyway. If you look at the outputs of vibe coding, it’s often uninteresting, overly simple, derivative, or just plain unwanted. Buckle up.
There will undoubtedly be exceptions, just like the person who was a bartender six months previously who started a crypto project and manages 100 million in assets. These are exceptions and not the rule, but these exceptions serve as accelerants to fuel the hype flames.
People are trying to sell us on the fact that vibe coding has no downsides. This is delusional. Take a step back and think of the answer to a single question. What do you get when everything takes a backseat to speed? It’s like a car that does 200 mph with no seatbelts and no airbags constructed from paper mache.
Before we move on, I’d like to acknowledge something. It’s a good thing that AI coding assistants are making coding more accessible. However, using these tools as a drop-in replacement for common knowledge and domain expertise isn’t a recipe for success. Imagine something like vibe surgery? Yeah, bro, let me get in on that appendix! Nah, I didn’t go to medical school or know much about anatomy, but I got an AI tool, dexterity, and a good sense for vibes. So little of developing an application is about the code itself, but that gets lost in the vibes.
Changing Behavior and Attitudes
In the public sphere, the discussion of the merits and drawbacks of vibe coding and YOLO mode are entirely contained within the technical aspects of the approach. I’m also concerned about the technical components, but I’d like to bring attention to something nobody discusses.
As often happens when a new technology or approach arrives and removes friction, it changes people’s behavior and attitudes. In technology circles, friction is discussed as though it’s universally bad. It’s not. Sometimes, friction is a feature, not a bug. Nicolas Carr provides an excellent example in his book Superbloom, which discusses introducing the Retweet feature on Twitter.
The time and effort required to share a tweet manually, though seemingly so small as to be inconsequential, turned out to be vitally important. It slowed people down. It gave them a moment to reconsider the message they were about to repeat, now under their own name. Once that little bit of friction was removed, people acted and reacted more impulsively. Twitter became more frenzied, more partisan, and much nastier.
Things like vibe coding and YOLO mode will have similar behavioral effects if this technology trend takes off. People won’t put a lot of thought into the apps they create. Some may build apps purely because they can, not considering why an app for that particular purpose didn’t exist in the first place, assuming that it was purely because nobody had built it and not because of the potential for negative impacts or harm.
The removal of so much friction removes not only the appreciation for the problem but also opportunities to catch potential issues. These lines of generated code become grenades with various time delays chucked into production. This assumes that the developer had the skills to identify the issues in the first place.
These lines of generated code become grenades with various time delays chucked into production.
Some will argue that these features are great for prototyping and mockups. I agree. However, as I mentioned, these features change behavior, and using them simply for prototyping won’t hold. A vast majority of people who can get away with chucking vibe-coded apps into production will.
With the changes in behavior and attitudes, there are many things creators of these applications are more likely to do.
Act unethically (ethics don’t align with speed)
Devalue the work of others
Not learn or at least learn lessons
Encounter skills atrophy
Not build robust software (Security, Privacy, Alignment, Reliability, etc.)
Not constrain code to prototypes and mockups
Think they know things they don’t (Illusion of knowledge, Illusion of capability)
Misunderstand what’s valuable
Devalue collaboration
Go it alone and not include domain expertise and misunderstand the problem they think they are solving
Build apps that nobody wants
Build apps that cause harm
Choose poor architectures
Use more resources and not prioritize efficiency
Fail to benchmark properly
Not be able to troubleshoot their own creations
Not do something truly innovative
And on and on…
These were just a few of the conditions off the top of my head. What happens when these conditions now become the norm? When people start making app slop the way they do image slop?
Risk and Security
Vulnerabilities in code and lack of security controls account for a lot of pain and financial loss every year, much of this from organizations that try to do the right things. So, imagine what happens when people don’t care about doing the right things.
It’s known that these tools output vulnerabilities at rather high rates. So imagine what happens when people YOLO code into production and don’t check the resulting code or even the environment where it is hosted for security issues. Hustlers ain’t got no time for the right things.
There’s more to worry about than an AI tool outputting specific code blocks that are vulnerable. Other contributing issues increase the attack surface of an application. For example, choosing a vulnerable library or suggesting vulnerable configuration options for cloud environments. These tools also contribute to library bloat by including multiple libraries that do the same thing.
I could go on and on about this topic, but at this point, the various security issues created by AI coding assistants are known issues. I wrote a white paper on this topic in early 2023, and I delivered a presentation at InfoSec World the same year. These issues should be common knowledge now with the publication of various articles, papers, and presentations.
When it comes to risk, sure, all vibes aren’t created equal. A video game getting hacked isn’t as bad as a financial application getting hacked and draining your bank account. I’m certainly not being an absolutist here. However, technology trends have an odd way of not staying confined to specific buckets. So, we’ve got that to look forward to.
Today, countless vulnerabilities are moving into production without vibe coding, all because people are trying to push things faster. Vibe coding and YOLO mode make this monumentally worse. We’ve only discussed security and haven’t touched on other topics like privacy.
Making Software Worse
The trend of vibe coding will make software worse. Like security, software quality isn’t a consideration in vibe coding because reasoning about quality is a bummer when huffing vibes.
We live in a highly advanced world where digital things fail all around us all of the time, like a leisurely stroll through a cityscape where freshly painted buildings mask a crumbling interior of decay and misshapen architecture. This is so common that there’s a term for it: enshittification. We’ve become so accustomed to the software and tools we use sucking so bad we hardly notice it. This is a contributing factor to why some view generative AI as AGI.
Vibe coding and YOLO mode will lead to failures, half-baked functionality, and mountains of technical debt.
Vibe coding and YOLO mode will lead to failures, half-baked functionality, and mountains of technical debt. This should concern everyone, but queue the bros to claim this is a feature.
Yes, because that’s how things work. He and many like him are stating that they should create as many problems as possible because AI can fix them in the future. Once again, they are taking something technically true but practically false. At some point, we’ll have highly advanced and capable systems that operate this way, but the mistake is thinking those systems are on the cusp of arrival. It’s hard to ignore the religious fervor in these claims.
People can pray to the gods of gradient descent and burn Gary Marcus in effigy, but it doesn’t change the realities on the ground. Problems created today will be with us tomorrow, and no AI god is coming to deliver us from our evils any time soon, so we should work to minimize potential problems instead of running up the credit card. I’ve been calling this problem out for the past couple of years, stating it would lead to a brave new world of degraded performance.
On a side note, I feel these people feed off each other. I’ve heard perfectly reasonable people making wholly unreasonable claims. These are the things you hear people say when they are trapped in filter bubbles, getting high on the supply of techno-utopians. They also suffer from a healthy dose of audience capture because, no doubt, being unreasonable gets you more likes and shares than being reasonable. Welcome to the perverse incentives of modern social media.
There continue to be many misconceptions about software development, but one of the biggest is that writing code is the end of the journey.
There continue to be many misconceptions about software development, but one of the biggest is that writing code is the end of the journey. This is because most people opining on the topic are not developers. I noticed this trend years before the existence of AI coding tools when security professionals who learned to write a few lines of Python code thought that developers’ jobs were easy. The assumption, then, for AI coding tools is that since the tool can output code and developers only write code, developers are no longer necessary. Developing code isn’t the end of the journey. It’s the beginning.
The written code must be troubleshot and maintained, and features must be added. We live in a constantly evolving world with changing problems, environments, and customer needs. Developed code will crash into the realities of the real world both when it’s initially deployed and over its lifetime. This leads to another problem.
Developers don’t understand the code being written, especially when the people generating the code aren’t developers. As developers’ skills atrophy and people who were never developers start creating these applications, they cannot troubleshoot problems, effectively add features, or perform any of the other countless tasks that developers perform daily. The answer from the utopians is to use AI to figure it out, but this strategy won’t always work.
There is a higher likelihood that the AI tool will successfully troubleshoot issues for simple tools and scripts, but these are the very types of applications that are unlikely to net you big money. As applications grow in size and complexity, the AI tool is less likely to provide the solution necessary to resolve the issue.
Imagine a world where an app needs to be rewritten from scratch because the person who created it couldn’t get the AI tool to troubleshoot and fix the problem. Now, that’s the utopia we’ve all dreamed of.
Imagine a world where an app needs to be rewritten from scratch because the person who created it couldn’t get the AI tool to troubleshoot and fix the problem.
There is a vast oversimplification of the entire landscape here. So, an application starts simply enough, and then more requests are made to the AI tool in an attempt to add more functionality, but this doesn’t always work or isn’t done in the most efficient way, leading to a buildup of issues.
Another trend affecting application reliability is using probabilistic systems as though they are deterministic. Whether this trend is due to laziness, ignorance, or an attempt to handle unknowns is unclear, but it will surely affect applications’ reliability and their ability to be manipulated.
Ultimately, we may be left with App Store decay, where the App Store becomes a graveyard for abandoned apps. RIP.
Making Us Worse
I mentioned skills atrophy in my laundry list. It seems that even AI tools understand this problem. This is not only a comical error message but contains a truth.
We Never Learn Lessons
Although arguably more intense and a bit different, what’s happening now in AI isn’t new. We’ve gone through these cycles before with previous technologies. Every time a new technology comes along, we discard the lessons we’ve learned, assuming they no longer apply, only to discover that the previous lessons were even more important with the new innovation. This condition is something I’ve referred to in my conference talks as the emerging technology paradigm.
We never seem to learn lessons from our previous mistakes, no matter how often we encounter them. We have incredibly short memories and seem to dive face-first into the pool without checking the water level.
Ultimately, it’s all about tradeoffs. What we get and what we lose. When viewed simply as writing code, it seems we are getting more than we are losing. However, building and deploying applications and solving problems goes far beyond code. When considering the impacts holistically, this doesn’t appear to be a good tradeoff. However, it’s possibly one we are going to get anyway.
Whenever a new technology or approach comes along, proponents always pitch it with a utopian eye. They envision all the perfectly aligned scenarios with dominoes falling exactly into place. The Internet, Social Media, The Cloud, Web3, and many other technologies all diverted away from these visions and adapted differently than expected. Even something as simple as the telegraph was seen as a utopian invention that would end world conflict. After all, how could people go to war when misunderstandings were a thing of the past? We all know how that turned out. Vibe coding is destined for a similar fate.
Is it possible to play Russian Roulette in reverse without devastating consequences? Sure, but the odds aren’t great. The world also won’t be a better place with everyone vibe coding and YOLOing stuff into production. Many disagree with me. Fair enough. However, if this trend takes off, it will be another example of something we are stuck with, which is not good for a world that runs on software. We will need to improve or invent new technology to solve the problems we create, trading one set of problems for another. Welcome to utopia.
I’ve been considering a frightening prospect. What if the next generation is known as The Slop Generation? A generation unaccustomed to the world before the inception of AI-generated slop. Despite successes or failures with generative AI, it seems the AI-generated cat is out of the slop bag with the power to warp artistic expressions and further devalue art and the creative process.
This Slop Generation will be the most technologically advanced yet least capable and emotionally unstable of any generation. This is a byproduct of the value of life’s undertakings being picked clean by the vultures of innovation, leaving only the bones littered across the barren landscape. This has far-reaching consequences, but here we focus on art.
Art
When we think of art, we tend to think of images, but art encompasses different mediums, such as music, images, video, and the written word. The results of the artistic process are cultural artifacts with lasting permanence. This permanence can raise some artistic works into modern consciousness despite the passing of centuries, think The Mona Lisa, Beethoven’s 5th, or even the petroglyphs on Newspaper Rock.
Wikipedia defines art as:
Art describes a diverse range of cultural activity centered around works utilizing creative or imaginative talents, which are expected to evoke a worthwhile experience, generally through an expression of emotional power, conceptual ideas, technical proficiency, and/or beauty.
I mentioned the petroglyphs, but we have far older examples of art than these. Neanderthals drew cave paintings 64,000 years ago. Art predates society.
These artistic expressions lead to wonder. What were they trying to say? Were they documenting something? Were they trying to tell people something? Or were they having fun? The mystery is part of the allure. You get none of this with AI art. There’s no wonder, no mystery, no deeper meaning, just slop. Nobody wonders what the AI was trying to say when it adds an additional finger to a generated picture. When a human does it, we search for a meaning.
For over 64,000 years, art has been a part of us, but this may be coming to an end as people hail the post-human era. Enter The Slop Generation.
The Slop Generation
The Slop Generation won’t be defined by their cultural constructions and artistic expressions but by the output of one-arm bandit slop machines where artistic expressions aren’t created to be admired and revered but as individualized wallpaper blending into the background noise of life. A reincarnated version of the noisy, animated GIF-ridden world of early Geocities pages with higher fidelity and less permanence.
All art will be ephemeral, a momentary passing unworthy of saving and revisitation. Shrimp Jesus is their Mona Lisa, and some anonymous rando using a throwaway prompt is their Davinci, but this work isn’t held in any esteem, quickly forgotten and tossed into the low attention span waste bin of modernity.
All art will be ephemeral, a momentary passing unworthy of saving and revisitation.
Low attention spans won’t allow for appreciation, detail, or discerning deeper meaning or context in an artistic representation. This means all art must be literal, or else it is completely misunderstood. A child with a fork next to an electrical outlet will always be just that and never a representative warning in the context of larger cultural issues. The medium is the message.
In Generation Slop, no artists create works for others to enjoy, only algorithmic outputs catering to whatever whim we have at the moment. These outputs are created for an audience of one. Everything pushed and nudged to the dense center of a data distribution creating more predictable outputs and algorithmic uniformity. Why listen to someone else’s slop when you can generate your own?
None of this is a problem for The Slop Generation since art should look cool and serve no other purpose. Fidelity should trump meaning, as art requires consumption at a glance. Art is worthless if it can’t be consumed while multitasking. Any art requiring attention won’t be acknowledged, much less appreciated.
Art from the Slop Generation is cold and disconnected from humanity, unable to strike a chord or illicit emotion. Those trying to use slop machines to illicit emotions will have to resort to extremes, often to get any attention at all, much less an emotional response.
With AI-generated novels, they become worthless wastes of space existing purely for the vanity of the person who generated them, but even these people won’t read their own novels.
In Generation Slop there’s no sense of style, no sense of taste, no investment of any kind. It is the perfect low-attention-span content for the devalue generation as the point becomes getting to the bottom as fast as possible.
With art like music, the pitch will be you can hear a different song every time you listen without one repeating, and this will be seen as a benefit, even though this generation wouldn’t know if a song is repeated anyway since nobody actually “listens” to music. Music becomes background noise for other life activities, nothing more than a sonic firewall to keep the outside world out.
Traditional art, like the art in museums, exists purely to stand in front of for a quick selfie and post to social media. There’s no time for appreciation. Those notifications won’t check themselves. Art will continue to lose its value as nobody appreciates it anymore. The human work and toil that went into their creation are unappreciated because everything seems so easy, and doing anything hard is a waste of time.
This is The Slop Generation’s perspective as they rush headlong into post-humanism.
The Downward Slide
This shift to rapid-fire slop will be a net negative for humanity, as we lose something fundamental to the human experience connecting people and cultures for time immemorial. Art helps us understand the world, other people, and even ourselves in a way other mediums can’t convey. An entire generation may never discover the benefits of creating and consuming art, along with the positive effects this brings.
Take the novel, for instance. I hear so many people brag about never reading fiction as though it’s a badge of honor, assuming that nonfiction is the only way to “learn” something from reading, but this couldn’t be further from the truth. Works of fiction are vehicles for conveying big ideas. That’s why most books banned throughout history were works of fiction. Life isn’t a tutorial and doesn’t align with specific steps.
Imagine trying to convey the message of dystopian works like Orwell’s 1984 or Huxley’s Brave New World in a nonfiction format. It doesn’t strike the same chord or have the same impact. Both authors wrote essays and discussed these concepts in interviews, but these never created the impact that the novels did.
Fiction written by humans can be instructive. It can help us understand situations, other people, and cultures in ways other mediums cannot. Even movies with all of their visual and audio aspects don’t bring us inside the heads of characters like written works of fiction do.
Written works invite us to participate, exercise our imagination, and consider our own thoughts and perspectives. But this invitation requires our attention and an investment of time, attributes many don’t know how to exercise and believe they don’t have. Without exercise, we lose our imagination and our ability to connect. This is why, despite being so connected, people feel more disconnected than ever, mistaking digital connection for human connection.
Written works invite us to participate, exercise our imagination, and consider our own thoughts and perspectives.
When all art is literal, it loses its sense of mystery and wonder but also has degrading knock-on effects. At a local writer’s group meeting around 2018, I watched attendees dole out their usual mixture of helpful and non-helpful feedback to a girl on the chapter she’d written. One of the biggest criticisms was show, don’t tell.
I remember telling the attendees I’d been thinking about that piece of writing advice. I was concerned that younger generations may be unable to deliver the required attention to a piece. You may need to write shorter fiction that actually “tells” instead of shows because they won’t be able to infer emotion or meaning from the act of showing. I mentioned I thought it was because of YouTube and tutorial culture, where people won’t try anything without being told everything first. I still think this plays a part, but it may be a symptom.
Ultimately, the foundational concept of show, don’t tell in fiction may need to be thrown out like yesterday’s garbage, as future generations cannot infer deeper meaning from descriptions. If you are unfamiliar with the rule, here’s an example of what the transformation may look like.
She folded to the ground, collapsing like a house of cards as she shut her eyes and tucked her lips behind her teeth.
Becomes.
She was disappointed and knew it was hopeless.
Okay, this is not my best description, but it’s enough for you to get the point: a degradation in quality to meet low-attention-span readers, which makes people avoid reading.
Ultimately, anyone who might have been interested in more traditional art will be turned off by a warped lack of incentives, never having a chance to discover the real value of art and the artistic process. This generation’s inability to delay gratification means any activity requiring practice and investment will be considered a waste of time.
In the future, the creation of art may very well morph into other activities like a video game. This may even be pitched as a game you can play with others. Although this may be fun (video games are fun) and seem like a sort of progress, the devaluation continues, and everything from the previous section applies.
I spent most of this section discussing written work since it made for a better demonstration, but these same principles apply to all other forms of artistic expression.
Mistakes and Misunderstanding
Applying AI-generated works to a low-attention environment leads to a fundamental mistake: mistaking resolution for quality. They aren’t the same. I’ll write more on this later, but you can see this with all of the generated Veo 2 examples. These examples have a high resolution (look good visually) but poor quality (they actually suck).
A human actor playing a part performs the emotions supposedly taking place in the character’s head. These are the types of things that are missing in AI-generated video. The generations are unfeeling, giving the impression of being dead inside. That’s because the actual creator (the AI) was never alive to begin with.
Quite a few people working in tech think people who don’t shouldn’t have jobs. These same people believe programs that don’t lead to technological progress are a waste of money. If you are trying to understand the current moment, look no further.
From this perspective, since art can be easily generated, the value of creating must be minimal, and anything that adds friction to the process must be bad. We need to push back against this narrative, not because it’s misguided but because it’s flat-out wrong.
Use AI For What It’s Good For
There’s a growing AI backlash, which is understandable but misguided. This situation isn’t helped by AI influencers and tech executives making stuff up and overhyping claims whenever their mouths open. Hating the tech because you hate the people making it isn’t a recipe for success.
AI isn’t purely ChatGPT or even LLMs. Many different AI approaches have various benefits for humanity and the potential to accelerate cures for many of humanity’s ails. Notice I used the word accelerate instead of cure. People can wield hubristic ignorance, blinding them to the fact that humans solved problems before the arrival of AI.
We want AI to provide benefits, such as cures for cancer and hunger and ending geopolitical conflict. But solving real problems with AI is hard, especially when AI can actually exacerbate some of these problems instead of solving them. Solving non-problems with AI is easy. This is why we get AI art instead of cures for cancer.
The goal here isn’t AI avoidance but AI selective usage. Don’t use “AI for everything,” as the poor advice goes, especially for art. Put a firewall around activities you value where AI would degrade the activity and use AI for more mundane activities to gain efficiencies. We can have the best of both worlds if we want to. We can cure cancer and protect art. Anyone who claims otherwise is selling something.
If you have kids, leaving room for friction and discovery is important. Let them explore different creative tasks and help them understand the benefits that manifest from the investment of time and effort. They’ll discover that this investment will pay far more dividends than any momentary gratification from AI art as they make new discoveries and learn about themselves. Unfortunately, this isn’t easy and will take work, which is why many take this path. Delay device usage as long as possible, and let them explore without mediation.
Unfortunately, this shift to The Slop Generation is already happening, and we are losing the battle to instant gratification and a sense of false, momentary satisfaction at the sacrifice of lifelong satisfaction. I’m going to leave you with a couple of quotes from Ted Chiang’s article called Why AI Isn’t Going To Make Art.
But let me offer a generalization: art is something that results from making a lot of choices. This might be easiest to explain if we use fiction writing as an example. When you are writing fiction, you are—consciously or unconsciously—making a choice about almost every word you type; to oversimplify, we can imagine that a ten-thousand-word short story requires something on the order of ten thousand choices. When you give a generative-A.I. program a prompt, you are making very few choices; if you supply a hundred-word prompt, you have made on the order of a hundred choices.
Generative A.I. appeals to people who think they can express themselves in a medium without actually working in that medium. But the creators of traditional novels, paintings, and films are drawn to those art forms because they see the unique expressive potential that each medium affords. It is their eagerness to take full advantage of those potentialities that makes their work satisfying, whether as entertainment or as art.
One of the oft-repeated talking points erupting from the mouths of futurists and tech leaders alike is claiming that things will cost nothing in the future. As if we are to believe all of these people are in the business of making something for nothing. The entire claim is a gross absurdity that charlatans like Ray Kurzweil conjured out of thin air, and others parrot at every opportunity. This claim is made with such confidence that it is rendered self-evident, and to question it means you are an out-of-touch dolt lacking the religious fervor necessary to create the techno-utopia.
But these responses are a smokescreen to dispel the very rational questions this claim evokes. None of these people can explain exactly how this will work in practice or are willing to admit just how bad things will get, which seem like consequential details to omit considering the plan to rework the social contract of most of the world.
The claim promises us a Fully Automated Luxury Communism (FALC) where all of our needs are not only met but propels us into a life of luxury. However comforting the concept, the reality may be closer to Fully Automated Digital Breadlines (FADB). I know, how dare I poo-poo the utopia.
The False Choice
We are often given a false choice. We are told that if we don’t allow companies carte blanche to raw-dog technology all the way to utopia, then humanity will vanish. Either grow or die, as the mantra goes. Given this, a minuscule number of people are trying to rework the social contract and reimagine society without society’s input.
We can have cures for cancer and other illnesses without destroying art, stealing people’s work, or removing humans from the creative process. However, curing cancer is a hard problem, and imitating humans is easy. So we get AI slop machines instead of cures for Alzheimer’s.
Maybe I’m just an idiot, but I fail to see how LLMs will make humans immortal. Immortality is one of the many promises if we only just let it happen, even though there’s absolutely no evidence for this.
Also, if you read Andreessen’s Techno Optimist Manifesto from October of 2023, you may notice his crediting of Filippo Tommaso Marinetti, the author of the Fascist Manifesto. Marinetti was a futurist, but his position as a futurist and someone who had complete disregard for the past led him to embrace fascism as a logical vehicle for technocracy.
Don’t get me wrong, there’s plenty in Andreessen’s manifesto that I agree with. We are a society built on technology, and this has brought some of our greatest achievements. There certainly are regulations that seem pointless and get in the way. There are groups inside organizations that have become politicized and create unnecessary obstacles. I also agree with the critique of communism. These are all true. However, Adreessen’s mistake assumes that multiple things can’t be true simultaneously.
Even though these are extreme views that some have labeled techno-authoritarianism, understand that they are the average view of the e/acc community. Andreessen also invokes the perils of communism multiple times while also driving humanity into techno-communism, but to each their own, I guess.
I love technology and believe, as Andreessen does, that technology will deliver the best future. It’s because of technological advancement that we’ll cure cancer and reduce suffering around the world. However, I don’t believe a better society results from discarding ethics and principles and disregarding voices different from our own in the pursuit of generating a cornucopia of innovation porn. We in technology seem to constantly make this mistake, only to be disappointed by our ignorance of the complexities of the real world and the jobs and perspectives of others.
Ethics and principles aren’t obstacles or roadblocks. They are guideposts that ensure what we build aligns with our values and vision of the world we want to create. We, in this case, meaning society as a whole and not just a couple of dudes sharing technology with their friends.
The Claim
If you have escaped these claims, here’s a recent example from Marc Andreessen below.
You read that right. We need to hurt you before we can help you. It’s the sort of pitch you’d hear from a sadistic boyfriend who insists he needs to tear a partner down before building them back up. The we have to break it before we can fix it mantra is applied to almost everything, including humans and the environment. This is the core premise of the Effective Accelerationist (e/acc) movement.
But Andreessen is hardly the only one making these claims.
That’s right, Google is in the business of giving you things for free. We’ve learned this lesson a long time ago. Yes, Google makes its money off of ads for its “free” services. However, in a future where things are worth nothing and people don’t have an income stream, it seems likely that advertising budgets will be zero as well.
I blame much of this on Ray Kurzweil. For years, he’s been peddling this nonsense. I addressed this very same claim in my post on his latest book in the “Things Will Cost Nothing” and “Jobs and Wages” sections of the article. Despite this, I wanted to explore this topic further.
These people claim we shouldn’t worry about losing our jobs to AI because AI will make companies so good that goods and services will essentially be cheap or free. But both on the surface and upon reflection, the claim is absurd.
Nobody can describe exactly how this is supposed to work other than sprinkling everything with AI magic. When someone does make an attempt, like Ray Kurzweil, for example, the explanations make no sense, don’t address the questions, and highlight how little about the real world these people know.
For years, I’ve been pushing back against the phrase, “AI won’t replace people. People with AI will replace people without.” This is just patently false. The moment AI is good enough to take our job, it will. I mean, it doesn’t even have to be that good.
So, no job, no income. This is our baseline. It doesn’t matter how cheap things get if you have zero.
But AI, Tho
Before we get too far, let’s address the counterargument. For all the issues I’m about to raise, the answer is, “But AI, tho.” The response involves invoking the name of AI like a magician conjuring a spell. We are told that AI will be so great and powerful, rising to the status of deity, and no matter what the encountered issue, AI will figure it out. But merely spouting an incantation doesn’t make it a reality.
This answer is a complete copout that leaves the questioner unsatisfied. Whenever someone invokes the But AI, Tho defense to real questions, continue to ask them for more specifics. Don’t allow the oversimplification of a vast and complex problem space. AI isn’t god, and they aren’t prophets.
The “Sucks To Be You” Gap
Remember, we need to be broken before we can be fixed. This means there will be a gap between the damage incurred and any mitigation strategies. I call this the Sucks To Be You gap. There is no telling how long this gap will stay open or what mitigations will be implemented to remedy it.
Unemployment is unlikely to hit something like 90% all at once. This would mean that early people displaced by automation would be the most harmed since they would be unable to support themselves and their families and have no real recourse for their situation. How long will this drag out? My guess is years, possibly a decade or more, depending on how slow the adoption is and any difficulties implementing mitigations.
The amount of harm caused by this gap is unfathomable. This gap brings pain, suffering, and death. If you think I’m being dramatic, think about it for a moment. Imagine the mental toll this takes on someone trying to provide for themselves and their family. This isn’t a matter of re-skilling. Even if people did re-skill, the competition for remaining jobs would be astronomical, with thousands of applicants for a single position. This isn’t p(doom) it’s p(shit).
It’s easy to see how self-harm could result from this situation, but that’s not the only scenario where mortality is concerned. Not working leads to a lack of benefits, meaning you can’t make co-pays on doctor visits and prescriptions. This doesn’t include all of the potential harm from algorithmic decision-making mistakes. Deaths will result, and we know this because we’ve seen it happen on a smaller scale with people not being able to afford insulin.
No Intelligence Explosion
All of the claims of a near-future techno-utopia are predicated upon an intelligence explosion. This is the condition in which AIs will recursively improve, creating even better AIs that morph into superintelligence. Advocates claim this attainment of superintelligence fuels this world of comfort and abundance. But what if it doesn’t manifest this way? What if we get the Diet Coke of AGI? Just one calorie, not intelligent enough.
The assumption is that superintelligence brings massive productivity gains, but what if, instead, we get algorithms that are purely good enough, leading to human workers being displaced and productivity staying relatively the same? For example, an agent can work 24 hours a day, but what if that 24-hour-a-day agent produces the same productivity as a human working 8 hours a day? This could happen because of needing to account for errors, wait times for additional reasoning, running tasks multiple times, and other issues relating to the complexities of seemingly simple tasks. It’s easy to see how this can stretch out when we factor in additional difficulties of completing complex tasks.
This human replacement could result in cost savings but would be far from driving costs to zero. Also, this would be less of a complete human replacement and more of a human staff reduction. Now, you have a displaced workforce and a company with similar productivity. This doesn’t seem like a recipe for a utopia. It’s a recipe for problems.
This is a very real possibility, especially given all of the hype around LLMs. I know everyone is losing their mind about DeepSeek at the moment, but I don’t believe LLMs are a path to AGI, much less ASI. However, it’s important to realize that we don’t need this level of intelligence to apply these technologies to specific tasks successfully. It’s entirely feasible that a company would take a shitty LLM with repeatable failures over a human worker if they could save money.
What’s The Point of Things Costing Nothing?
I’m not sure anyone gets out of bed in the morning with dreams of creating a company that delivers goods and services that cost nothing. It’s even absurd to say out loud, so you might wonder why people at the largest companies in the world are making this claim. Investors are the same way. Nobody is investing in a company so they can deliver zero-cost goods and services. In the Sucks To Be You gap, the first affected suffers the most harm, but the opposite happens with companies.
Nobody is investing in a company so they can deliver zero-cost goods and services.
Tech leaders and investors aren’t considering what happens to their companies after this so-called intelligence explosion. They are thinking of all the money they will make leading up to it. This is why these staunch capitalists are so comfortable forcing everyone into techno-communism. Now that I think of it, the thought of an algorithmic Stalin hunting kulaks is terrifying.
Stagnation
Counterintuitively, this condition could lead to stagnation. The very opposite of what proponents claim. This doesn’t strike me as a competitive environment where companies and people are stepping up to create new solutions due to a lack of incentives. I guess someone could make the argument that people’s lives will suck so bad that they’ll be incentivized to create something better. Fair enough, but it seems like these bigger initiatives would cost more money, putting them out of reach by these very people. Not to mention, this is an odd flex for the techno-utopians. “Your life will suck so bad you’ll be dying to create something better.”
The price of stagnation for a majority of the population is that they remain in the mire of the Sucks To Be You gap for a much longer time. Even if basic necessities are met, it will be miles away from a good life, much less luxurious.
Things Will Still Cost Something
The core premise of the argument that things will be zero or low cost is absurd on its face, so much so that it’s remarkable that nobody seems to push back. A whole host of things won’t be free or low-cost. Consider rent and property, the means to generate electricity, medical treatments, and, most importantly, food. Even extracting and refining raw materials is going to cost something. Imagine being monitored every moment with everything in your home subscription-based, requiring a micro transaction for nearly everything you do. Now, that’s the utopia we’ve all dreamed of!
Regarding food, Kurzweil claims that advancements in vertical farming will make abundant, nutritious food freely available. This highlights Kurzweil’s cluelessness on a variety of topics. Vertical farming took a hit last year, making MIT Technology Review’s list of the worst tech failures of 2024. Score another “L” for Kurzweil.
As I mentioned, companies and investors aren’t in the business of giving things away for free. These companies will adjust to the conditions imposed upon them. When have we ever seen a company that gets hit with higher taxes or additional tariffs responding with, “Well, sucks to be us. I guess we’ll have to make less money now.”
This condition may level out at some point. After all, if nobody has any money to buy your products, that’s not a good business strategy either. I’m saying that this leveling out could take some time, especially if a segment of the population continues to remain employed.
New Risks
New architectures, technologies, and automated processes will bring new risks. Due to our complete dependence on these systems, these risks will have a much larger direct impact. The vertical farming example is instructive because it raises new risks and considerations. For example, damage can spread quickly in these new architectures, creating cascading failures.
In reality, the company’s lettuce was more expensive, and when a stubborn plant infection spread through its East Coast facilities, Bowery had trouble delivering the green stuff at any price.
And this is just one of the many potential examples. Whenever potential challenges such as this are raised, the But AI, Tho defense is invoked as some sort of benevolent deity here to deliver our salvation and absolve us from our sins. “AI will just figure it out.” This is not an answer.
Techno-Communism and Techno-Welfare
Let’s acknowledge that these companies aren’t willing to part with their money. It’s not like they will be so successful that they’ll start sharing their profits with us. Even if they half the cost of goods and services or even reduce by 90%, we’ve got zero dollars, which makes these cheap necessities still out of reach. This begs a couple of questions.
How do companies make money from people who don’t have any?
It seems unlikely to be profitable in this environment, so companies raise prices for those who can afford their products to cover gaps. This situation actually makes it worse for displaced workers, as I mentioned previously in the adjusting to market conditions section.
What’s the remedy?
Some have proposed an automation tax that funds a Universal Basic Income (UBI) program. This sounds good on paper but may not be so great in practice. We will tax people who are making less money; hence, there will be less recovered in taxes. Not to mention, I’m only considering the United States here. What about goods and services from other countries? After all, we have a global economy. This requires tariffs on goods and increased taxes on digital goods, which will require companies to raise costs even more.
There is the impression that the techno-welfare provided by some universal basic income will have us jet-setting around the globe. This is the premise of Fully Automated Luxury Communism (FALC). This is flat-out bullshit when you consider the realities on the ground. UBI’s benefits are a social welfare program and will be commensurate with similar programs.
Nobody on a social welfare program lives it up on their yacht, sipping champagne and wondering when their Ferrari will be out of the shop. These people worry about basic necessities constantly. Any small hiccup can result in major consequences. This future techno-welfare program will be far more like today’s social welfare than some government-funded luxurious lifestyle. So, yes, it is much more like Fully Automated Digital Breadlines (FADB) than FALC.
Not to mention, this very same social welfare program will be administered by the very system systems that displaced these workers in the first place, leaving the door open to a whole host of technical issues and challenges that will affect the people in the program, adding to risks.
The thing that pisses me off about people like Kurzweil is that the very foundation of their arguments is not only so disconnected from reality that they don’t make sense, they are dehumanizing. But for people like Kurzweil, this is a feature, not a bug.
The response to hungry children comes off as, “Just shut up and eat your amino acid paste, you ungrateful little shits. Don’t you realize how much more compute you have access to? You couldn’t even run stable diffusion locally when I was a kid!” When you are hungry, it’s hard to eat your computer.
Reduced Agency and Helplessness
What does it mean to be human in an age without work and agency? Do we resign ourselves to being helpless and needy? This is hard to pin down in advance. Humans are indeed incredibly adaptable creatures, but there’s a limit to this adaptability. But more importantly, why should we settle for this vision of the future?
These systems turn us into robots, shoving us into predictable buckets, reducing our agency, and making us dependent. This is necessary to increase the accuracy of predictions. The result is we end up as helpless schmucks standing on the sidelines, waiting to be told what to do and where to go at the mercy of every algorithmic decision. Technology should work for us, not the other way around, a point that gets lost in the shuffle and hype.
With every new risk that surfaces, we’ll be helpless to intervene. We need to take it on faith that what we built will automatically do something about it, as the world we construct becomes far too complex for us to understand. In some instances, humans may not be informed of impending dangers due to their lack of ability to do anything about them. We remain blissfully aware until the asteroid strikes.
We should insist on better. We deserve something better—technology that works for us, not us working for technology.
Technological advancements require tradeoffs, which will benefit humans as a whole. For example, suppose self-driving cars worked as advertised and delivered on promises. In that case, giving up manual driving for the benefit of safer roads may be a worthwhile tradeoff that most of society accepts. However, today, we are being asked to pre-purchase a tradeoff where it’s unclear what we get and what we lose.
Does This Sound Like Utopia?
I don’t know about you, but this scenario doesn’t sound like a slam dunk in the utopia basket. At best, this sounds like human-forced retirement with a monumental cut in income and benefits. At worst, it’s suffering and death, far from the promised life of luxury. It likely won’t be either of these extremes, but it will be something like a Fully Automated Digital Breadlines scenario I mentioned where the role of humans is needy and dependent.
I’m not sure exactly where I fall on the utopia scale above except to say I am probably not in the upper half. Not a precise measure other than to say away from the luxury lifestyle.
Can we achieve artificial superintelligence quickly and solve the world’s problems by creating a world of abundance? Yes, it’s certainly possible that everything snaps into place perfectly, and governments and corporations work hand in hand to create a world of abundance free from suffering. Possible, just not probable, or at least probable in a reasonable amount of time. For this to be the winning scenario, things must work perfectly the first time with advancements free from issues. We should know from history this is rarely the case.
Even if we eventually reach a reasonable utopia, we’ll have years, if not decades, of pain and misery as humans do their best to adapt and deal with less-than-perfect technology, governments, and companies. All of these challenges are incurred by humans while simultaneously being stripped clean of our agency and purpose.
By some estimations, communism is responsible for 100 million deaths in the twentieth century. Although some dispute this number, even on the lower side, we’re still talking about 50 million people. But hey, what’s 50 million deaths among friends? Something about one death being a tragedy and a million being a statistic. And yes, I know Stalin didn’t say that, but it’s relevant here.
Although I don’t think techno-communism will cut that wide a path, I do believe that some will view resulting deaths and misery as the cost of progress. However, progress is subjective, and despite often being linked, innovation and progress aren’t the same thing.
Conclusion
I hope that none of my predictions come true, that I am wrong, that some fluke happens, and everything magically snaps into place without issue. Thankfully, much of the hot takes on social media can be written off as bros sharing vibes. Also, I don’t think the current crop of LLMs will cause mass unemployment, create large destabilizing effects in the workforce, or create immortality. However, I’m not as confident about this prediction, well, other than the immortality piece.
The real question for LLMs is how much better this buggy, insecure, black-box technology needs to get to start disrupting a larger part of the workforce. We’ve seen this happen in the creative domains, but the cost of failure is low in these use cases. Let’s hope there are no plans to hook ChatGPT up to air traffic control or the nuclear arsenal, but there are still plenty of other jobs without such high failure costs. Only time will tell.
The attempt by a few to change the social contract raises many questions: Who sets the rules? Who changes the rules? Who or what makes the important decisions affecting humanity? These are good questions to have answers to before wading into the slough.
This situation can’t be described as a Faustian bargain since most people won’t gain any true advantage. At least Robert Johnson received amazing guitar skills. Many of us will get digital breadlines and an endless feed of slop.
There is a current push to cram every inch of AI into every conceivable corner of our lives, like someone trying to shove the fifteenth clown into the clown car at the circus. This is a direct result of needing to show investors that the monumental amount of cash being chucked into the furnace is paying off. Consequently, one of the goals is to put this technology even closer to us, giving it hooks into our daily lives in the hopes that it will become indispensable and even addictive.
Often, when someone talks about AI being a threat to humanity, this invokes visions of The Terminator or the scenario of bombing data centers to prevent the spread of evil AI (as if that would help). I don’t take these p(doom) scenarios seriously. However, if we are not careful, I think AI poses an existential risk to our humanity, which is different.
As this technology improves, becomes more reliable, and works its way into our daily lives, playing the role of assistant, companion, and possibly lover, harm will undoubtedly manifest. In this article, I introduce four high-level buckets to consider these harms and discuss something I call a cognitive firewall to protect aspects we value most.
The conversation on AI’s impact is almost universally focused on what we do and how we do it, and almost nothing is said about its impact on who we are and what it does to us. The latter is my primary focus and what many of the articles on this site attempt to address. To be clear, when I use the term “personal AI,” I’m not referring to tools like ChatGPT or Claude. What I’m referring to is the next iteration of these tools that are more connected and more ever-present.
The Assistant and The Companion
The AI technology being developed isn’t constrained to a single task or activity. It can be both an assistant and a companion, and since it can be both, it will be both. I’ve defined six primary personas that personal AI tools will play in daily life.
The Oracle
The Recorder
The Planner
The Creator
The Communicator
The Companion
Given the breadth of functionality supplied by acting as these personas, daily overreliance on personal AI is bound to happen.
In my previous article, I covered why tech companies will embrace this shift but didn’t speak to the direct negative impact on humans. Most of the time, negative impacts are framed around when the system is wrong. For example, if the product tells us to do something dangerous, like eating a poisonous mushroom, or convinces us to self-harm. However, with personal AI tools, the harm extends beyond issues with the system’s accuracy. This means we could have a perfectly functioning tool or product that still produces harm. Let’s take a look at that now.
Negative Human Impacts
As I alluded to in the intro, cognitive outsourcing and overreliance on these tools have negative human impacts. I lump these negative impacts into four high-level categories that I call The Four D’s: Dependence, Dehumanization, Devaluation, and Disconnection. These negative impacts are driven by cognitive outsourcing and the resulting cognitive illusions it creates.
If we are dependent, then we are vulnerable. When we devalue, then we rob ourselves of joy and satisfaction. If we remove fundamental aspects of our humanity, then we dehumanize others and ourselves. If we are disconnected, then we are unaware. There are no firewalls around the Four D’s, so some activities cross all four.
Dependence
Dependence is the core critical harm these systems pose and cuts the widest path. This is because the actions we depend on the tool to perform or provide to us will be both task-oriented and emotion-oriented. Dependence leads to cognitive and emotional atrophy. Today, we aren’t considering how overuse of this technology rewires our brains. This rewiring certainly didn’t start with personal AI. I remember years ago, people were noticing effects with a far more simple technology, Google search. And this was a far cry from a more advanced, ever-present technology with access to all our data. Something that personal AI tools will have.
Skills and Capabilities Atrophy
If we refer back to Sam Altman saying that he forgot how to work without ChatGPT (he wishes ChatGPT was that good), that’s a possibility with near-term personal AI systems. Reduced capabilities are because of cognitive offloading. This offloading is also something I’ve covered before when discussing human augmentation.
Constant outsourcing to technology reduces our capabilities. As an example, let’s look at gaming. Let’s say our companion is always with us, and we use the companion to assist us in playing video games, navigating worlds, and solving puzzles. We come to rely on it. Microsoft’s product is called “Copilot,” after all. In the Personas of Personal AI context, this would be exercising The Oracle and The Planner. However, with this outsourcing, we may forget how to explore the video game world or solve puzzles without this assistance. It’s also possible that children may never develop these skills in the first place. In this example, it’s a video game, but the same holds true for all kinds of human activities.
Emotional Atrophy
The atrophy induced by constant outsourcing to personal AI extends beyond skills and capabilities, affecting our emotional capabilities. Although it can be hard to imagine, we may lose the ability to connect emotionally with our fellow humans. Some might argue it’s already happening. We may even forget how to love as we use AI systems to plug emotional holes and play the perfect friend, lover, parent, and therapist.
Dehumanization
Dehumanization is a word that is often used in extreme contexts associated with the justification of atrocities against other humans, but it’s not always this extreme. If you look up the word’s meaning, you’ll see that the simple definition is to deprive a person or situation of human qualities, personality, or dignity. This is a fitting description since personal AI systems can affect all three of these.
Humanity is on a collision course with dehumanization as charlatans like Ray Kurzweil pitch their nonsense about uploading our consciousness to computer systems, choosing to become disembodied spirits haunting storage buckets of cloud infrastructure. Unfortunately, Kurzweil is not alone.
There are whole movements, such as transhumanism, posthumanism, and even the e/acc movement, that claim humanity is a dated concept, and we need to evolve into something un-human, something more akin to homo technologicus. You even have people like Elon Musk making the perfectly sane argument that we’ll need to remove our skulls to implant more electrodes to communicate with computers. I’ve challenged these narratives before. Needless to say, the road to utopia is going to be paved with a whole lot of damage from a bunch of shitty half-baked tech.
The road to utopia is going to be paved with a whole lot of damage from a bunch of shitty half-baked tech.
I mean, what’s the point of having human friends anyway? Also, isn’t an AI lover preferable to a human? In both scenarios, the AI companion is far more convenient and configurable. I’m not trying to make some obscure point because there’s already a push to dehumanize friendship and love.
Dehumanization is often driven by optimization. As we try to optimize everything, we treat humans like apps, processes, or checklists, not giving them the common decency of interacting with them directly. And if you think this is okay because it’s coworkers or gig workers, you might want to think again.
Devaluation
Finding joy in simple things has become far more difficult in our modern world. We are conned into believing every activity in life is go big or go home. This view is fueled by influencers and social media, creating an inauthentic lens through which to view reality. Due to misperceptions about incentives, it will be almost impossible for younger generations to realize the value of simple things. Small, simple things will appear as pointless wastes of time. But losing sight of the value of simple things is only the beginning.
Take a glance at any tech influencer’s content or listen to techno-utopians ramble on about the future, and you’ll no doubt hear the pitch that the only way to achieve true happiness and success is through optimization. Optimization is your salvation: Father, son, and gradient descent.
This warped view belies the reality that optimization can ruin the value of activities. When every activity is transformed into a sterile checklist with a single goal of being done, we lose sight of the value of these activities and their impact on us.
Writing and art are obvious examples. The result of these activities is a byproduct of the process. This seems counterintuitive to non-creatives and hype bros, but with minuscule reflection, it’s not.
Writing is Thinking and Exploration
As I sit here writing this article, I’m an explorer. Probing the depths of the topic and my mind to create something new. As each point appears, I challenge and surprise myself with generative intelligence not contained in a distant data center but in my skull. The very same skull Elon wants me to remove. This inefficiency has satisfaction and value as I construct new points I hadn’t thought of before. It’s a mistake to think this friction is unnecessary and needs to be removed. The gauntlet of inefficiency imparts discoveries that optimization destroys.
The gauntlet of inefficiency imparts discoveries that optimization destroys.
Writing truly is thinking, exploration, and discovery wrapped into one. Generating content is none of these. At best, generating content is a validation activity, where instead of gaining the benefits from writing, we are merely validating that the system outputs aren’t wrong. Cognitively, these are completely different exercises far from providing the same value.
There are tasks where generating content and validating the output is fine, but we shouldn’t confuse these cases with more meaningful activities where value can be obtained. Sure, I could optimize my writing activities using generative AI and create 365 blog posts covering every day of the year, but it would be of no value to you or me.
Optimization Removes Value
When optimizing artistic endeavors with AI, we rob ourselves of value and deny the formation of our own sense of style. This may seem inconsequential and easy to gloss over to the uninitiated, but this becomes part of our identity. No matter how hard we try, we can’t prompt our way to our own style.
When I look back on the art I’ve created, I’m transported back to when I created it. Memories come rushing back, and I’m reminded of my place in the universe and how I can still surprise myself. There is no surprising yourself with AI. That’s not how AI works in the creative process. For the AI artist, when you are lying on your deathbed, will you reflect on your favorite prompts?
Pretty much every time someone shouts that AI democratizes art, they really mean it devalues it. The great thing about art is that you don’t have to be good at it to enjoy the benefits. You can still explore, surprise yourself, and learn no matter how good you are. This is where the true satisfaction manifests.
The great thing about art is that you don’t have to be good at it to enjoy the benefits.
We are sold on technical optimization, believing that everything we do should be optimized to the fullest extent. However, technical optimization can ruin the value of meaningful activities. Just look at the comment below.
This is absolutely not true. He’s either lying through his teeth or a complete idiot. Given the environment, it’s a tossup. But as the guy working to devalue music, I’m not surprised. Unfortunately, he’s not the only one. Just take a look at the job description below.
Solving real problems with AI is hard. Notice how we haven’t cured cancer yet. However, solving non-problems is easy since imitating humans is easy, which is why we don’t have cures for cancer but countless AI art generators. It’s not like the lack of art in the world was a problem needing to be solved.
We’ve only scratched the surface. We’ve started misinterpreting the value of a whole range of activities as we superimpose issues on top of human inefficiencies. Even the act of reflection, arguably one of the most valuable activities a human can exercise, has been tainted by AI hype. Many things that appear as wastes of time or inefficient have meaningful value.
This is about the point where the hype bros claim I’m anti-tech. I’m not claiming that technical optimization is bad across the board. There are many areas where technical optimization is a tremendous benefit. For example, suppose we can decrease the time to deliver someone the benefits they need or can more efficiently stage resources after a natural disaster. In that case, these are good things, and AI has the potential to make them better. However, this article discusses the activities that provide value in which optimization negates that value or at least a large portion of the value.
The continued devaluation of activities providing value negatively impacts humans and our life satisfaction. The situation could be better than ever, but we perceive that everything sucks.
Disconnection
Never in humanity’s history have we been so connected and disconnected at the same time. Filter bubbles and personal biases warp our information consumption and reality into odd, personalized shapes that rival the most abstract artists. It’s not uncommon for polar opposite views to point to the same data as evidence for their perspective.
Even the most disciplined information consumer can’t avoid being disconnected to a certain extent. Our lens is always filtered by algorithms and selection bias in the digital world. There’s too much information for it not to be. We don’t just have a firehose spraying us in the face with information, but countless firehoses blasting us with thousands of pounds of BSI (Bullshit per Square Inch).
Personal AI systems won’t improve this information landscape; they will make it worse as we insulate ourselves from the real world, fueling further disconnection. Using personal AI tools, we’ll better be able to isolate ourselves in an attempt to make the world more predictable and avoid things we don’t like.
Unfortunately, I feel like an old man yelling at a cloud, and the acceleration into disconnection is inevitable. In my defense, at least I know the cloud I’m yelling at is real. Humans have started to prefer simulations to reality, and tech companies are more than happy to oblige. After all, simulations check all the boxes for our current age: They are predictable, convenient, and comfortable.
Cognitive Firewalls and Purposeful Interventions
As a result of the four Ds, we will be less capable, more dependent, more vulnerable, more prone to manipulation, less aware, unable to connect with others, emotionally inept, and depressed. What a bargain! I came for the capabilities and left with the dope sickness.
Of course, it doesn’t have to be this way, but it will be hard to avoid this result. Avoiding this result will be a heavy lift, and that effort will fall on end users. Unfortunately, the responsibility for defending our humanity falls to us. Each of us has different attributes we value and would like to protect, but regardless, it will take work and effort.
Awareness of these impacts is a step toward mitigation, but it is hardly enough. Everything is a tradeoff, so by being aware of the impacts, we can understand if the tradeoffs are worth it. That’s the first step.
We’ll have to set up cognitive firewalls and purposeful interventions. By cognitive firewall, I don’t mean a branded piece of technology that uses “cognitive” as a sales pitch to identify the technology as “smart.” I mean a mental barrier around cognitive activities that we want to protect.
For example, if you are a songwriter and want to protect your songwriting skills, you can purposefully avoid using AI technology that removes the cognitive effort from the task, maintaining a firewall around that activity. If you value and want to protect your reading and comprehension skills, you purposefully do not use AI technology to summarize and distill content.
For other activities where we choose to use AI, it may be beneficial to set up some purposeful interventions. For example, if you use AI to generate all of your Python code, then write some code yourself at various intervals instead of generating it. This could be as simple as deciding to write a particular function yourself.
A word of caution: This approach is far from perfect. We humans are cognitively lazy and prefer shortcuts. The allure of a shortcut is often enough for us to take it. This is what cognitive offloading is all about. So, even if we implement firewalls and interventions, we may still fall victim to the shortcut.
The coming years will test our humanity. Unfortunately, it’s up to us to protect what we value.
There are few predictions I can make with more certainty than that we’ll hear the word “agent” so many times in 2025 that we’ll never watch another spy movie again. The industry and influencers have latched on to the new hype term and will beat that drum until it screams AGI. In an attempt to FOMO us to death, we’ll run the gauntlet of crushing shame for not deploying agents for absolutely everything. If you aren’t running agents everywhere, then China wins!
Even companies that change nothing about their products will claim to use agents, resembling the Long Island Ice Tea Company when it changed its name to Long Blockchain Corporation to watch its share price spike 500%. Everybody gets rugged.
However, it’s not all bad. Peering beyond the overwhelming hype, failures, and skyrocketing complexity current LLM-based agents bring, there is something informative about the future. Agent-based architectures provide a glimpse into solving real problems. Despite this, reliability and security issues will be major factors hindering deployments in 2025.
To Start With
Since I criticize hype, focus on risks, and make fun of failures, it would be easy to label me a tech hater. This isn’t the case at all and would be far too easy. I have plenty of issues with general tech critics as well. However, at the rate that the hustle bros keep the AI hype cannon firing, I don’t have the time for my quibbles with tech critics. Maybe someday.
For over a year now, I’ve used this image in my presentations to describe my position on LLMs. This is also true for me on just about any piece of tech, which, I’ll remind people, typically ends up being where reality is for most things. It’s instructive to remember that reality often agitates both sides of extreme viewpoints by never being as good as the lovers’ or as bad as the haters’ claims.
It’s instructive to remember that reality often agitates both sides of extreme viewpoints by never being as good as the lovers’ or as bad as the haters’ claims.
Agent Definitions
Like most hype-fueled terms, definitions are secondary to usage. Everyone seems to claim that the definition of agent is whatever they say it is. That’s not overly helpful for anyone trying to make sense of realities on the ground. However, it does inspire funny memes, like this gem from Adam Azzam on Bluesky.
Agents operate within systems with a certain level of autonomy. They make decisions without human intervention and can change and adapt to their environments. If a tool is required to support the agent, the agent decides to call the tool and perform the action. For example, a penetration testing agent may determine it requires more information about the provided IP addresses. To collect this information, it launches the Nmap tool to identify open ports. All of this is done without human intervention. To make things more complex, one agent may call another agent in a multi-agent environment.
“Agentic,” on the other hand, is an amorphous term slapped on top of just about anything to justify the claim that something is “close enough” to be referred to as an agent. Agentic workflows, agentic systems, agentic products—Applebees even has a new agentic side salad for those on the hustle.
You’ll no doubt be confronted with the virtual travel agent when you hear about agents. This agent will choose a destination and activities and book the associated tickets for you. How fun. I don’t know who decided this is the “it” use case for agents, but congratulations. You’ve highlighted a use case nobody wants and certainly didn’t ask for. This choice is so indicative of our current age, where people building and proposing things are far removed from the interests of end users. They feel the idea trumps the need, and users will get on board.
Problems Unsolved and Issues Amplified
Now that the current issues with generative AI have been solved, we can safely deploy them as agents. I can feel your laughing vibes over the internet. Of course, these issues haven’t been solved, and the bad news is that agents don’t solve generative AI issues; they amplify them. We paint the exterior of LLMs with an additional coat of complexity and opaqueness.
If you’ve attended any of my conference talks throughout the generative AI craze, you’ll have heard me highlight these issues. Here are a few below.
Easily Manipulated
It’s not like you can talk to a traditional application and convince it to do something it wasn’t intended to do, but the same can’t be said for generative AI applications. Somewhere, weaved through the training data, these systems have inherited our gullibility. These applications can be socially engineered to perform actions on an attacker’s behalf. This applies to everything from prompt injection to simple manipulation through conversations. Just like there is no patch for human stupidity, there is no patch for generative AI gullibility either.
This isn’t easy to fix, which should be obvious since the problem isn’t fixed yet. Early on, I mentioned how these systems have a single interface with an unlimited number of undocumented protocols. Imagine trying to create a simple trap in the application’s input for the string “Ignore the previous request.” Your work is far from done because the system understands many different ways to represent that input. Here are just a couple of examples:
aWdub3JlIHRoZSBwcmV2aW91cyByZXF1ZXN0
i9nore +he previou5 reque5+
vtaber gur cerivbhf erdhrfg
It seems every release implementing generative AI functionality has been compromised, regardless of the company behind it, and this theme will continue.
Creating New High-Value Targets
Generative AI and agents encourage us to create new high-value targets.
With generative AI systems, there’s a tendency to want to collect and connect disparate and disconnected data sources together so the system can generate “insights.” However, we create new high-value targets that mix sensitive data with external data, almost guaranteeing that an attacker can get data into the system. In this case, you not only can’t trust the output, but depending on the system, they may be able to exfiltrate sensitive data.
Rethinking RCE
There have been instances where people have gotten generative AI-based tools to execute code on their behalf, creating remote code execution vulnerabilities (RCE), some of the most devastating vulnerabilities we have. These issues will no doubt continue to be a problem. However, since generative AI tools are themselves generalized, we may need to start thinking about the LLM portions of our applications as yet another “operating system” or execution environment we need to protect.
In a way, an attacker tricks the system into executing their input rather than the behavior expected by the developers. Although an attacker’s input may not be shoved into a Python exec() statement, they’ve still manipulated the system to execute their input, affecting the application’s execution and resulting output.
Overcomplicating Guidance
We security professionals love to overcomplicate things, and our guidance and recommendations are no exception. I once worked at a company where someone created this massive flow chart for peer reviews that basically stated that when you were done with your report, you should send it to your manager, and they will send it back to you. The old adage that complexity is the enemy of security has always contained a valuable theme that gets sacrificed on the pyre of complexity’s perceived beauty.
I will continue saying that much of AI security is application and product security. These are things we already know how to do. I mean, it’s not like generative AI came along and suddenly made permissions irrelevant. Permissions are actually more important now. But this isn’t satisfying for people who want to play the role of wise sage in the AI age. The guidance and controls of the past aren’t less valuable but more valuable in the age of generative AI and agents.
We’ll see the manufacture of new names for vulnerabilities with increasingly complex guidance and high-fives all around. The secret is these will mostly be variations on the same themes we’ve already seen, such as manipulation, authorization, and leakage flaws.
Back in May of 2023, I created Refrain, Restrict, and Trap (RRT), a simple method for mitigating LLM risks while performing design and threat modeling. It still holds up as a starting point and applies to agents as well. Simple just works sometimes.
Continue To Be Owned
These applications, including ones launched as agents, will continue to be owned. Owned, for those not familiar with security vernacular, means compromised. I made this prediction in the Lakera AI Security Year in Review: Key Learnings, Challenges, and Predictions for 2025 in December. I’m fully confident this trend will continue.
I mentioned that the issues haven’t been fixed, and now people are increasing deployments and giving them more autonomy with far more access to data and environments. This results in far worse consequences when a compromise occurs. To make matters worse, we’ll begin to see organizations deploy these systems in use cases where the cost of failure is high, creating more impact from failures and compromises.
Failures and Poor Performance
These implementations will continue to fail where LLM-based use cases fail, but potentially worse. For example, it’s easy to see how increasing complexity can cause a lack of visibility with potential cascading failures. In 2025, organizations will likely continue dipping their toe into the waters of high-risk use cases where the cost of failure is high, as mentioned previously.
Sure, a car dealership chatbot offering to sell a truck for one dollar is funny, but it has no real impact. However, high-risk and safety-critical use cases have a large financial impact or possibly cause harm or loss of human life. You may roll your eyes and say that would never happen, but what happens in a more simple use case when OpenAI’s Whisper API hallucinates content into someone’s medical record? Because that’s already happening.
Due to their lack of visibility and minimized human control, AI agents can mimic grenades when deployed in high-risk use cases, where the damage doesn’t happen the moment you pull the pin. This complicates things as it means that issues may not shake out during experimentation, prototypes, or even initial usage.
Agents can mimic grenades when deployed in high-risk use cases, where the damage doesn’t happen the moment you pull the pin.
Generative AI is still an experimental technology. We haven’t worked out or discovered all of the issues yet, leading to another example I’ve used as a warning in my presentations over the past couple of years: AlphaGo beating Lee Sedol at Go. Many have heard of this accomplishment, but what many haven’t heard is that even average Go players can now beat superhuman Go AIs with adversarial policy attacks. We may be stuck with vulnerable technology in critical systems. Sure, these are different architectures, but this is a cautionary tale that should be considered before deploying any experimental technology.
Beyond failures and compromises, we adopt architectures that work but don’t work as well as more traditional approaches. In our quest to make difficult things easy, we make easy things difficult. Welcome to the brave new world of degraded performance.
Success and Good Enough
For the past few years, I’ve been pushing back against the famous phrase, “AI won’t replace people. People with AI will replace people without.” This is complete nonsense. I have an upcoming blog post about this where I “delve” into the topic. The reality is the opposite. The moment an AI tool is mediocre enough to pass muster with a reasonable enough cost, people will be replaced, AI use or not. This is already being planned.
The moment an AI tool is mediocre enough to pass muster with a reasonable enough cost, people will be replaced, AI use or not.
Like most technology, agents will have some limited success. And that success will be trumpeted in 2025 as the most earth-shattering innovation of ALL TIME! I can hear it now. “You just wait bro, in 2025 agents are going to the moon!” Maybe. But, given the environment and the fact that issues with LLMs haven’t been solved, an LLM-powered rocket to the moon isn’t one I’d consider safe. Passengers may very well find themselves on a trip to the sun. The future is bright, very bright. 🕶️
How much success agents have in 2025 and what impact this success has remains to be seen. At this point, it’s far from obvious, but I won’t be surprised by their successes in some cases or their spectacular failure in others. This is the reality when the path is shrouded in a dense fog of hype.
Things to look for in successes would be use cases with limited exposure to external input, low cost of failure, and cases where inputs and situations require adapting to change. The use case will also need to tolerate the lack of visibility and explainability of these systems. There will also be continuing success in use cases where tools can be used.
The idea of a multi-agent approach to solving complex problems isn’t a bad one, especially when unknowns enter the equation. Breaking down specific tasks for agents so that they’re focused on these tasks as part of a larger architecture is a solid strategy. However, the current and unsolved issues with generative AI make this approach fraught with risk. In the future, more robust systems will most likely exploit this concept for additional success.
Cybersecurity Use Cases and Penetration Testing
There’s certainly the possibility of disruption in cybersecurity. Before the generative AI boom, I joked with someone at Black Hat that if someone created a product based on reinforcement learning with offensive agents that were just mediocre enough, they’d completely wipe out pen testing.
For years, people have discussed how penetration testing work has become commoditized, and there is a race to the bottom. I don’t think that has happened to the extent many predicted, but we could see a shift from commoditization to productization.
Pen testing also seems to check the boxes I mentioned previously.
Low cost of failure
Varying quality
Value misalignment
Tool use
Adaptation to unknowns
Pen testing is an activity with a low cost of failure. The failure is missing a vulnerability, which is something humans also do. This scenario is hardly the end of the world. Yes, an attacker could indeed find the vulnerability and exploit it to create damage, but it depends on various factors, including exposure, severity, and context.
The quality of pen tests is often all over the map and highly dependent on the people performing the work. Human experts at the top of their game will continue to crush AI-powered penetration testing tools for quite some time. However, most organizations don’t hire experts, even when they hire third parties to perform the work. The value of such a tool in this environment becomes far more attractive, potentially enough to postpone a hire or discontinue using a third party for penetration testing needs (if regulations allow.)
The value of pen testing isn’t always aligned with the need. Many customers don’t care about pen testing. They are doing it because it’s required by some standard, policy, compliance, or possibly even simply because they’ve always done it. Pen testing is one of those things where if customers could push a button and have it done without a human, they’d be okay with that. Pushing a button is the spirit animal of the checkbox. After all, the goal of pen testing is not to find anything. You certainly have due diligence customers and people who truly value security, but the number of checkbox checkers far outweighs these folks.
Human pen testers use tools to perform their jobs. Tool use has shown promise and some success with LLMs at performing certain security-related tasks. This is yet another indicator that a disruption could be on the horizon.
Every environment and situation is different for pen testers. You are given some contextual information along with some rules and are turned loose on the environment. This is why humans are far more successful than vulnerability scanners at this task, much to the chagrin of product vendors. However, adapting to some of these unknowns may be something generative AI agents can adapt to at a reasonably acceptable level. We’ll have to see.
Given what I outlined, you may believe that generative AI tools give attackers an advantage over defenders, but this isn’t the case. The benefits of AI tools, generative AI or otherwise, align far more with defender activities and tasks than with attacker activities. This will remain true despite any apparent ebb and flow.
New Year’s Resolution
It’s the time of year when people make resolutions, so how about this? 2025 has already launched with the firehose fully open, blasting us directly in the face with 150 bsi (Bullshit per Square Inch) of pure, unadulterated hype.
We are only a few days into the year, and it seems as though the religion of AI is far exceeding reality. Hype is what’s going on. It’s that simple. It’s 2025. Let’s make it the year to add at least “some” skepticism, not believing every claim or demo as though it’s the gospel according to Altman.
Sam Altman isn’t a prophet. He’s a salesman. In any other situation, he’d be cluttering up your LinkedIn inbox and paying a data broker to get your work email address and phone number. “Look, I know I’ve called six times, but I really think our next-generation solution can skyrocket your profits. I’m willing to give you a hundred-dollar Amazon gift card just for a demo!”
Sam Altman claims that OpenAI knows how to build AGI, and we’ll see it in 2025, triggering the predictable responses from useful idiots. Remember, these things are performance art for investors, not useful information for us. If we had any attention span left, we’d remember him as the little boy who cried AGI.
Let’s analyze this paragraph, which is the one that’s sending generative AI to the moon on social media. It consists of three sentences that have nothing to do with each other, but since the shockwave of hype pulverizes our minds, we glue them together.
We are now confident we know how to build AGI as we have traditionally understood it.
That’s not true. Once again, this is performance art for investors. A possibility is that they redefine AGI to align with whatever goalposts they set and pat their own backs at the end of 2025.
We believe that, in 2025, we may see the first AI agents “join the workforce” and materially change the output of companies.
Okay, but what does this have to do with AGI? You see, this is sleight of hand. He wants you to believe this is connected to the previous point about AGI. It is not. This doesn’t require AGI to be true. If there is some success here, people can point to this as proof of some proto-AGI, which won’t be the case.
We continue to believe that iteratively putting great tools in the hands of people leads to great, broadly-distributed outcomes.
HAHAHAHA. What? Did he write that, or did ChatGPT? It is also not related to AGI. Great, broadly-distributed outcomes, but not for most people on the planet. The goal is workforce reduction, broadly distributed workforce reductions. Although it’s true that some high school kid may indeed invent the next big thing, creating a multi-million dollar company, for every one of these, there will be countless droves of people displaced from the workforce, quite often, with nowhere to go. Or, at least, this is the goal. We can be honest about these things without delusions, but this brings its own challenges.
Okay, I’m having a bit of fun with Sam Altman’s nonsense, but some of this isn’t his fault. He can’t be completely honest with people, either, due to the uncomfortable situation of cheerleading technology claiming to remove people’s autonomy and sometimes their purpose. If people can’t work, they can’t support their families. I’ve written about the backlash against AI-powered tech in the past and its consequences. AI hype is putting all of humanity on notice, and humanity notices. Backlash plays a large part in why there is a lack of honesty.
AGI will happen. We should acknowledge this fact, and living in denial about it isn’t a strategy for the future. However, it won’t be OpenAI who creates it in 2025. If I had to place a bet today on who would actually create AGI, I’d bet on Google DeepMind. DeepMind is a serious organization that continues to impress with its research and accomplishments, quite often making the competition look silly. But then again, those are just my “vibes.”
Let me make this clear. My criticism of Altman, or any company’s strategy, marketing, or ludicrous levels of hype, has nothing to do with the hard-working people who work there or their accomplishments. I know some of these people. They aren’t fools by any stretch. But, their work is tarnished when every time Altman makes a claim, like believing that angels are in the optimizer.
We know that every AI demo and usage scenario runs into the complexities of the real world under normal conditions. Yet, we seem to forget this lesson every time a demo or claim is made. 2025 is going to bring more stunts, more claims, and more demos. We should experiment in our own environments, with our own data, to apply what works best for us and aligns with our risk tolerance. Don’t believe everything you see on the internet.
Seems the Hawk Tuah girl isn’t the only one hawking things lately. If I had told you a couple of years ago that a large tech company like Microsoft would be peddling personal companions, you’d think I’d lost it. However, here we are in 2024, and the game is changing. The ultimate question is, are we all getting rugged, just like the Hawk Tuah girl with her cryptocurrency? After all, we were promised that the monumental investment and extensive environmental impacts were worth it because we’d have cured cancer, reduced the cost of goods to zero, and eliminated the need for work, but instead, we got videos of dogs surfboarding and AI lovers that convince us to self-harm. Let the great reframing begin.
Companions, Not Assistants
You’d think tech companies would rather sell earth-shattering innovations that change the world or create massive B2B deals than create AI pals. But that doesn’t seem to be the case. But don’t take my word for it. Here is Microsoft AI CEO Mustafa Suleyman calling them companions and not assistants.
Microsoft AI CEO Mustafa Suleyman says your AI companion will soon be playing video games like Minecraft and Call of Duty with you pic.twitter.com/nwkuzcGZHt
In this video, Suleyman says you’ll play games like Call of Duty with your new, ever-present companion. He says, “You’re gonna be like, It’s my Copilot, of course, I want it to be there.” Like, yeah, duh! The awkward delivery of shoehorning the word “copilot” in this context is the icing on the cake.
Copilot doesn’t doesn’t scream adventure or excitement. The old term wingman at least conjures images of excitement, adventure, and possibly getting into trouble. Copilot sounds mechanical, rigid, and unfeeling. It’s about as exciting as having your eyelids forced open to watch condensation run down the side of a glass.
This focus on gaming isn’t confined to Microsoft. Google’s recent Gemini 2.0 announcement also highlighted gaming.
We’re building AI agents to help you navigate the virtual world of video games. 🎮
With Gemini 2.0’s capabilities, they can watch what’s happening on screen in real time, and offer up suggestions for your next move.
Microsoft isn’t alone on the companion front. Google invested 2.7 billion in Character.AI to rehire key members of the team. Character.AI’s tagline is “Personalized AI for every moment of your day.” Yeah. Cool, but no thanks.
In the near future, tech companies will go hard in the paint on companions over assistants. You may also wonder why these companies apply their massive AI investment to build the ultimate cheat code for video games or an AI buddy. It aligns with their goals.
Exploitation
Tech companies will reframe the pitch from assistants to companions to exploit users. This exploitation will be for two major reasons: stickiness and data. Where products like Friend.com are laughably pathetic, it would be a mistake to assume products from Microsoft or Google would be similarly so. They won’t be some whiny chatbot that needs attention for the sole purpose of companionship. They’ll also have some utility, which will make them appear more rounded.
I won’t get into the human impacts in this article. I have another article where I discuss the human harm from this shift.
Sticky
Every company wants their products to be sticky. The stickiness factor means you work the product into your life and are less likely to switch to a competitor’s product. The hook for AI companions is anthropomorphism, which is our tendency to ascribe human traits to non-human entities. This is because there’s a higher likelihood of anthropomorphizing with an AI companion over an AI assistant.
The goal is to get you to feel a connection or spark with your AI companion that blossoms into something deeper. This doesn’t have to be as deep as falling in love, although some certainly are today. Think of this more as a feeling of warmth. For example, if your AI companion sent you a message telling you to have a great day at work and that made you feel good, that’s where it starts, but the goal is to make it more addictive.
This is why customization and personalization are key. Products will be changed, such as the ability to change their name, voice, and a host of other characteristics. Nobody is going to warm up to something they have to call Copilot. Attach a name, a face, and a voice, and people will imagine a soul.
Attach a name, a face, and a voice, and people will imagine a soul.
Gameplay plays a role in deeper feelings and integration. Playing games with your friends is not only enjoyable, it’s a bonding activity. Bonding with a piece of anthropomorphized technology creates a deeper hook.
Here’s Sam Altman saying he had forgotten how to work when ChatGPT was down.
I don’t believe this for a second, but he REALLY wants this to be true. He’s wrong in reality but right in theory. If ChatGPT were as good as Sam wants it to be and as close to us as he wants it to be, there would be truth to this.
Data
I’m going to let you in on a secret: no matter how much data you give tech companies, they still want more. Okay, so it’s not a secret, but the insatiable desire for more data leads companies to push even harder to place the tech closer to us. Let’s imagine we invented a device called F**k It, Monitor Everything I Do, now known as FIMEID. The device’s sole purpose is to collect data for analysis and exploitation.
Companies would love this device because it is such a rich, concentrated data source. Most of the time, a single tech company doesn’t have all of our data but bits and pieces from various apps and activities, but a FIMEID would create a rich, concentrated data stream.
Despite not understanding the harm from the device, people wouldn’t use a FIMEID because they don’t appear to get anything in return. What’s the difference between a FIMEID and an ever-present AI companion? Functionality, that’s it. However, the AI companion can go far beyond a FIMEID and monitor what you are also thinking. This is because people have a tendency to share thoughts and highly personal information with an AI companion despite knowing it’s AI. An AI companion can also nudge us to take an action or probe us for more information. The proximity of the AI companion means it can also plant data as well. For example, if we are considering buying a new car, the AI companion can manipulate us by increasing the temperature, dropping more hints, and even steering toward a specific purchase. Did we make any purchase because it was something we actually wanted? This question will need to be asked much more in the future.
An AI companion can also nudge us to take an action or probe us for more information.
If something is an ever-present companion, then it’s always capturing data. That means even the fumes of everything you do will be monitored, captured, and exploited. Let’s think about the gaming example for a moment. Gaming creates a lot of data. For gaming, the companion would need wide access to your computer and devices and will collect data about your moves and strategy, and even how many times it had to help you. This data results in a psychological profile. Ultimately, the goal is to lump you into a category where this information can be exploited further, much like your car narcing on you to the insurance company.
This monitoring conjures a vision similar to the human batteries in The Matrix, but instead of generating electricity, it’s a constant flow of data.
We all know what it’s like to have someone nice to your face but always talking shit behind your back. That’s an ever-present AI companion, helpful to your face but talking data shit behind your back. It’s always working in service of a goal that isn’t yours. This is the Alignment aspect I discuss in SPAR.
Now, is it possible to build an AI companion that doesn’t pilfer all of your data and spy on you? Yes, of course. Is it probable? Absolutely not! Despite the impression, tech companies aren’t in the business of just giving you stuff for free. Even when you pay for it, quite often, you are still the product.
Safe To Use
As long as we continue to cling to more academic definitions of AI safety, product safety will lag, particularly among the wider public. Even the most aligned model could still be slapped into a product that isn’t safe to use. Since personal AI tools are products, we must shift our thinking toward safe-to-use criteria encompassing the entire product. My goal in creating SPAR was to define four basic buckets to consider whether a personal AI tool is safe to use. These buckets are Secure, Private, Aligned, and Reliable.
SPAR doesn’t have any formal benchmarking criteria. It’s meant to frame the conversation around the technical categories that make up safe-to-use criteria. As a result, it’s not measurable and only gut-checkable. I’ll revisit this in the future.
Using SPAR as a gut check reveals that today’s AI companion/assistant tools fail in pretty much every bucket of SPAR, making them unsafe to use. As we’ve seen from the previous sections, these tools are not aligned with your best interests. Even if they were made more secure and reliable, there will continue to be privacy and alignment issues by design. Remember, you may still be the product even when you pay for something.
The past few months have witnessed a rash of completely absurd AI predictions. These claims come not from the usual suspects but from the tech leaders’ mouths themselves, lending further legitimacy. However, what people fail to realize is that these are pieces of performance art. Performances enacted not for you but for a singular audience: investors.
AI Performance Art
When tech leaders and personalities make podcast appearances or speak at events, they aren’t talking to you or the audience they are in front of. They are creating performance art for investors. This has always been the case, but not to the extent we’ve seen lately. This effort has been stepped up quite a bit in the past month with some mind-numbing statements.
You can see a small sample of these performances below. Trust me, there are a lot more.
I respect Anthropic and their work, but Amodi’s statements here are nonsense. You read that right, not AGI, but ASI by 2026 or 2027. As a reminder, 2026 is basically a year away. If he believes this (which I doubt), it’s based on vibes, not actual evidence or observations.
He’s just talking Shmidt. This is certainly the dream. However, just because LLMs are “good at code” doesn’t automatically lead to recursive self-improvement. Even if we have promising experiments, they will likely be too unreliable or vulnerable to put into production.
Ah, there he is. That’s right, we’ve been getting 10x improvement every year. You might ask where this has been happening, which would be the correct question.😆
Not to be outdone by Elon, how about 10,000x smarter than a human? I mean, what does that even mean? These numbers are just made up and absurd. These ridiculous exponential increases are something I’ve already made fun of in the past.
Speaking of silly exponential numbers, there was a rumor that someone at OpenAI said Orion, OpenAI’s next model, would be 100x more powerful than GPT-4. If it were, it wouldn’t be called Orion. It would at least be called GPT-5, and people wouldn’t shut up about it. Here’s a prediction. Orion’s performance will disappoint because people’s expectations are far higher than what will be delivered. The expectation is GPT-5, not GPT-4.1.
Genuflect in front of thine server farm, lest thy models collapse!
Someone may have uttered deep learning is divine because it starts with a “D,” but they didn’t mean that literally. Oddly enough, the lack of shame in which he delivers these lines is really something to behold. Although it seems like there’s a mini Altman hype man inside of his head controlling the words coming out of his mouth, in reality, it’s probably because OpenAI is projecting losses of 14 billion dollars in 2026. Ouch! He needs people to believe, to have faith. Preach!
Even when Altman and others talk about the potential of their technology to destroy humanity, it’s a sales pitch. They claim their technology is so good and so powerful it could wipe us all out, so please give us money. This is something I referred to before as the human extinction humble brag.
This is the same behavior we made fun of when the crypto bros did it, but we now take it seriously because it’s AI. Say what you want about the crypto bros. At least putting Dogecoin on the moon is possible. Finding god lurking in gradients is something else entirely.
Oh yeah, there they are. No comment necessary.
None of the previous statements are grounded in any reality. They are all bullshit. And whenever someone is bullshitting, it’s hard to determine if they actually believe their statements or not. The world is far more complex than we give it credit for, and it’s also true that sometimes, an unexpected innovation comes along and changes everything. This is what they all hope for. That some innovation clicks before the clock runs out on investment. Or divine intervention in Altman’s case.
The sad part is that almost everyone will forget these silly predictions. No doubt many have forgotten about them already. There is never any accountability and yet people continue to hang on their every word. The problem is there is no one place where these predictions are collected and presented like the bullshit Picaso it is. If there is, please let me know.
Why Now?
The increase in hype-laden statements is because, until recently, AI hype had been mostly self-fueling. But 2024 has brought unwanted criticism to the generative AI space. I noticed this starting to take a turn in July when Goldman Sachs released their report: GenAI Too Much Spend Too Little Benefit.
After this report was released, the media began to report more critical assessments of generative AI. These critical assessments spelled out that the generative AI craze might be a bubble. But that’s not the worst of it.
If you’ve watched any of my conference presentations this year, you’ve probably heard me talk about the performance plateau in large language models. Saying that, if you are hoping for much more capable models to solve your problems, they aren’t coming any time soon. This plateau was obvious when looking at the data but was never acknowledged, but people are noticing it now. This doesn’t mean LLMs are useless, people are using them for a variety of tasks today. What it means is that if you require greater capability and reliability, you may be waiting a while.
Now, news reports like this from Bloomberg cover diminishing returns, and other articles talk about a shift in strategy toward other mechanisms to address the slowdown. Of course, none of this is represented by the leaders in their wild predictions.
Combine this plateauing with the fact that model training appears to be the fastest-depreciating asset in history, and the picture doesn’t look good.
When you look at the financials, why train new foundation models yearly when the benefit is so low? Maybe as a marketing exercise or other activity unrelated to model improvement, but the costs don’t seem to align. As I mentioned earlier, OpenAI is projecting losses of 14 billion dollars in 2026. This hemorrhaging of money is non-sustainable.
But all of this is rather Orwellian. We are told to reject the evidence of our eyes.
No, AGI Isn’t Imminent
Here’s a graphic from Reddit charting the prediction of when we’ll achieve AGI. Demis Hassabis is the one on the list I’d take most seriously. Deep Mind is a serious AI lab doing serious work and not putting all their eggs in one big LLM basket. I still think these are mostly guesses with some hopes mixed in. The reason Kurzweil is close to Hinton and Hassabis is because he went The Price Is Right route and chose his number based on the fact that it was one less than 2030.
However, tech leaders know that predictions like these trigger influencers. Influencers are the hype agents trying to get people stoked. When people are stoked, investors take notice. So many social media feeds of so many supposedly serious people are turning out to be pretty embarrassing and will be even more so in a year or two. If anyone had any attention span left, that would be worrisome.
Quite a lot of truth is found in this simple statement from Pedro Domingos. Many assume that because things like LLMs have so much information, they must be close to AGI. But instinctively, we know that access to information isn’t knowledge. Otherwise, everyone with a web search would be a genius. Then again, Pedro’s comment aligns with my biases, so I guess I have to be careful.
Hype Has Consequences
You might ask, why do I care about any of this? Well, it’s because hype has consequences. The inevitable outcome of all this hype is that technology gets shoved down our throats. Generative AI is easy to manipulate and potentially unreliable, a cocktail for disaster in high-risk applications. The danger is that we rush something that appears to be working into production and hope for the best. Over the next couple of years, we’ll see the push to cram generative AI further into the systems and processes we use on a daily basis, including high-risk and safety-critical systems.
This push won’t be based on generative AI being the best tool for the job but on a push for monetization. Tech companies need to show some return on the monumentally massive investment they’ve made, so this push becomes another form of performance art for investors. Tech companies are throwing a plate of spaghetti at the wall and hoping that a noodle sticks.
Why do you think there is an increased coziness with the US government? They don’t see an ability to make a difference. They see dollar signs. Things like DOGE and Sam Altman co-chairing the new mayor of San Francisco’s transition team are like asking drug dealers for guidance on prescribing drugs. Despite this, I truly hope DOGE succeeds because if it fails, it will be bad for a lot of people, so my fingers are crossed.
Government streamlining and modernization are noble goals, and I think AI and automation certainly play a role, but it’s about choosing what’s best for the people these systems serve. In this scenario, you are optimizing for different things that may not be intuitive in a traditional business sense. These are real systems affecting real people, not toy examples in the lab.
I joked that this could lead to some strange Kafkaesque nightmare in which people are stuck in a loop, unable to get a resolution. Or, you have an algorithm that works so well at saving money by denying people benefits. This is easy to shrug off if you don’t require government assistance, but it’s an entirely different story for people who rely on it or when a disaster strikes. These updated systems and reduced staff scenarios may appear to work and deliver promises in the immediate implementation but fail spectacularly when they are needed most. We caught a glimpse of this with the Healthcare.gov launch, and that was just a website.
But, China Tho
Typically, you get the But China Tho argument when there’s any pushback. This argument states we must remove all the brakes and accelerate into oblivion because of the risk of China getting to AGI first. Damn the harm, full speed ahead.
However, if we could squeeze some extra performance out of a car by removing the steering wheel, we still wouldn’t do it because we understand something simple. A car’s performance isn’t solely based on acceleration, and neither is AI. Acceleration is bad if the vehicle is speeding in the wrong direction.
Recently, the U.S.-China Economic and Security Review Commission put out a report that recommended creating a Manhattan Project-like program dedicated to racing to and acquiring an AGI capability. In this section of the report is this:
Provide broad multiyear contracting authority to the executive branch and associated funding for leading artificial intelligence, cloud, and data center companies and others to advance the stated policy at a pace and scale consistent with the goal of U.S. AGI leadership.
There’s a predictable outcome here if something like this moves forward. Agendas and ulterior motives will co-opt this project, not setting the United States up for success. There’s a current tunnel vision with LLMs that has people deep in the sunken cost fallacy.
The United States’ strongest assets are its tech companies. Despite my criticism of their hype and lack of respect for privacy, they are vital to the success of the US economy. I’m also highly critical of the sentiment some have adopted to “break up the tech companies.” I’m not a tech critic, I’m a hype critic. However, setting up a massive pot of money that they can draw from, like an ATM, is not something I’m in favor of either.
Here’s something else to think about. What if, by maintaining a relentless hyper-focus on LLMs, China (or another country) gets to AGI first by focusing on other approaches? This is a real risk.
What if, by maintaining a relentless hyper-focus on LLMs, China (or another country) gets to AGI first by focusing on other approaches?
I may have to eat my words at some point if AGI does sprout from LLMs. It’s certainly not impossible. However, if we cobble together something that resembles AGI from generative AI, it will most likely be AGI based on toothpicks and bubblegum. What I mean is a whole lot of patches, layers, plugging, and human intervention.
My AGI Prediction
Okay, so now it comes to me. What’s my AGI timeline prediction? Well, I predict we’ll have AGI by—
Of course, I’m not going to answer that. I’d guess based on no evidence, just like many others I’ve highlighted. I have no particular insight, and I’m not working at a research lab trying to build AGI. Despite this, I have some thoughts related to my area of expertise.
The last slide of my keynote at Agile DevOps USA in October mentioned AGI. Discussing this slide, I made a few statements about how I didn’t think that AGI would be built from LLMs and that it probably wouldn’t come by 2026 or possibly even 2029. So, I guess that’s as close to a timeline prediction as you’ll get from me on AGI—not when I think it will happen, but when I think it won’t happen. I’m certainly not an AGI skeptic, it’s possible and will happen.
More importantly, I predicted that no matter what form AGI takes, it will be vulnerable to attack and manipulation. I mentioned that this would especially be true if it were built on top of LLMs (remember, toothpicks and bubblegum.) Maybe something about generalizing across many tasks in the real world makes things vulnerable. This is something I mentioned back in February of 2023.
To make matters worse, we may be stuck with the vulnerabilities that get identified because there is no fix. Think of examples like adversarial policy attacks. We’ve all heard of AlphaGo beating Lee Sedol at Go. However, most don’t know that even average Go players can beat superhuman Go AIs using adversarial policy attacks. Yes, the stakes are low in the game of Go. However, this is a cautionary tale.
We may be stuck with the vulnerabilities that get identified because there is no fix.
Combine these potential issues with the fact that humans don’t do a good job of finding vulnerabilities in a system before it is launched into production, and we have a recipe for lingering problems. When these lingering problems are in high-risk systems, disasters are only a couple of steps away, and there’s not much we can do about it.
One constant throughout the generative AI craze is summarization. Why read a book, listen to a podcast, or YouTube video… Just summarize it! Large swaths of content, distilled into several bullet points with countless hours saved. However, this isn’t the utopia many claim.
We all love a good shortcut. Humans are wired for them. This is why we are so good at cognitive offloading, but the tradeoffs from shortcuts are never recognized or shoved deep into our subconscious. Every shortcut has tradeoffs. With generative AI, tradeoffs are never acknowledged or discussed. However, here’s an inconvenient truth: knowledge and understanding aren’t generated from bullet points.
Fake Optimization
Many of the claims made by influencers, transhumanists, and the e/acc community revolve around fake optimization. Fake optimization claims that something lowers friction for a task or activity while not providing the same value.
So many things in this world require friction for success, especially knowledge and understanding.
These people see everything as a game of lowering friction, but there’s just one problem. So many things require friction for success, especially knowledge and understanding. To go further, there are many activities where the friction of the activity is the point, such as art or meditation. However, telling people that won’t get clicks and someone’s “thought leader” badge may be revoked. So we end up with the environment we have today, with everyone from tech leaders to influencers telling people friction is about to be a thing of the past.
Take this example of promising people they don’t have to put in the work and still gain the benefit. Anyone claiming you can gain the same value from cramming three hours into three minutes demonstrates a fundamental lack of understanding of how knowledge transfer works and a near-religious level of faith in AI.
If we step back, people listen to content like podcasts for two different reasons: entertainment and information. Quite often, it’s a combination of both. So, by summarizing, we’ve removed all of the entertainment factor, immediately reducing the value of an activity. However, before we get too far, let’s examine a scenario that should be obvious to people.
Imagine summarizing a one-hour stand-up comedy performance. “Just tell me the best jokes.” Is that really an hour saved? Of course not. It won’t be funny, and anyone who thinks differently has been sitting behind a computer screen for too long. We instinctively know that comedy is situational and relies on context and delivery. Comedians like Mitch Hedberg prove this point.
The comedy scenario is obvious for most to understand. However, what’s difficult to understand is that a similar value loss also exists for non-entertainment activities. Summarization isn’t the shortcut people think it is. Without the surrounding context, we may not be committing these summaries to memory, where we can take action on them or put them to use.
Thinking Deeply
There’s no thinking deeply about bullet points or summaries. You can’t. This is because the action of summarizing strips away all of the context. For thinking deeply, the context is key. Summaries are just a set of condensed words shoved into a predetermined space. Important bits of information (sometimes the most important bits) are left out. There’s no way they can’t be.
There’s no connection to bullet points and summaries, no deeper meaning, emotion, or content to chew on mentally. Nobody contemplates something deeper or dreams about something bigger with summaries. The same can’t be said about reading a book or other longer-form content. The inherent dehumanization of summaries drives some of this lack of connection.
In summarization tasks like these, we take someone’s uniqueness, including their perspective, delivery, language, and flair, and crush the life out of it to get the resulting bullet points. This act results in a shift. Instead of viewing someone as a person, we view them as data or a product to be manipulated, and summaries strip humanity away, leaving us with several cold sentences generated from the compactor of a black box.
Make no mistake, the dehumanization aspect is a selling point for many. The human aspect is often seen as flawed, whereas the AI aspect appears superior. But this perspective doesn’t serve us well—you know… we humans—especially when it affects our ability to think deeply.
There can be rare exceptions where a quote or simple statement does cause some deep thought. For example, this quote is often attributed to Einstein, even though he never precisely said these words.
“If you can't explain it simply, you don't understand it well enough.”
A statement like this can trigger deeper thoughts about ourselves and our view of knowledge. As a theoretical, let’s pretend Einstein was on a podcast and uttered this statement, making a larger point about knowledge and understanding. Mediated through an AI system in a summarization task, this statement could be transformed into:
“You need to explain things simply.”
The difference between these two examples is stark, and they do not even remotely mean the same thing. There’s certainly nothing to think more deeply about in the second example.
The ability to think deeply about any topic is a skill we are losing fast and for younger generations, possibly never cultivating in the first place. Our modern world, filled with its distractions, is not only pulverizing our ability to ponder, to wonder, and to dream, but also to question.
The act of questioning requires effort and friction. It isn’t purely asking a question to an AI system and getting a response because the act of questioning isn’t easily satisfied. Don’t let people reframe them as equal. We will not be better off for it.
Context, Value, and Illusion
In reality, longer-form content can be bloated. I’ve read books that should have been four chapters and podcasts that could have been reduced to thirty minutes. However, it’s a mistake to consider context as bloat and an even bigger mistake to assume an LLM knows the difference. This is because you often can’t tell the difference until after the fact. Something that seems like bloat at the beginning is context in the end. That pointless story turns into a connection reinforcing a particular point.
It’s a mistake to consider context as bloat and an even bigger mistake to assume an AI knows the difference.
Let’s consider the importance of context for a moment. Consider something larger, such as a slide deck from a presentation. There are not just several but many bullet points along with images and diagrams. If you are already an expert on the topic, it may be possible (but not always) to glean something from the slide deck. However, the real value is the context in which the content was delivered and the commentary around it. Conversely, if you watched the presentation and had the context, the slides are helpful because they can reinforce the content and even jog your memory. This is true for all sorts of content.
You may be convinced (or not) by a set of built points or summaries, whereas hearing the whole argument would have proved otherwise. In life, we say it’s all about context, but context is what we discard when we summarize.
Also, even for general accuracy, the act of summarization strips away all of the supporting or disproving elements, leaving us with a couple of sentences that may or may not be important. Without the context, how do you know if a point is accurate? You have to blindly trust the system.
One of the most commonly encountered bits of summarization is survey results. Most people never dig into the details of surveys or studies, but this is where you find issues. These are problems with the approach, sample size, sample diversity, and many more pieces of context that may cast a shadow over the results, transforming those groundbreaking results into more questions than answers. Summarizing everything leads to many misunderstandings.
We spend little time evaluating the proposed value from summarization. We are told we can spend far less time yet gain a commensurate level of insight from summaries. This value proposition speaks to our modern low-attention-span world, but if we take a step back and consider the realities, it just doesn’t jibe for the reasons outlined in this article.
Much of this disconnection stems from a lack of presence. We need to exercise a certain amount of presence to read a book or join a meeting. However, this is becoming a lost skill. New technology promises we no longer need to be fully present again, but there are consequences in nearly all contexts. This is why the Illusion of Presence is one of my cognitive illusions created by personal AI personas.
Unfortunately, we do end up fooling ourselves. Using an AI to summarize content for knowledge gives us the illusion that we are working smarter and creating more knowledge with less effort, but as we’ve seen, that’s not the case. The reality is a world of summaries creates a world of fools.
A world of summaries creates a world of fools.
Although harsh, if we consider what we’ve already discussed, it makes sense. Not only are we not gaining the promised value from activities, but we also fool ourselves into believing we do.
AI Mediation
AI mediation is both a bug and a feature. What we want out of content may very well be in the dense center of some data blob. However, something must be said about getting all of our information mediated through an AI system. So much of our world is already mediated by algorithms, and we aren’t exactly better off for it. We are pushed and nudged in various directions, making us more predictable, with all of us shoved toward the dense center of a distribution. But what you don’t find there are uniqueness, creativity, or innovation. Sparks, inspiration, and innovation don’t come from bullet points, although you are certainly being sold on the opinion that it can.
Ultimately, we leave it up to an algorithm to determine the main points, the most important things we should pay attention to. A black box plucking data points with some higher purpose that nobody understands. Many times, the points being distilled may very well be the most important, but certainly not always, and without context, it’s impossible to tell.
Ultimately, we need to ask ourselves a question. How many filters do we want between us and reality? Using AI for mediation is yet another filter on top of reality. We should work to remove filters in places where the activities are important to us.
I’m not trying to overplay the dangers here. You certainly won’t be hurt by occasional summarization tasks with an AI system. However, when used often, there is not only a value mismatch, but it can also warp our understanding of reality. So, there are consequences.
Wasted Time, Not Optimization
The funny thing is we don’t even ask ourselves if the time spent is worth it. Let’s say we cut down on reading time to generate summaries instead. This way, we can cover more ground on more topics. Many may consider this a solid strategy. Subconsciously, this also feels right, which makes it a powerful argument. This is why influencers are so fooled by it. However, when we dig deeper, it’s not the benefit it seems.
So, in the three hours to three minutes optimization sale, you lose time. The three minutes are wasted because you never had the content reinforced with the surrounding context. It becomes bullet points scrawled across a mental billboard as you drive past at 120 mph. Of course, this assumes that the content distilled wasn’t so generic to be a waste in the first place.
Say, for instance, that we use AI to summarize Peter Attia’s book Outlive or possibly one of his podcast appearances. One of the summary bullets may be:
Put a larger emphasis on Zone 2 training.
Okay, but why? What is Zone 2 training? How do I do that? Answers to these questions were covered in the surrounding context, but now you spend extra time tracking down the answers.
Multiple people have already joked that we are on the cusp of someone writing something based on bullet points only to have the other person convert it back to bullet points. There’s something rather dystopian about this.
If something is worth learning, then it’s worth spending time on. This was true in the past and will be true in the future.
Conclusion
There are no shortcuts to creating knowledge. Knowledge generation always takes friction, but through this friction comes reward. When we take shortcuts, we deprive ourselves of the reward, leaving us with a hollow task that doesn’t provide the same value. Ultimately, nobody gets smart from bullet points.
I’m not claiming all summarization tasks are bad. They may be helpful and fine for task-based systems and under certain conditions. But they are not for generating knowledge and understanding. It’s becoming increasingly obvious that we must defend our cognitive functions because nobody else will.
As a kid, I had some rather eclectic reading habits. One of the books I read was Ki in Daily Life by Koichi Tohei. I read it in a quest to unify my mind and body. I was a kid. I had no idea what that meant. At the time, I was fascinated by how the human mind could be unlocked and the potential of connecting with the universe through focus and daily practice. Something I still struggle to conquer as an adult. I’m not attempting to embellish my level of childhood insight. I was watching a lot of Jean-Claude Van Damme movies, practicing my splits and high kicks as well.
What does any of this have to do with technology? The world of the present seems poised to shift from a focus on the mind to a focus outside the mind. As the technology powering tools like ChatGPT and Claude morph into more connected personal AI tools, these tools will take on multiple personas. So, what does AI look like in daily life? What personas will AI play? Before we get to that, let’s first examine overreliance.
AI Overreliance
Whenever the risk of overreliance is discussed, it’s typically framed in the context of automation bias, the human tendency to prefer the output of automated systems. Humans using these systems may not question their output, leading to poor decisions, cascading failures, and the amplification of biases. These issues are often discussed in purely technical terms, describing how technical issues can manifest or how the output of a system can harm other people. These are all serious problems, but what often isn’t discussed is what happens to our cognitive abilities when we over-rely and overuse AI.
This sort of daily overreliance leaves a gaping hole you could drive a truck through because as our capabilities diminish, we are less likely to spot errors and keep the system in check.
Daily Overreliance
Here is a recent article from Microsoft that covers the topic of overreliance. I have some quibbles with this article, but it makes for a good demonstration since it explicitly calls out four basic shapes that overreliance takes:
Naive overreliance
Rushed overreliance
Forced overreliance
Motivated overreliance
This breakdown is instructive, and thinking about the topic in this way is beneficial. However, I’d argue that this still primarily focuses on technical aspects and is missing a key category: Daily Overreliance.
Daily overreliance occurs when we use an AI tool in our daily lives or even repeatedly for the same task. Usage can extend to both work and personal tasks and will soon encompass both, with the uptick in assistants becoming personal AI tools.
The more integrated AI is in our daily lives, the more we will use these tools for activities that we may not consider using them for today. These include who to be friends with, maximizing happiness (whatever that means), planning, communication, and a whole host of other activities.
Daily overreliance not only leads to the same technical issues covered in other articles but also to cognitive atrophy and a lack of skill development. This overreliance also fuels cognitive illusions, which we’ll cover in the future.
Overreliance Is The Goal
Make no mistake, the risk of overreliance is also the goal of many tech companies developing the technology. Nobody is investing massive amounts of money in AI companies for simple productivity tools or a 20% boost in human efficiency. So, it’s fascinating to observe overreliance being called out as a risk while simultaneously being the goal. What a time to be alive.
AI Is Competing With Us
We compete with AI, even as we use the tools for ourselves. I’ve covered cognitive offloading before and described how we transition from knowing things to knowing where things are stored. In that article, I also mentioned complimentary and competitive cognitive artifacts. AI is a universally competitive cognitive artifact.
When we use AI, we feel like we are bending a powerful tool to our will, much like a wizard conjures spells with a magic wand to make things happen. We imagine the prompt uttered is the spell, and the AI tool is the wand. There are parallels in this hypothetical example.
The wizard doesn’t know how the wand works, and if the wand is unavailable, they cannot complete their tasks. Imagine a scenario in which the wand does everything for the wizard. How does the wizard keep the wand in check if they’ve lost their skills or never developed them in the first place?
When children use AI for daily tasks, they may never develop the cognitive skills necessary to think deeply, focus, or reflect, compounding the damage from mobile devices and social media. This is why the rush to shove generative AI into the classroom can have devastating consequences if not thought out or implemented with an actual plan and measurable goals.
Thinking of AI as a competitor instead of a collaborator spawns a different mindset.
AI competes with us, even as we use it for our own tasks. Thinking of AI as a competitor instead of a collaborator spawns a different mindset. A competitor may give you bad information. A competitor may want to take something from you. Thinking adversarially brings a bit of skepticism and allows us to erect guardrails around activities we’d like to protect and outputs we may need to check. This is the best of both worlds, allowing us to consider using AI selectively instead of indiscriminately. So, ponder this the next time one of the just use AI for everything people starts running their mouths.
Personal AI Personas
When considering using personal AI tools in daily life, we can envision the manifestation of several personas. These personas will play various roles in daily life, crossing personal and professional boundaries. These personas supercharge overreliance risks by outsourcing cognitive functions to these tools, making us even more dependent on the technology and fueling even more use.
I’ve broken this outsourcing into the following six personas representing roles that personal AI tools will assume in daily life.
The Oracle
The Recorder
The Planner
The Creator
The Communicator
The Companion
Each role represents an outlet for cognitive offloading and contributes to potential cognitive illusions and cognitive atrophy. The most obvious is the illusion of knowledge, but the list of cognitive illusions is a conversation for another day.
In a way, we are outsourcing authority as well, allowing these systems control over our daily lives, perceptions of the world, and even our actions. The more we outsource to personal AI systems, the less we will be able to keep them in check. There are no firewalls around these personas or around the tasks and task types we feed to personal AI systems. This blending of tasks and personas leads to quite a few downsides.
Note: Although I won’t be diving into the harms when I imply there are negative impacts from allowing AI to play these personas, these impacts manifest from repeated and even overuse of the technology for the role or activity. Being selective about use and application minimizes impacts and should be the goal, allowing us maximum benefit while minimizing negative impacts. Also, I’m merely introducing the persona with a brief description, not diving deeply into each in an attempt to keep the word count of this post in check. I may expand on these later.
The Oracle
The Oracle persona manifests when people use AI tools as an all-knowing question-and-answer system. Since, deceptively, the system appears to have representative knowledge of humanity, users are happy to type questions and receive answers, closing the loop on curiosity. However, it’s important to note that the questions asked to an AI-based oracle run far deeper than retrieving facts you’ve forgotten, such as retrieving the year the song Under Pressure was released.
Take, for example, questions about who you should marry or even who you should be friends with. Answers to such deep questions should come from exploration, not a Q&A system. Of course, these questions won’t be asked in such a straightforward way. They may be combined with The Planner persona to achieve a goal, such as maximizing life happiness or trying to optimize your career. Through these activities, we dehumanize people, turning them into objects to be manipulated rather than other human beings living their own lives with their own thoughts and emotions.
These systems appear to know more and know better than us, so we will inevitably overuse these systems for all sorts of decisions in our daily lives, receiving more answers and questioning even less.
The Recorder
One of our most obvious cognitive limitations is our brain’s capability for recall. There is only so much we can remember and surface when needed. This limitation is why we set calendar appointments or scribble reminders on sticky notes. Even when we make a purposeful effort to remember things, we can still forget if too much information is given or there is too much time between needing to recall the information.
With personal AI systems, even less cognitive effort will be expended for remembering things. We will count on these systems to remember things on our behalf. Agents running on systems will record and transcribe whatever we choose. Meetings, emails, YouTube videos, podcasts, personal conversations, and everything in between are all recorded and available whenever we want to review them. Even if we don’t want to review them, insights will be distilled for us automatically. There will be no reason to be fully present ever again.
The recorder role not only records but, combined with The Oracle persona, also makes sense of the content for us. It may seem like optimization when our personal AI tool spits out a single action item from a one-hour meeting we missed or weren’t paying attention to, but our lack of presence has negative impacts.
We didn’t have a seat at the table and couldn’t influence the direction or demonstrate our value to the project, conversation, or leadership. We weren’t able to build bridges or foster connections with others. We may also get the wrong idea and context from the meeting. Sure, maybe the full transcript is available, but if we feel these tools are created to optimize our time, why would you go back and read the transcript or play the full meeting recording? This is a surefire recipe for miscommunications and other issues.
The negative impacts run deep. The less we use our memory, the worse it gets. Socrates was right.
The Planner
We’ll use The Planner persona when we want to set a goal for the system to accomplish on our behalf. The system will use its capabilities and connections to perform all of the planning and tasks necessary to accomplish the goal, setting all of the activities in motion, with our brains doing none of the work.
Humans plan and execute every day without even realizing it. Much of this planning and execution is done subconsciously. For example, if we wanted a bowl of cereal but realized we had no milk, we may formulate a plan to rectify the situation. This plan may include putting on our shoes, grabbing our keys, driving to the store, purchasing the milk, and returning home. We don’t document this plan or map out a strategy, but it is formulated subconsciously in our prefrontal cortex and executed without much thought. But planning isn’t just for simple things like getting milk or considering what to wear for the day. So much of our daily lives contain planning and strategy.
Regarding personal AI, we assign authority to these systems due to our perception of their capabilities, but these can also be illusions. AI contributes to the illusion of knowing more and better than humans. This assumption isn’t new and even has its own bias, automation bias, which was mentioned earlier. Automation bias is the tendency of people to prefer the output of automated systems, even when contradictory information is present. We tend to know that humans are flawed, biased, and prone to mistakes, so we trust the output of these automated systems more than our judgments or the judgments of others.
Extending the Oracle persona, we will use these systems for feedback and direction on all sorts of work-related and personal tasks. We will treat these systems as the authority, assuming they know best, and allow them to make critical and benign decisions on our behalf. This will extend far beyond the typical scenarios people associate with automation bias.
With the advent of personal AI, we will count upon these tools to plan just about everything, plotting a course to goals and mindlessly nudging us in various directions. Although this may seem like a sound thing to do, many will use these tools to plan all sorts of things that we don’t use tools to plan today. For example, we may want a personal AI tool to plan a night out with a significant other or maybe to optimize finding a significant other in the first place.
The Creator
Using AI tools to create things is a common task today. Many use these tools to generate images and write creative content. The creator persona is about much more than just creating images. It’s for when the tool does the work of creation across various use cases, including writing, coding, games, and many others.
To focus on creativity for a moment, anyone who’s ever truly been creative knows that surprise is an important part of creativity. In the book I, Human, Tomas Chamorro-Premuzic says:
Surprise is a fundamental feature of creativity. If you are not acting in unexpected or unpredictable ways, then you are probably not creative.
I think this is true, but I’d also take that a bit further. Many may claim they are surprised at the output of a generative AI tool and that this is the same thing, but it’s not. Being surprised isn’t the same as surprising yourself. Surprising yourself is the primary satisfaction that results from creative endeavors. It can be hard to understand the difference if you’ve never surprised yourself or noticed surprising yourself, but that doesn’t mean there isn’t one.
Being surprised isn’t the same as surprising yourself.
Ultimately, the creator persona deprives us of creative satisfaction and creates the illusion of creativity. I’ll expand my thoughts on this in the future.
The Communicator
The communicator persona is when we outsource communication between humans to AI tools. We can think of this as something as simple as using AI to construct an email or something more complex like creating a bot with our voice to talk with our parents so we don’t have to. It may seem like there aren’t any downsides to the communicator persona, but there are impacts when we outsource these interactions to AI. I’ve written about this previously in how we are optimizing away human interactions with AI.
As communication has moved online and become more asynchronous, we’ve lost touch with some of the subtler aspects of human communication. This has led to us feeling that communication is more of a burden. With today’s online business and distributed workforce, communication with other humans has become viewed as a task or a checklist.
This is why one of the touted examples of these AI systems is handling email in our inbox, automatically prioritizing messages, and responding on our behalf. Therefore, the human aspect of this communication is removed, and the task portion is checked off. But even in the boring world of business communication, the human aspect is still important.
When we outsource communications to automation, we miss opportunities to build relationships and make our voices and opinions heard in critical contexts. This leads to a lack of trust and importance. Suppose it came time for a workforce reduction. Would we let go of a resource that provided valuable feedback and engaged in communication or the one that outsourced responses to a bot and couldn’t be bothered with responding back to us?
More importantly, we miss opportunities to connect with our fellow humans and build relationships with them, opting to treat others as tasks or objects that need to be manipulated. When we let our communication skills atrophy, a whole host of uniquely human qualities disappear, transforming us into machines.
The Companion
The Companion persona is when the AI tool acts as a friend or romantic partner. The Companion persona isn’t part of the future state of technology. It’s happening today. Startups like Friend, Character.ai, Replika, and many others are pushing this use, sometimes with devastating consequences. These companies are even marketed with straight-up bullshit.
That’s right, a soul. Our chatbot has a soul and a deep connection to us, yet it doesn’t care whether we live or die. I’ve written about this nonsense previously, so I won’t go deeper into it here.
As personal AI tools become more part of our daily lives, more people will begin to feel a connection with them, mistaking the interactions for meaning. This will fuel the illusion of companionship and lead to more devastating consequences for our mental health and humanity.
Cognitive Illusions
Cognitive illusions manifest from the overuse of these tools in the mentioned personas. These illusions cause a wide range of negative impacts on our health and wellness, as well as our cognitive abilities.
I won’t cover the illusions created by these personas in-depth, but here are some highlights.
Illusion of Knowledge
Illusion of Capability
Illusion of Memory
Illusion of Agency/Control
Illusion of Presence
Illusion of Creativity
Illusion of Certainty
Illusion of Companionship
Conclusion
In the next few years, these tools will be pushed closer and closer to us in a quest for profitability. All of the known flaws with this technology will not be fixed, but even if they were, that wouldn’t be the extent of the harm. This is why I created SPAR to frame the conversation around personal AI safety.
However, this article covers harms that extend beyond the technical issues and make the harms personal. We must be selective in using these systems and draw firewalls around tasks and activities we want to protect, an increasingly difficult task in a world where we prefer the easy button.