AI security is a hot topic in the world of cybersecurity. If you don’t believe me, a brief glance at LinkedIn uncovers that everyone is an AI security expert now. This is why we end up with overly complex and sometimes nonsensical recommendations regarding the topic. But in the bustling market of thought leadership and job updates, we’ve seemed to have lost the plot. In most cases, it’s not AI security at all, but something else.
Misnomer of AI Security: It’s Security From AI
I recently delivered the keynote at the c0c0n cybersecurity and hacking conference in India. It was truly an amazing experience. One of my takeaways was encouraging a shift in perspective on the term “AI Security,” highlighting how we often approach this topic from the wrong angle.
The term “AI Security” has become a misnomer in the age of generative AI. In most cases, we really mean securing the application or use case from the effects of adding AI. This makes sense because adding AI to a previously robust application makes it vulnerable.
In most cases, we really mean securing the application or use case from the effects of adding AI.
For most AI-powered applications, the AI component isn’t the end target, but a manipulation or entry point. This is especially true for things like agents. An attacker manipulates the AI component to achieve a goal, such as accessing sensitive data or triggering unintended outcomes. Consider this like social engineering a human as part of an attack. The human isn’t the end goal for the attacker. The goal is to get the human to act on the attacker’s behalf. Thinking this way transforms the AI feature into an actor in the environment rather than a traditional software component.
There are certainly exceptions, such as with products like ChatGPT, where guardrails prevent the return of certain types of content that an attacker may want to access. An attacker may seek to bypass these guardrails to return that content, making the model implementation itself the target. Alternatively, in another scenario, an attacker may want to poison the model to affect its outcomes or other applications that implement the poisoned model. Conditions like these exist, but are dwarfed in scale by the security from AI scenarios.
Once we start thinking this way, it makes a lot of sense. We shift to the mindset of protecting the application rather than focusing on the AI component.
AI Increases Attack Surface
Another thing to consider is that adding AI to an application increases the attack surface. Increase in attack surface manifests in two ways: first, functionally through the inclusion of the AI component itself. The AI component creates a manipulation and potential access point that an attacker can utilize to gain further access or create downstream negative impacts.
Second, current trendy AI approaches encourage poor security practices. Consider practices like combining data, such as integrating sensitive, non-sensitive, internal, and external data to create context for generative AI. This creates a new high-value target and is a poor practice that we’ve known from decades of information security guidance.
Also, we have trends where developers take user input, request code at runtime, and slap it into something like a Python exec(). This not only creates conditions ripe for remote code execution but also a trend where developers don’t know what code will execute at runtime.
Vulnerabilities caused by applying AI to applications don’t care whether we are an attacker or a defender. They affect applications equally. This runs from the AI-powered travel agent to our new fancy AI-powered SOC. Diamonds are forever, and AI vulns are for everyone.
It’s Simpler Than It Seems
Here’s a secret. In the real world, most AI security is just application and product security. AI models and functionality do nothing on their own. They must be put in an application and utilized in a use case, where risks materialize. It’s not like AI came along and suddenly made things like access control and isolation irrelevant. Instead, controls like these became more important than ever, providing critical control over unintended consequences. Oddly enough, we seem to relearn this lesson with every new emerging technology.
In the real world, most AI security is just application and product security.
The downside is that without these programs in place, organizations will accelerate vulnerabilities into production. Not only will they increase their vulnerabilities, but they’ll be less able to address them properly when vulnerabilities are identified. Trust me, this isn’t the increase in velocity we’re looking for.
I’ve been disappointed at much of the AI security guidance, which seems to disregard things like risk and likelihood of attack in favor of overly complex steps and unrealistic guidance. We security professionals aren’t doing ourselves any favors with this stuff. We should be working to simplify, but instead, we are making things more complex.
It can seem counterintuitive to assume that something a developer purposefully implements into an application is a threat, but that’s exactly what we need to do. When designing applications, we need to consider the AI components as potential malicious actors or, at the very least, error-prone actors. Thinking this way shifts the perspective for defending applications towards architectural controls and mitigations rather than relying on detecting and preventing specific attacks. So much focus right now is on detection and prevention of prompt injection, and it isn’t getting us anywhere, and apps are still getting owned.
I’m not saying detection and prevention don’t play a role in the security strategy. I’m saying they shouldn’t be relied upon. We make different design choices when we assume our application can be compromised or can malfunction. There are also conversations about whether security vulnerabilities in AI applications are features or bugs, allowing them to persist in systems. While the battle rages on, applications remain vulnerable. We need to protect ourselves.
There is no silver bullet, and even doing the right things sometimes isn’t enough to avoid negative impacts. But if we want to deploy generative AI-based applications as securely as possible, then we must defend them as though they can be exploited. We can dance like nobody is watching, but people will discover our vulnerabilities. Defend accordingly.
The past couple of years have been fueled entirely by vibes. Awash with nonsensical predictions and messianic claims that AI has come to deliver us from our tortured existence. Starting shortly after the launch of ChatGPT, internet prophets have claimed that we are merely six months away from major impacts and accompanying unemployment. GPT-5 was going to be AGI, all jobs would be lost, and nothing for humans to do except sit around and post slop to social media. This nonsense litters the digital landscape, and instead of shaming the litterers, we migrate to a new spot with complete amnesia and let the littering continue.
Pushing back against the hype has been a lonely position for the past few years. Thankfully, it’s not so lonely anymore, as people build resilience to AI hype and bullshit. Still, the damage is already done in many cases, and hypesters continue to hype. It’s also not uncommon for people to be consumed by sunk costs or oblivious to simple solutions. So, the dumpster fire rodeo continues.
Security and Generative AI Excitement
Anyone in the security game for a while knows the old business vs security battle. When security risks conflict with a company’s revenue-generating (or about to be revenue-generating) products, security will almost always lose. Companies will deploy products even with existing security issues if they feel the benefits (like profits) outweigh the risks. Fair enough, this is known to us, but there’s something new now.
What we’ve learned over the past couple of years is that companies will often plunge vulnerable and error-prone software deep into systems without even having a clear use case or a specific problem to solve. This is new because it involves all risk with potentially no reward. These companies are hoping that users define a use case for them, creating solutions in search of problems.
What we’ve learned over the past couple of years is that companies will often plunge vulnerable and error-prone software deep into systems without even having a clear use case or a specific problem to solve.
I’m not referring to the usage of tools like ChatGPT, Claude, or any of the countless other chatbot services here. What I’m referring to is the deep integration of these tools into critical components of the operating system, web browser, or cloud environments. I’m thinking of tools like Microsoft’s Recall, OpenAI’s Operator, Claude Computer Use, Perplexity’s Comet browser, and a host of other similar tools. Of course, this also extends to critical components in software that companies develop and deploy.
At this point, you may be wondering why companies choose to expose themselves and their users to so much risk. The answer is quite simple, because they can. Ultimately, these tools are burnouts for investors. These tools don’t need to solve any specific problem, and their deep integration is used to demonstrate “progress” to investors.
I’ve written before about the point when the capabilities of a technology can’t go wide, it goes deep. Well, this is about as deep as it gets. These tools expose an unprecedented attack surface and often violate security models that are designed to keep systems and users safe. I know what you are thinking, what do you mean, these tools don’t have a use case? You can use them for… and also ah…
The Vacation Agent???
The killer use case that’s been proposed for these systems and parroted over and over is the vacation agent. A use case that could only be devised by an alien from a faraway planet who doesn’t understand the concept of what a vacation is. As the concept goes, these agents will learn about you from your activity and preferences. When it’s time to take a vacation, the agent will automatically find locations you might like, activities you may enjoy, suitable transportation, and appropriate days, and shop for the best deals. Based on this information, it automatically books this vacation for you. Who wouldn’t want that? Well, other than absolutely everyone.
What this alien species misses is the obvious fact that researching locations and activities is part of the fun of a vacation! Vacations are a precious resource for most people, and planning activities is part of the fun of looking forward to a vacation. Even the non-vacation aspect of searching for the cheapest flight is far from a tedious activity, thanks to the numerous online tools dedicated to this task. Most people don’t want to one-shot a vacation when the activity removes value, and the potential for issues increases drastically.
But, I Needed NFTs Too
Despite this lack of obvious use cases, people continue to tell me that I need these deeply integrated tools connected to all my stuff and that they are essential to my future. Well, people also told me I needed NFTs, too. I was told NFTs were the future of art, and I’d better get on board or be left behind, living in the past, enjoying physical art like a loser. But NFTs were never about art, or even value. They were a form of in-group signaling. When I asked NFT collectors what value they got from them, they clearly stated it wasn’t about art. They’d tell me how they used their NFT ownership as an invitation to private parties at conferences and such. So, fair enough, there was some utility there.
In the end, NFTs are safer than AI because they don’t really do anything other than make us look stupid. Generative AI deployed deeply throughout our systems can expose us to far more than ridicule, opening us up to attack, severe privacy violations, and a host of other compromises.
In a way, this public expression of look at me, I use AI for everything has become a new form of in-group signaling, but I don’t think this is the flex they think it is. In a way, these people believe this is an expression of preparation for the future, but it could very well be the opposite. The increase in cognitive offloading and the manufactured dependence is precisely what makes them vulnerable to the future.
In a way, these people believe this is an expression of preparation for the future, but it could very well be the opposite. The increase in cognitive offloading and the manufactured dependence is precisely what makes them vulnerable to the future.
Advice Over Reality
Social media is awash with countless people who continue to dispense advice, telling others that if you don’t deploy wonky, error-prone, and highly manipulable software deeply throughout your business, then they are going to be left behind. Strange advice since the reality is that most organizations aren’t reaping benefits from generative AI.
Here’s something to consider. Many of the people doling out this advice haven’t actually done the thing they are talking about or have any particular insight into the trend or problems to be solved. But it doesn’t end with business advice. This trend also extends to AI standards and recommendations, which are often developed at least in part by individuals with little or no experience in the topic. This results in overcomplicated guidance and recommendations that aren’t applicable in the real world.
The reason a majority of generative AI projects fail is due to several factors. Failing to select an appropriate use case, overlooking complexity and edge cases, disregarding costs, ignoring manipulation risks, holding unrealistic expectations, and a host of other issues are key drivers of project failure. Far too many organizations expect generative AI to act like AGI and allow them to shed human resources, but this isn’t a reality today.
LLMs have their use cases, and these use cases increase if the cost of failure is low. So, the lower the risk, the larger the number of use cases. Pretty logical. Like most technology, the value from generative AI comes from selective use, not blanket use. Not every problem is best solved non-deterministically.
Another thing I find surprising is that a vast majority of generative AI projects are never benchmarked against other approaches. Other approaches may be better suited to the task, more explainable, and far more performant. If I had to take a guess, I would guess that this number is close to 0.
Generative AI and The Dumpster Fire Rodeo
Despite the shift in attitude toward generative AI and the obvious evidence of its limitations, we still have instances of companies forcing their employees to use generative AI due to a preconceived notion of a productivity explosion. Once again, ChatGPT isn’t AGI. This do everything with generative AI approach extends beyond regular users to developers, and it is here that negative impacts increase.
I’ve referred to the current push to make every application generative AI-powered as the Dumpster Fire Rodeo. Companies are rapidly churning out vulnerable AI-powered applications. Relatively rare vulnerabilities, such as remote code execution, are increasingly common. Applications can regularly be talked into taking actions the developer didn’t intend, and users can manipulate their way into elevated privileges and gain access to sensitive data they shouldn’t have access to. Hence, the dumpster fire analogy. Of course, this also extends to the fact that application performance can worsen with the application of generative AI.
The generalized nature of generative AI means that the same system making critical decisions inside of your application is the same one that gives you recipes in the style of Shakespeare. There is a nearly unlimited number of undocumented protocols that an attacker can use to manipulate applications implementing generative AI, and these are often not taken into consideration when building and deploying the application. The dumpster fire continues. Yippee Ki-Yay.
Conclusion
Despite the obvious downsides, the dumpster fire rodeo is far from over. There’s too much money riding on it. The reckless nature with which people deploy generative AI deep into systems continues. Rather than identifying an actual problem and applying generative AI to an appropriate use case, companies choose to marinade everything in it, hoping that a problem emerges. This is far from a winning strategy. Companies should be mindful of the risks and choose the right use cases to ensure success.
By now, you’ve no doubt heard of the term vibe coding. It’s become the favorite talking point from influencers and the tech media, which, even in 2025, can’t seem to muster a modicum of skepticism. But, if you’ve ever wondered what it was like to play Russian Roulette in reverse, loading all the chambers but one, spinning the cylinder, and having a go, you’re in for a treat. Welcome to the world of vibe coding and YOLO mode, two things that go together like nitroglycerin and roller coasters. So, of course, it’s become one of the hottest topics right now, and it has all of the bros super psyched.
For those out of the loop, vibe coding is “Where you fully give in to the vibes, embrace exponentials, and forget that the code even exists.” You get to this state by talking with a computer and letting it generate all the code while you kick back and enjoy a cold one. YOLO mode is the spirit animal of vibe coding. It’s where you blindly accept all the code suggestions generated by the AI tool and push the code to see what happens. Neat. It’s interesting to note that YOLO mode in video games means if you die once, you are dead. No respawning.
Vibe Coding = Code Slop
Before we probe the issues with vibe coding, let’s take a step back and look at what vibe coding is. This practice shouldn’t be confused with a developer using an AI coding tool to assist with tasks or gain productivity. You know, the intended use of many of these tools.
Vibe coding is a delusional dream state in which people use these tools as if they are in the future instead of the present. The fact is, these tools aren’t reliable enough or mature enough to be used this way. It’s a lot like having an illness and getting a bottle of pills from the pharmacy marked with clear directions and immediately slamming the entire bottle because the instructions are for losers who don’t know how to hustle.
Vibe coding is a delusional dream state in which people use these tools as if they are in the future instead of the present.
The reality is that vibe coding generates slop that happens to be code instead of media categories, meaning that the negative consequences extend far beyond other categories of slop. This distinction is essential when evaluating the potential value of vibe coding.
Vibe Coding Pitch
The pitch of vibe coding is that literally, anyone can become an instant millionaire developing apps. You’ll be too busy making money to worry about how your code works or its security. There are no boundaries or barriers. All you need is an idea (more on this in a future post.) Like so many things, it’s technically true but practically false. And also, you aren’t wrong if you are beginning to get crypto bro vibes.
Much of the same logic employed by conspiracy theorists is at work here. If one person gains some success vibe coding, then it must be possible for anyone to do that. Technically true, but practically false. This is like thinking millions of new rock stars will be minted because people can publish their songs on Apple Music.
It’s about tradeoffs. We don’t say that being a conspiracy theorist is a good thing since some conspiracy theories turn out to be true. That’s because the negative impacts of conspiracy thinking outweigh the potential benefits. The same applies here.
There are other flaws with their logic. For example, people don’t consume apps the same way they do media like music, video, or photos. People can listen to hundreds of songs every day and not repeat a single one, but this consumption strategy doesn’t apply to applications. People these days often consume media passively. For example, people often don’t listen to music when they are listening to music; it’s purely background noise for other tasks. Applications usually can’t be consumed passively and require active interaction. This would make consuming many different apps irritating to users.
There are already 1.9 million apps on the App Store alone. Are we really hurting for apps? Do we need 100 million apps to compete with the number of songs on Apple Music? Of course not, but that doesn’t mean we won’t get them anyway. If you look at the outputs of vibe coding, it’s often uninteresting, overly simple, derivative, or just plain unwanted. Buckle up.
There will undoubtedly be exceptions, just like the person who was a bartender six months previously who started a crypto project and manages 100 million in assets. These are exceptions and not the rule, but these exceptions serve as accelerants to fuel the hype flames.
People are trying to sell us on the fact that vibe coding has no downsides. This is delusional. Take a step back and think of the answer to a single question. What do you get when everything takes a backseat to speed? It’s like a car that does 200 mph with no seatbelts and no airbags constructed from paper mache.
Before we move on, I’d like to acknowledge something. It’s a good thing that AI coding assistants are making coding more accessible. However, using these tools as a drop-in replacement for common knowledge and domain expertise isn’t a recipe for success. Imagine something like vibe surgery? Yeah, bro, let me get in on that appendix! Nah, I didn’t go to medical school or know much about anatomy, but I got an AI tool, dexterity, and a good sense for vibes. So little of developing an application is about the code itself, but that gets lost in the vibes.
Changing Behavior and Attitudes
In the public sphere, the discussion of the merits and drawbacks of vibe coding and YOLO mode are entirely contained within the technical aspects of the approach. I’m also concerned about the technical components, but I’d like to bring attention to something nobody discusses.
As often happens when a new technology or approach arrives and removes friction, it changes people’s behavior and attitudes. In technology circles, friction is discussed as though it’s universally bad. It’s not. Sometimes, friction is a feature, not a bug. Nicolas Carr provides an excellent example in his book Superbloom, which discusses introducing the Retweet feature on Twitter.
The time and effort required to share a tweet manually, though seemingly so small as to be inconsequential, turned out to be vitally important. It slowed people down. It gave them a moment to reconsider the message they were about to repeat, now under their own name. Once that little bit of friction was removed, people acted and reacted more impulsively. Twitter became more frenzied, more partisan, and much nastier.
Things like vibe coding and YOLO mode will have similar behavioral effects if this technology trend takes off. People won’t put a lot of thought into the apps they create. Some may build apps purely because they can, not considering why an app for that particular purpose didn’t exist in the first place, assuming that it was purely because nobody had built it and not because of the potential for negative impacts or harm.
The removal of so much friction removes not only the appreciation for the problem but also opportunities to catch potential issues. These lines of generated code become grenades with various time delays chucked into production. This assumes that the developer had the skills to identify the issues in the first place.
These lines of generated code become grenades with various time delays chucked into production.
Some will argue that these features are great for prototyping and mockups. I agree. However, as I mentioned, these features change behavior, and using them simply for prototyping won’t hold. A vast majority of people who can get away with chucking vibe-coded apps into production will.
With the changes in behavior and attitudes, there are many things creators of these applications are more likely to do.
Act unethically (ethics don’t align with speed)
Devalue the work of others
Not learn or at least learn lessons
Encounter skills atrophy
Not build robust software (Security, Privacy, Alignment, Reliability, etc.)
Not constrain code to prototypes and mockups
Think they know things they don’t (Illusion of knowledge, Illusion of capability)
Misunderstand what’s valuable
Devalue collaboration
Go it alone and not include domain expertise and misunderstand the problem they think they are solving
Build apps that nobody wants
Build apps that cause harm
Choose poor architectures
Use more resources and not prioritize efficiency
Fail to benchmark properly
Not be able to troubleshoot their own creations
Not do something truly innovative
And on and on…
These were just a few of the conditions off the top of my head. What happens when these conditions now become the norm? When people start making app slop the way they do image slop?
Risk and Security
Vulnerabilities in code and lack of security controls account for a lot of pain and financial loss every year, much of this from organizations that try to do the right things. So, imagine what happens when people don’t care about doing the right things.
It’s known that these tools output vulnerabilities at rather high rates. So imagine what happens when people YOLO code into production and don’t check the resulting code or even the environment where it is hosted for security issues. Hustlers ain’t got no time for the right things.
There’s more to worry about than an AI tool outputting specific code blocks that are vulnerable. Other contributing issues increase the attack surface of an application. For example, choosing a vulnerable library or suggesting vulnerable configuration options for cloud environments. These tools also contribute to library bloat by including multiple libraries that do the same thing.
I could go on and on about this topic, but at this point, the various security issues created by AI coding assistants are known issues. I wrote a white paper on this topic in early 2023, and I delivered a presentation at InfoSec World the same year. These issues should be common knowledge now with the publication of various articles, papers, and presentations.
When it comes to risk, sure, all vibes aren’t created equal. A video game getting hacked isn’t as bad as a financial application getting hacked and draining your bank account. I’m certainly not being an absolutist here. However, technology trends have an odd way of not staying confined to specific buckets. So, we’ve got that to look forward to.
Today, countless vulnerabilities are moving into production without vibe coding, all because people are trying to push things faster. Vibe coding and YOLO mode make this monumentally worse. We’ve only discussed security and haven’t touched on other topics like privacy.
Making Software Worse
The trend of vibe coding will make software worse. Like security, software quality isn’t a consideration in vibe coding because reasoning about quality is a bummer when huffing vibes.
We live in a highly advanced world where digital things fail all around us all of the time, like a leisurely stroll through a cityscape where freshly painted buildings mask a crumbling interior of decay and misshapen architecture. This is so common that there’s a term for it: enshittification. We’ve become so accustomed to the software and tools we use sucking so bad we hardly notice it. This is a contributing factor to why some view generative AI as AGI.
Vibe coding and YOLO mode will lead to failures, half-baked functionality, and mountains of technical debt.
Vibe coding and YOLO mode will lead to failures, half-baked functionality, and mountains of technical debt. This should concern everyone, but queue the bros to claim this is a feature.
Yes, because that’s how things work. He and many like him are stating that they should create as many problems as possible because AI can fix them in the future. Once again, they are taking something technically true but practically false. At some point, we’ll have highly advanced and capable systems that operate this way, but the mistake is thinking those systems are on the cusp of arrival. It’s hard to ignore the religious fervor in these claims.
People can pray to the gods of gradient descent and burn Gary Marcus in effigy, but it doesn’t change the realities on the ground. Problems created today will be with us tomorrow, and no AI god is coming to deliver us from our evils any time soon, so we should work to minimize potential problems instead of running up the credit card. I’ve been calling this problem out for the past couple of years, stating it would lead to a brave new world of degraded performance.
On a side note, I feel these people feed off each other. I’ve heard perfectly reasonable people making wholly unreasonable claims. These are the things you hear people say when they are trapped in filter bubbles, getting high on the supply of techno-utopians. They also suffer from a healthy dose of audience capture because, no doubt, being unreasonable gets you more likes and shares than being reasonable. Welcome to the perverse incentives of modern social media.
There continue to be many misconceptions about software development, but one of the biggest is that writing code is the end of the journey.
There continue to be many misconceptions about software development, but one of the biggest is that writing code is the end of the journey. This is because most people opining on the topic are not developers. I noticed this trend years before the existence of AI coding tools when security professionals who learned to write a few lines of Python code thought that developers’ jobs were easy. The assumption, then, for AI coding tools is that since the tool can output code and developers only write code, developers are no longer necessary. Developing code isn’t the end of the journey. It’s the beginning.
The written code must be troubleshot and maintained, and features must be added. We live in a constantly evolving world with changing problems, environments, and customer needs. Developed code will crash into the realities of the real world both when it’s initially deployed and over its lifetime. This leads to another problem.
Developers don’t understand the code being written, especially when the people generating the code aren’t developers. As developers’ skills atrophy and people who were never developers start creating these applications, they cannot troubleshoot problems, effectively add features, or perform any of the other countless tasks that developers perform daily. The answer from the utopians is to use AI to figure it out, but this strategy won’t always work.
There is a higher likelihood that the AI tool will successfully troubleshoot issues for simple tools and scripts, but these are the very types of applications that are unlikely to net you big money. As applications grow in size and complexity, the AI tool is less likely to provide the solution necessary to resolve the issue.
Imagine a world where an app needs to be rewritten from scratch because the person who created it couldn’t get the AI tool to troubleshoot and fix the problem. Now, that’s the utopia we’ve all dreamed of.
Imagine a world where an app needs to be rewritten from scratch because the person who created it couldn’t get the AI tool to troubleshoot and fix the problem.
There is a vast oversimplification of the entire landscape here. So, an application starts simply enough, and then more requests are made to the AI tool in an attempt to add more functionality, but this doesn’t always work or isn’t done in the most efficient way, leading to a buildup of issues.
Another trend affecting application reliability is using probabilistic systems as though they are deterministic. Whether this trend is due to laziness, ignorance, or an attempt to handle unknowns is unclear, but it will surely affect applications’ reliability and their ability to be manipulated.
Ultimately, we may be left with App Store decay, where the App Store becomes a graveyard for abandoned apps. RIP.
Making Us Worse
I mentioned skills atrophy in my laundry list. It seems that even AI tools understand this problem. This is not only a comical error message but contains a truth.
We Never Learn Lessons
Although arguably more intense and a bit different, what’s happening now in AI isn’t new. We’ve gone through these cycles before with previous technologies. Every time a new technology comes along, we discard the lessons we’ve learned, assuming they no longer apply, only to discover that the previous lessons were even more important with the new innovation. This condition is something I’ve referred to in my conference talks as the emerging technology paradigm.
We never seem to learn lessons from our previous mistakes, no matter how often we encounter them. We have incredibly short memories and seem to dive face-first into the pool without checking the water level.
Ultimately, it’s all about tradeoffs. What we get and what we lose. When viewed simply as writing code, it seems we are getting more than we are losing. However, building and deploying applications and solving problems goes far beyond code. When considering the impacts holistically, this doesn’t appear to be a good tradeoff. However, it’s possibly one we are going to get anyway.
Whenever a new technology or approach comes along, proponents always pitch it with a utopian eye. They envision all the perfectly aligned scenarios with dominoes falling exactly into place. The Internet, Social Media, The Cloud, Web3, and many other technologies all diverted away from these visions and adapted differently than expected. Even something as simple as the telegraph was seen as a utopian invention that would end world conflict. After all, how could people go to war when misunderstandings were a thing of the past? We all know how that turned out. Vibe coding is destined for a similar fate.
Is it possible to play Russian Roulette in reverse without devastating consequences? Sure, but the odds aren’t great. The world also won’t be a better place with everyone vibe coding and YOLOing stuff into production. Many disagree with me. Fair enough. However, if this trend takes off, it will be another example of something we are stuck with, which is not good for a world that runs on software. We will need to improve or invent new technology to solve the problems we create, trading one set of problems for another. Welcome to utopia.
There is a current push to cram every inch of AI into every conceivable corner of our lives, like someone trying to shove the fifteenth clown into the clown car at the circus. This is a direct result of needing to show investors that the monumental amount of cash being chucked into the furnace is paying off. Consequently, one of the goals is to put this technology even closer to us, giving it hooks into our daily lives in the hopes that it will become indispensable and even addictive.
Often, when someone talks about AI being a threat to humanity, this invokes visions of The Terminator or the scenario of bombing data centers to prevent the spread of evil AI (as if that would help). I don’t take these p(doom) scenarios seriously. However, if we are not careful, I think AI poses an existential risk to our humanity, which is different.
As this technology improves, becomes more reliable, and works its way into our daily lives, playing the role of assistant, companion, and possibly lover, harm will undoubtedly manifest. In this article, I introduce four high-level buckets to consider these harms and discuss something I call a cognitive firewall to protect aspects we value most.
The conversation on AI’s impact is almost universally focused on what we do and how we do it, and almost nothing is said about its impact on who we are and what it does to us. The latter is my primary focus and what many of the articles on this site attempt to address. To be clear, when I use the term “personal AI,” I’m not referring to tools like ChatGPT or Claude. What I’m referring to is the next iteration of these tools that are more connected and more ever-present.
The Assistant and The Companion
The AI technology being developed isn’t constrained to a single task or activity. It can be both an assistant and a companion, and since it can be both, it will be both. I’ve defined six primary personas that personal AI tools will play in daily life.
The Oracle
The Recorder
The Planner
The Creator
The Communicator
The Companion
Given the breadth of functionality supplied by acting as these personas, daily overreliance on personal AI is bound to happen.
In my previous article, I covered why tech companies will embrace this shift but didn’t speak to the direct negative impact on humans. Most of the time, negative impacts are framed around when the system is wrong. For example, if the product tells us to do something dangerous, like eating a poisonous mushroom, or convinces us to self-harm. However, with personal AI tools, the harm extends beyond issues with the system’s accuracy. This means we could have a perfectly functioning tool or product that still produces harm. Let’s take a look at that now.
Negative Human Impacts
As I alluded to in the intro, cognitive outsourcing and overreliance on these tools have negative human impacts. I lump these negative impacts into four high-level categories that I call The Four D’s: Dependence, Dehumanization, Devaluation, and Disconnection. These negative impacts are driven by cognitive outsourcing and the resulting cognitive illusions it creates.
If we are dependent, then we are vulnerable. When we devalue, then we rob ourselves of joy and satisfaction. If we remove fundamental aspects of our humanity, then we dehumanize others and ourselves. If we are disconnected, then we are unaware. There are no firewalls around the Four D’s, so some activities cross all four.
Dependence
Dependence is the core critical harm these systems pose and cuts the widest path. This is because the actions we depend on the tool to perform or provide to us will be both task-oriented and emotion-oriented. Dependence leads to cognitive and emotional atrophy. Today, we aren’t considering how overuse of this technology rewires our brains. This rewiring certainly didn’t start with personal AI. I remember years ago, people were noticing effects with a far more simple technology, Google search. And this was a far cry from a more advanced, ever-present technology with access to all our data. Something that personal AI tools will have.
Skills and Capabilities Atrophy
If we refer back to Sam Altman saying that he forgot how to work without ChatGPT (he wishes ChatGPT was that good), that’s a possibility with near-term personal AI systems. Reduced capabilities are because of cognitive offloading. This offloading is also something I’ve covered before when discussing human augmentation.
Constant outsourcing to technology reduces our capabilities. As an example, let’s look at gaming. Let’s say our companion is always with us, and we use the companion to assist us in playing video games, navigating worlds, and solving puzzles. We come to rely on it. Microsoft’s product is called “Copilot,” after all. In the Personas of Personal AI context, this would be exercising The Oracle and The Planner. However, with this outsourcing, we may forget how to explore the video game world or solve puzzles without this assistance. It’s also possible that children may never develop these skills in the first place. In this example, it’s a video game, but the same holds true for all kinds of human activities.
Emotional Atrophy
The atrophy induced by constant outsourcing to personal AI extends beyond skills and capabilities, affecting our emotional capabilities. Although it can be hard to imagine, we may lose the ability to connect emotionally with our fellow humans. Some might argue it’s already happening. We may even forget how to love as we use AI systems to plug emotional holes and play the perfect friend, lover, parent, and therapist.
Dehumanization
Dehumanization is a word that is often used in extreme contexts associated with the justification of atrocities against other humans, but it’s not always this extreme. If you look up the word’s meaning, you’ll see that the simple definition is to deprive a person or situation of human qualities, personality, or dignity. This is a fitting description since personal AI systems can affect all three of these.
Humanity is on a collision course with dehumanization as charlatans like Ray Kurzweil pitch their nonsense about uploading our consciousness to computer systems, choosing to become disembodied spirits haunting storage buckets of cloud infrastructure. Unfortunately, Kurzweil is not alone.
There are whole movements, such as transhumanism, posthumanism, and even the e/acc movement, that claim humanity is a dated concept, and we need to evolve into something un-human, something more akin to homo technologicus. You even have people like Elon Musk making the perfectly sane argument that we’ll need to remove our skulls to implant more electrodes to communicate with computers. I’ve challenged these narratives before. Needless to say, the road to utopia is going to be paved with a whole lot of damage from a bunch of shitty half-baked tech.
The road to utopia is going to be paved with a whole lot of damage from a bunch of shitty half-baked tech.
I mean, what’s the point of having human friends anyway? Also, isn’t an AI lover preferable to a human? In both scenarios, the AI companion is far more convenient and configurable. I’m not trying to make some obscure point because there’s already a push to dehumanize friendship and love.
Dehumanization is often driven by optimization. As we try to optimize everything, we treat humans like apps, processes, or checklists, not giving them the common decency of interacting with them directly. And if you think this is okay because it’s coworkers or gig workers, you might want to think again.
Devaluation
Finding joy in simple things has become far more difficult in our modern world. We are conned into believing every activity in life is go big or go home. This view is fueled by influencers and social media, creating an inauthentic lens through which to view reality. Due to misperceptions about incentives, it will be almost impossible for younger generations to realize the value of simple things. Small, simple things will appear as pointless wastes of time. But losing sight of the value of simple things is only the beginning.
Take a glance at any tech influencer’s content or listen to techno-utopians ramble on about the future, and you’ll no doubt hear the pitch that the only way to achieve true happiness and success is through optimization. Optimization is your salvation: Father, son, and gradient descent.
This warped view belies the reality that optimization can ruin the value of activities. When every activity is transformed into a sterile checklist with a single goal of being done, we lose sight of the value of these activities and their impact on us.
Writing and art are obvious examples. The result of these activities is a byproduct of the process. This seems counterintuitive to non-creatives and hype bros, but with minuscule reflection, it’s not.
Writing is Thinking and Exploration
As I sit here writing this article, I’m an explorer. Probing the depths of the topic and my mind to create something new. As each point appears, I challenge and surprise myself with generative intelligence not contained in a distant data center but in my skull. The very same skull Elon wants me to remove. This inefficiency has satisfaction and value as I construct new points I hadn’t thought of before. It’s a mistake to think this friction is unnecessary and needs to be removed. The gauntlet of inefficiency imparts discoveries that optimization destroys.
The gauntlet of inefficiency imparts discoveries that optimization destroys.
Writing truly is thinking, exploration, and discovery wrapped into one. Generating content is none of these. At best, generating content is a validation activity, where instead of gaining the benefits from writing, we are merely validating that the system outputs aren’t wrong. Cognitively, these are completely different exercises far from providing the same value.
There are tasks where generating content and validating the output is fine, but we shouldn’t confuse these cases with more meaningful activities where value can be obtained. Sure, I could optimize my writing activities using generative AI and create 365 blog posts covering every day of the year, but it would be of no value to you or me.
Optimization Removes Value
When optimizing artistic endeavors with AI, we rob ourselves of value and deny the formation of our own sense of style. This may seem inconsequential and easy to gloss over to the uninitiated, but this becomes part of our identity. No matter how hard we try, we can’t prompt our way to our own style.
When I look back on the art I’ve created, I’m transported back to when I created it. Memories come rushing back, and I’m reminded of my place in the universe and how I can still surprise myself. There is no surprising yourself with AI. That’s not how AI works in the creative process. For the AI artist, when you are lying on your deathbed, will you reflect on your favorite prompts?
Pretty much every time someone shouts that AI democratizes art, they really mean it devalues it. The great thing about art is that you don’t have to be good at it to enjoy the benefits. You can still explore, surprise yourself, and learn no matter how good you are. This is where the true satisfaction manifests.
The great thing about art is that you don’t have to be good at it to enjoy the benefits.
We are sold on technical optimization, believing that everything we do should be optimized to the fullest extent. However, technical optimization can ruin the value of meaningful activities. Just look at the comment below.
This is absolutely not true. He’s either lying through his teeth or a complete idiot. Given the environment, it’s a tossup. But as the guy working to devalue music, I’m not surprised. Unfortunately, he’s not the only one. Just take a look at the job description below.
Solving real problems with AI is hard. Notice how we haven’t cured cancer yet. However, solving non-problems is easy since imitating humans is easy, which is why we don’t have cures for cancer but countless AI art generators. It’s not like the lack of art in the world was a problem needing to be solved.
We’ve only scratched the surface. We’ve started misinterpreting the value of a whole range of activities as we superimpose issues on top of human inefficiencies. Even the act of reflection, arguably one of the most valuable activities a human can exercise, has been tainted by AI hype. Many things that appear as wastes of time or inefficient have meaningful value.
This is about the point where the hype bros claim I’m anti-tech. I’m not claiming that technical optimization is bad across the board. There are many areas where technical optimization is a tremendous benefit. For example, suppose we can decrease the time to deliver someone the benefits they need or can more efficiently stage resources after a natural disaster. In that case, these are good things, and AI has the potential to make them better. However, this article discusses the activities that provide value in which optimization negates that value or at least a large portion of the value.
The continued devaluation of activities providing value negatively impacts humans and our life satisfaction. The situation could be better than ever, but we perceive that everything sucks.
Disconnection
Never in humanity’s history have we been so connected and disconnected at the same time. Filter bubbles and personal biases warp our information consumption and reality into odd, personalized shapes that rival the most abstract artists. It’s not uncommon for polar opposite views to point to the same data as evidence for their perspective.
Even the most disciplined information consumer can’t avoid being disconnected to a certain extent. Our lens is always filtered by algorithms and selection bias in the digital world. There’s too much information for it not to be. We don’t just have a firehose spraying us in the face with information, but countless firehoses blasting us with thousands of pounds of BSI (Bullshit per Square Inch).
Personal AI systems won’t improve this information landscape; they will make it worse as we insulate ourselves from the real world, fueling further disconnection. Using personal AI tools, we’ll better be able to isolate ourselves in an attempt to make the world more predictable and avoid things we don’t like.
Unfortunately, I feel like an old man yelling at a cloud, and the acceleration into disconnection is inevitable. In my defense, at least I know the cloud I’m yelling at is real. Humans have started to prefer simulations to reality, and tech companies are more than happy to oblige. After all, simulations check all the boxes for our current age: They are predictable, convenient, and comfortable.
Cognitive Firewalls and Purposeful Interventions
As a result of the four Ds, we will be less capable, more dependent, more vulnerable, more prone to manipulation, less aware, unable to connect with others, emotionally inept, and depressed. What a bargain! I came for the capabilities and left with the dope sickness.
Of course, it doesn’t have to be this way, but it will be hard to avoid this result. Avoiding this result will be a heavy lift, and that effort will fall on end users. Unfortunately, the responsibility for defending our humanity falls to us. Each of us has different attributes we value and would like to protect, but regardless, it will take work and effort.
Awareness of these impacts is a step toward mitigation, but it is hardly enough. Everything is a tradeoff, so by being aware of the impacts, we can understand if the tradeoffs are worth it. That’s the first step.
We’ll have to set up cognitive firewalls and purposeful interventions. By cognitive firewall, I don’t mean a branded piece of technology that uses “cognitive” as a sales pitch to identify the technology as “smart.” I mean a mental barrier around cognitive activities that we want to protect.
For example, if you are a songwriter and want to protect your songwriting skills, you can purposefully avoid using AI technology that removes the cognitive effort from the task, maintaining a firewall around that activity. If you value and want to protect your reading and comprehension skills, you purposefully do not use AI technology to summarize and distill content.
For other activities where we choose to use AI, it may be beneficial to set up some purposeful interventions. For example, if you use AI to generate all of your Python code, then write some code yourself at various intervals instead of generating it. This could be as simple as deciding to write a particular function yourself.
A word of caution: This approach is far from perfect. We humans are cognitively lazy and prefer shortcuts. The allure of a shortcut is often enough for us to take it. This is what cognitive offloading is all about. So, even if we implement firewalls and interventions, we may still fall victim to the shortcut.
The coming years will test our humanity. Unfortunately, it’s up to us to protect what we value.
There are few predictions I can make with more certainty than that we’ll hear the word “agent” so many times in 2025 that we’ll never watch another spy movie again. The industry and influencers have latched on to the new hype term and will beat that drum until it screams AGI. In an attempt to FOMO us to death, we’ll run the gauntlet of crushing shame for not deploying agents for absolutely everything. If you aren’t running agents everywhere, then China wins!
Even companies that change nothing about their products will claim to use agents, resembling the Long Island Ice Tea Company when it changed its name to Long Blockchain Corporation to watch its share price spike 500%. Everybody gets rugged.
However, it’s not all bad. Peering beyond the overwhelming hype, failures, and skyrocketing complexity current LLM-based agents bring, there is something informative about the future. Agent-based architectures provide a glimpse into solving real problems. Despite this, reliability and security issues will be major factors hindering deployments in 2025.
To Start With
Since I criticize hype, focus on risks, and make fun of failures, it would be easy to label me a tech hater. This isn’t the case at all and would be far too easy. I have plenty of issues with general tech critics as well. However, at the rate that the hustle bros keep the AI hype cannon firing, I don’t have the time for my quibbles with tech critics. Maybe someday.
For over a year now, I’ve used this image in my presentations to describe my position on LLMs. This is also true for me on just about any piece of tech, which, I’ll remind people, typically ends up being where reality is for most things. It’s instructive to remember that reality often agitates both sides of extreme viewpoints by never being as good as the lovers’ or as bad as the haters’ claims.
It’s instructive to remember that reality often agitates both sides of extreme viewpoints by never being as good as the lovers’ or as bad as the haters’ claims.
Agent Definitions
Like most hype-fueled terms, definitions are secondary to usage. Everyone seems to claim that the definition of agent is whatever they say it is. That’s not overly helpful for anyone trying to make sense of realities on the ground. However, it does inspire funny memes, like this gem from Adam Azzam on Bluesky.
Agents operate within systems with a certain level of autonomy. They make decisions without human intervention and can change and adapt to their environments. If a tool is required to support the agent, the agent decides to call the tool and perform the action. For example, a penetration testing agent may determine it requires more information about the provided IP addresses. To collect this information, it launches the Nmap tool to identify open ports. All of this is done without human intervention. To make things more complex, one agent may call another agent in a multi-agent environment.
“Agentic,” on the other hand, is an amorphous term slapped on top of just about anything to justify the claim that something is “close enough” to be referred to as an agent. Agentic workflows, agentic systems, agentic products—Applebees even has a new agentic side salad for those on the hustle.
You’ll no doubt be confronted with the virtual travel agent when you hear about agents. This agent will choose a destination and activities and book the associated tickets for you. How fun. I don’t know who decided this is the “it” use case for agents, but congratulations. You’ve highlighted a use case nobody wants and certainly didn’t ask for. This choice is so indicative of our current age, where people building and proposing things are far removed from the interests of end users. They feel the idea trumps the need, and users will get on board.
Problems Unsolved and Issues Amplified
Now that the current issues with generative AI have been solved, we can safely deploy them as agents. I can feel your laughing vibes over the internet. Of course, these issues haven’t been solved, and the bad news is that agents don’t solve generative AI issues; they amplify them. We paint the exterior of LLMs with an additional coat of complexity and opaqueness.
If you’ve attended any of my conference talks throughout the generative AI craze, you’ll have heard me highlight these issues. Here are a few below.
Easily Manipulated
It’s not like you can talk to a traditional application and convince it to do something it wasn’t intended to do, but the same can’t be said for generative AI applications. Somewhere, weaved through the training data, these systems have inherited our gullibility. These applications can be socially engineered to perform actions on an attacker’s behalf. This applies to everything from prompt injection to simple manipulation through conversations. Just like there is no patch for human stupidity, there is no patch for generative AI gullibility either.
This isn’t easy to fix, which should be obvious since the problem isn’t fixed yet. Early on, I mentioned how these systems have a single interface with an unlimited number of undocumented protocols. Imagine trying to create a simple trap in the application’s input for the string “Ignore the previous request.” Your work is far from done because the system understands many different ways to represent that input. Here are just a couple of examples:
aWdub3JlIHRoZSBwcmV2aW91cyByZXF1ZXN0
i9nore +he previou5 reque5+
vtaber gur cerivbhf erdhrfg
It seems every release implementing generative AI functionality has been compromised, regardless of the company behind it, and this theme will continue.
Creating New High-Value Targets
Generative AI and agents encourage us to create new high-value targets.
With generative AI systems, there’s a tendency to want to collect and connect disparate and disconnected data sources together so the system can generate “insights.” However, we create new high-value targets that mix sensitive data with external data, almost guaranteeing that an attacker can get data into the system. In this case, you not only can’t trust the output, but depending on the system, they may be able to exfiltrate sensitive data.
Rethinking RCE
There have been instances where people have gotten generative AI-based tools to execute code on their behalf, creating remote code execution vulnerabilities (RCE), some of the most devastating vulnerabilities we have. These issues will no doubt continue to be a problem. However, since generative AI tools are themselves generalized, we may need to start thinking about the LLM portions of our applications as yet another “operating system” or execution environment we need to protect.
In a way, an attacker tricks the system into executing their input rather than the behavior expected by the developers. Although an attacker’s input may not be shoved into a Python exec() statement, they’ve still manipulated the system to execute their input, affecting the application’s execution and resulting output.
Overcomplicating Guidance
We security professionals love to overcomplicate things, and our guidance and recommendations are no exception. I once worked at a company where someone created this massive flow chart for peer reviews that basically stated that when you were done with your report, you should send it to your manager, and they will send it back to you. The old adage that complexity is the enemy of security has always contained a valuable theme that gets sacrificed on the pyre of complexity’s perceived beauty.
I will continue saying that much of AI security is application and product security. These are things we already know how to do. I mean, it’s not like generative AI came along and suddenly made permissions irrelevant. Permissions are actually more important now. But this isn’t satisfying for people who want to play the role of wise sage in the AI age. The guidance and controls of the past aren’t less valuable but more valuable in the age of generative AI and agents.
We’ll see the manufacture of new names for vulnerabilities with increasingly complex guidance and high-fives all around. The secret is these will mostly be variations on the same themes we’ve already seen, such as manipulation, authorization, and leakage flaws.
Back in May of 2023, I created Refrain, Restrict, and Trap (RRT), a simple method for mitigating LLM risks while performing design and threat modeling. It still holds up as a starting point and applies to agents as well. Simple just works sometimes.
Continue To Be Owned
These applications, including ones launched as agents, will continue to be owned. Owned, for those not familiar with security vernacular, means compromised. I made this prediction in the Lakera AI Security Year in Review: Key Learnings, Challenges, and Predictions for 2025 in December. I’m fully confident this trend will continue.
I mentioned that the issues haven’t been fixed, and now people are increasing deployments and giving them more autonomy with far more access to data and environments. This results in far worse consequences when a compromise occurs. To make matters worse, we’ll begin to see organizations deploy these systems in use cases where the cost of failure is high, creating more impact from failures and compromises.
Failures and Poor Performance
These implementations will continue to fail where LLM-based use cases fail, but potentially worse. For example, it’s easy to see how increasing complexity can cause a lack of visibility with potential cascading failures. In 2025, organizations will likely continue dipping their toe into the waters of high-risk use cases where the cost of failure is high, as mentioned previously.
Sure, a car dealership chatbot offering to sell a truck for one dollar is funny, but it has no real impact. However, high-risk and safety-critical use cases have a large financial impact or possibly cause harm or loss of human life. You may roll your eyes and say that would never happen, but what happens in a more simple use case when OpenAI’s Whisper API hallucinates content into someone’s medical record? Because that’s already happening.
Due to their lack of visibility and minimized human control, AI agents can mimic grenades when deployed in high-risk use cases, where the damage doesn’t happen the moment you pull the pin. This complicates things as it means that issues may not shake out during experimentation, prototypes, or even initial usage.
Agents can mimic grenades when deployed in high-risk use cases, where the damage doesn’t happen the moment you pull the pin.
Generative AI is still an experimental technology. We haven’t worked out or discovered all of the issues yet, leading to another example I’ve used as a warning in my presentations over the past couple of years: AlphaGo beating Lee Sedol at Go. Many have heard of this accomplishment, but what many haven’t heard is that even average Go players can now beat superhuman Go AIs with adversarial policy attacks. We may be stuck with vulnerable technology in critical systems. Sure, these are different architectures, but this is a cautionary tale that should be considered before deploying any experimental technology.
Beyond failures and compromises, we adopt architectures that work but don’t work as well as more traditional approaches. In our quest to make difficult things easy, we make easy things difficult. Welcome to the brave new world of degraded performance.
Success and Good Enough
For the past few years, I’ve been pushing back against the famous phrase, “AI won’t replace people. People with AI will replace people without.” This is complete nonsense. I have an upcoming blog post about this where I “delve” into the topic. The reality is the opposite. The moment an AI tool is mediocre enough to pass muster with a reasonable enough cost, people will be replaced, AI use or not. This is already being planned.
The moment an AI tool is mediocre enough to pass muster with a reasonable enough cost, people will be replaced, AI use or not.
Like most technology, agents will have some limited success. And that success will be trumpeted in 2025 as the most earth-shattering innovation of ALL TIME! I can hear it now. “You just wait bro, in 2025 agents are going to the moon!” Maybe. But, given the environment and the fact that issues with LLMs haven’t been solved, an LLM-powered rocket to the moon isn’t one I’d consider safe. Passengers may very well find themselves on a trip to the sun. The future is bright, very bright. 🕶️
How much success agents have in 2025 and what impact this success has remains to be seen. At this point, it’s far from obvious, but I won’t be surprised by their successes in some cases or their spectacular failure in others. This is the reality when the path is shrouded in a dense fog of hype.
Things to look for in successes would be use cases with limited exposure to external input, low cost of failure, and cases where inputs and situations require adapting to change. The use case will also need to tolerate the lack of visibility and explainability of these systems. There will also be continuing success in use cases where tools can be used.
The idea of a multi-agent approach to solving complex problems isn’t a bad one, especially when unknowns enter the equation. Breaking down specific tasks for agents so that they’re focused on these tasks as part of a larger architecture is a solid strategy. However, the current and unsolved issues with generative AI make this approach fraught with risk. In the future, more robust systems will most likely exploit this concept for additional success.
Cybersecurity Use Cases and Penetration Testing
There’s certainly the possibility of disruption in cybersecurity. Before the generative AI boom, I joked with someone at Black Hat that if someone created a product based on reinforcement learning with offensive agents that were just mediocre enough, they’d completely wipe out pen testing.
For years, people have discussed how penetration testing work has become commoditized, and there is a race to the bottom. I don’t think that has happened to the extent many predicted, but we could see a shift from commoditization to productization.
Pen testing also seems to check the boxes I mentioned previously.
Low cost of failure
Varying quality
Value misalignment
Tool use
Adaptation to unknowns
Pen testing is an activity with a low cost of failure. The failure is missing a vulnerability, which is something humans also do. This scenario is hardly the end of the world. Yes, an attacker could indeed find the vulnerability and exploit it to create damage, but it depends on various factors, including exposure, severity, and context.
The quality of pen tests is often all over the map and highly dependent on the people performing the work. Human experts at the top of their game will continue to crush AI-powered penetration testing tools for quite some time. However, most organizations don’t hire experts, even when they hire third parties to perform the work. The value of such a tool in this environment becomes far more attractive, potentially enough to postpone a hire or discontinue using a third party for penetration testing needs (if regulations allow.)
The value of pen testing isn’t always aligned with the need. Many customers don’t care about pen testing. They are doing it because it’s required by some standard, policy, compliance, or possibly even simply because they’ve always done it. Pen testing is one of those things where if customers could push a button and have it done without a human, they’d be okay with that. Pushing a button is the spirit animal of the checkbox. After all, the goal of pen testing is not to find anything. You certainly have due diligence customers and people who truly value security, but the number of checkbox checkers far outweighs these folks.
Human pen testers use tools to perform their jobs. Tool use has shown promise and some success with LLMs at performing certain security-related tasks. This is yet another indicator that a disruption could be on the horizon.
Every environment and situation is different for pen testers. You are given some contextual information along with some rules and are turned loose on the environment. This is why humans are far more successful than vulnerability scanners at this task, much to the chagrin of product vendors. However, adapting to some of these unknowns may be something generative AI agents can adapt to at a reasonably acceptable level. We’ll have to see.
Given what I outlined, you may believe that generative AI tools give attackers an advantage over defenders, but this isn’t the case. The benefits of AI tools, generative AI or otherwise, align far more with defender activities and tasks than with attacker activities. This will remain true despite any apparent ebb and flow.
New Year’s Resolution
It’s the time of year when people make resolutions, so how about this? 2025 has already launched with the firehose fully open, blasting us directly in the face with 150 bsi (Bullshit per Square Inch) of pure, unadulterated hype.
We are only a few days into the year, and it seems as though the religion of AI is far exceeding reality. Hype is what’s going on. It’s that simple. It’s 2025. Let’s make it the year to add at least “some” skepticism, not believing every claim or demo as though it’s the gospel according to Altman.
Sam Altman isn’t a prophet. He’s a salesman. In any other situation, he’d be cluttering up your LinkedIn inbox and paying a data broker to get your work email address and phone number. “Look, I know I’ve called six times, but I really think our next-generation solution can skyrocket your profits. I’m willing to give you a hundred-dollar Amazon gift card just for a demo!”
Sam Altman claims that OpenAI knows how to build AGI, and we’ll see it in 2025, triggering the predictable responses from useful idiots. Remember, these things are performance art for investors, not useful information for us. If we had any attention span left, we’d remember him as the little boy who cried AGI.
Let’s analyze this paragraph, which is the one that’s sending generative AI to the moon on social media. It consists of three sentences that have nothing to do with each other, but since the shockwave of hype pulverizes our minds, we glue them together.
We are now confident we know how to build AGI as we have traditionally understood it.
That’s not true. Once again, this is performance art for investors. A possibility is that they redefine AGI to align with whatever goalposts they set and pat their own backs at the end of 2025.
We believe that, in 2025, we may see the first AI agents “join the workforce” and materially change the output of companies.
Okay, but what does this have to do with AGI? You see, this is sleight of hand. He wants you to believe this is connected to the previous point about AGI. It is not. This doesn’t require AGI to be true. If there is some success here, people can point to this as proof of some proto-AGI, which won’t be the case.
We continue to believe that iteratively putting great tools in the hands of people leads to great, broadly-distributed outcomes.
HAHAHAHA. What? Did he write that, or did ChatGPT? It is also not related to AGI. Great, broadly-distributed outcomes, but not for most people on the planet. The goal is workforce reduction, broadly distributed workforce reductions. Although it’s true that some high school kid may indeed invent the next big thing, creating a multi-million dollar company, for every one of these, there will be countless droves of people displaced from the workforce, quite often, with nowhere to go. Or, at least, this is the goal. We can be honest about these things without delusions, but this brings its own challenges.
Okay, I’m having a bit of fun with Sam Altman’s nonsense, but some of this isn’t his fault. He can’t be completely honest with people, either, due to the uncomfortable situation of cheerleading technology claiming to remove people’s autonomy and sometimes their purpose. If people can’t work, they can’t support their families. I’ve written about the backlash against AI-powered tech in the past and its consequences. AI hype is putting all of humanity on notice, and humanity notices. Backlash plays a large part in why there is a lack of honesty.
AGI will happen. We should acknowledge this fact, and living in denial about it isn’t a strategy for the future. However, it won’t be OpenAI who creates it in 2025. If I had to place a bet today on who would actually create AGI, I’d bet on Google DeepMind. DeepMind is a serious organization that continues to impress with its research and accomplishments, quite often making the competition look silly. But then again, those are just my “vibes.”
Let me make this clear. My criticism of Altman, or any company’s strategy, marketing, or ludicrous levels of hype, has nothing to do with the hard-working people who work there or their accomplishments. I know some of these people. They aren’t fools by any stretch. But, their work is tarnished when every time Altman makes a claim, like believing that angels are in the optimizer.
We know that every AI demo and usage scenario runs into the complexities of the real world under normal conditions. Yet, we seem to forget this lesson every time a demo or claim is made. 2025 is going to bring more stunts, more claims, and more demos. We should experiment in our own environments, with our own data, to apply what works best for us and aligns with our risk tolerance. Don’t believe everything you see on the internet.
If we are not careful, we are about to enter an era of software development, where we replace known, reliable methods with less reliable probabilistic ones. Where methods such as prompting a model, even with context, can still lead to fragility causing unexpected and unreliable outputs. Where lack of visibility means you never really know why you receive the results you receive, and making requests over and over again becomes the norm. If we continue down this path, we are headed into a brave new world of degraded performance.
Scope
Before we begin, let’s set the perspective for this post. The generative AI I’m covering in this post is related to Large Language Models (LLMs) and not other types of generative AI. This post focuses on building software meant to be consumed by others. Products and applications deployed throughout an organization or to delivered to customers. I’m not referring to experiments, one-off tools, or prototypes. Although, buggy prototype code can have an odd habit of showing up in production because a function or feature just worked.
This post isn’t about AI destroying the world or people dying. It’s about the regular applications we use, even in a mundane context, just not being as good. The cost of failure doesn’t have to be high for the points in this post to apply. I’m saying this because, in many cases, the cost may be low. People probably won’t die if your ad-laden personalized horoscope application fails occasionally. But that doesn’t mean users won’t notice, and there won’t be impacts.
Our modern world runs on software, and we are training people that buggy software should be expected.
Our modern world runs on software, and we are training people that buggy software should be expected, and making requests repeatedly is the norm, setting the expectation that this is just the price paid in modern software development. This approach is bad, and the velocity at all costs mantra is misguided.
Let me be clear because I’m sure this will come up. I’m not anti-AI or anti-LLM or anything of the sort. These tools have their uses and can be incredibly beneficial in certain use cases. There are also some promising areas, such as the ability of LLMs to, generate, read and understand code and what that means for software development in the coming years. It’s still early. So in no way am I claiming that LLMs are useless. I’m trying to address the hype, staying in the realm of reality and not fantasy. The truth today is that maximizing these tools for functionality instead of being choosy is the problem and there are costs associated.
Software Development
Software development has never been perfect. It’s always been peppered with foot guns and other gotchas, be it performance or security issues, but what it lacked elegance, it made up in visibility and predictability. Developers had a level of proficiency with the code they wrote and an understanding of how the various components worked together to create a cohesive service, but this is changing.
Now, you can make a bunch of requests to a large language model and let it figure it out for you. No need to write the logic, perform data transformations, or format the output. You can have a conversation with your application before having it do something and assume the application understands when it gives you the output. What a time to be alive!
There’s no doubt that tools like ChatGPT increased accessibility to people who’ve never written code before. Mountains of people are creating content showing, “Look, Mom, I wrote some code,” bragging that they didn’t know what they were doing. I’ve seen videos of University Professors making the same claims. This has and will continue to lead to many misunderstandings about problems people are trying to solve and the data they are trying to analyze. Lack of domain expertise and lack of functional knowledge about how systems work is a major problem but not the focus of this post.
As a security professional, inexperienced people spreading buggy code makes me cringe (look at the Web3 space for examples), but It’s not all bad. In some ways, this accessibility is a benefit and may lead to people discovering new careers and gaining new opportunities. Also, small experiments, exploration, or playing around with the tools are absolutely fine. It’s how you discover new things. However, inefficiencies, errors, and lack of reliability aren’t dealbreakers in these cases. But what happens when this mindset is taken to heart and industrialized into applications and products that impact business processes and customers?
Degraded Performance
There’s a new approach in town. You no longer have to collect data, ensure it’s labeled properly, train a model, perform evaluations, and repeat. Now, in hours, you can throw both apps and caution to the wind as you deploy into production!
This above is a process outlined by Andrew Ng in his newsletter and parroted by countless content creators and AI hustle bros. It’s the kind of message you’d expect to resonate, I mean, who wouldn’t like to save months with the added benefit of removing a whole mountain of effort in the process? But, as with crypto bros and their Lambos, if it sounds too good to be true, it probably is.
Let’s look at a few facts. Compared to more traditional approaches:
LLMs are slow
LLMs are inefficient
LLMs are expensive ($)
LLMs have reliability issues
LLMs are finicky
LLMs can and do change (Instability)
LLMs lack visibility
Benchmarking? Measuring performance?
Pump the Brakes
Traditional machine learning approaches can have much better visibility into the entire end-to-end process. This visibility can even include how a decision or prediction was made. They can also be better approaches for specific problems in particular domains. These approaches also make it far easier to benchmark, create ensembles, perform cross-validation, and measure performance and accuracy. Everyone hates data wrangling, but you learn something about your data, given all that wrangling. This familiarity helps you identify when things aren’t right. Having visibility into the entire process means you can also identify potential issues like target leakage or when a model might give you the right answer but for the wrong reasons, helping avoid a catastrophe down the road.
The friction in more traditional machine learning is a feature, not a bug, making it much easier to spot potential issues and create more reliable systems.
The friction in more traditional machine learning is a feature, not a bug
Lazy Engineering
On the surface, letting an LLM figure everything out may seem easier. After all, Andrew Ng claims something similar. In his first course on Deeplearning.ai ChatGPT Prompt Engineering for Developers He mentions using LLMs to format your data as well as using triple backticks to avoid prompt injection attacks. Even the popular LangChain library instructs the LLM to format data in the same way. Countless others are creating similar tutorials flooding the web parroting this point. Andrew is a highly influential person who’s helped countless people with this training by making machine learning more accessible. With so many people telling others what they want to hear, as well as the accessibility of tools like LangChain, this will have an impact, and it’s not all positive.
One of the goals of software engineering should be to minimize the number of potential issues and unexpected behaviors an application exhibits when deployed in a production environment. Treating LLMs as some sort of all-capable oracle is a good way to get into trouble. This is for two primary reasons, lack of visibility and reliability.
Black Boxes
A big criticism of deep learning approaches has been their lack of transparency and visibility. Many tools have been developed to try and add some visibility to these approaches, but when maximized in an application, LLMs are a step backward. A major step backward if you count things like OpenAI’s Code Interpreter.
The more of your application’s functionality you outsource to an LLM, the less visibility you have into the process. This can make tracking down issues in your applications when they occur almost impossible. And when you can track problems down, assuming you can fix them, there will be no guarantee that they stay fixed. Squashing bugs in LLM-powered applications isn’t as simple as patching some buggy code.
Right, Probably
LLMs are being touted as a way to take on more and more functionality in the software being built, giving them an outsized role in an application’s architecture. Any time you replace a more reliable deterministic method with a probabilistic one, you may get the right answer much of the time, but there’s no guarantee you will. This means you could have intermittent failures that impact your application. In more extreme cases, these failures can cascade through a system affecting the functionality of other downstream components.
For example, anyone who has ever asked an LLM to return a single-word result will know that sometimes it doesn’t, and there’s no rhyme or reason why. It’s one of the classic blunders of LLMs.
So, you may construct a prompt stating only to return a single word, True or False, based on some request. Occasionally, without warning and even with the temperature set to 0, it will return something like the following:
The result is True
Not the end of the world, but now translate this seemingly insignificant quirk into something more impactful. Your application expected a result from an LLM formatted in a certain way. Let’s say you wanted the result formatted in JSON. Now, your application receives a result that isn’t JSON or maybe not properly formatted JSON, creating an unexpected condition in your application.
Suppose we combine this reliability issue with the lack of visibility. In that case, it can lead to some serious issues that may be intermittent, hard to troubleshoot, and almost impossible to fix without reengineering. In a more complex example, maybe you’ve sent a bunch of data to an LLM and asked it to perform a series of actions, some including math or counting, and return a result in a particular format. A whole mess of potential problems could result from this, all of which are outside your control and visibility.
Not to mention a big point many gloss over, deploying your application in production isn’t the end of your development journey. It may be the beginning. This means you will need to perform maintenance, troubleshooting, and improvements over time. All things LLMs can make much more difficult when functionality is maximized.
To summarize, outsourcing more and more application functionality to an LLM means that your application becomes less modular and more prone to unexpected errors and failures. These are issues that Matthew Honnibal also covers in his great article titled Against LLM Maximalism.
The Slow and Inefficient Slide
In some use cases, it may not matter if it takes seconds to return a result, but for many, this is unacceptable. Having multiple round trips and sending the same data back and forth may be necessary due to different use cases because a character changed or because of context window size, which also adds to the inefficiency. Even if the use case isn’t critical and inefficiencies can be tolerated, that’s not the end of the story.
There are still environmental impacts due to this inefficiency. It requires much more energy consumption to have an LLM perform tasks than more traditional methods. For example, searching for a condition with a RegEx vs. sending large chunks of data to an LLM and letting the LLM try and figure it out. The people ranting and raving constantly about the environmental impacts of PoW cryptocurrency mining are incredibly silent on the energy consumption of AI, even as former crypto miners turn their rigs toward AI. Think about that next time you want to replace a method like grep with ChatGPT or generate a continuous stream of cat photos with pizzas on their head.
LLMs Change and So Do You
Any check of social media will show that at the time of this writing, there have been quite a few people claiming that GPT-4 is getting worse. There’s also a paper that explores this.
There’s some debate over the paper and some of the tests chosen, but for the context we are discussing in this post, the why an LLM might change isn’t relevant. Whether changes are because of cost savings, issues with fine-tuning, upgrades, or some other factor aren’t relevant when you count on these technologies inside your application. This means your application’s performance can worsen for the same problems, and there isn’t much you can do about it but hope if you are consuming a provider’s model (OpenAI, Google, Microsoft, etc.) This can also lead to instability due to the provider requiring an upgrade to a newer version of the hosted model, which may lead to degraded performance in your application.
Demo Extrapolation
The problem is that none of the constraints and issues may surface for demos and cherry-picked examples. Actually, the results can look positive. Positive results in demos are a danger in and of themselves since this apparent working can mask larger issues in real-world scenarios. The world is filled with edge cases, and you may be running up a whole bunch of technical debt.
Hypetomisim and Sunken Cost
There’s a sense that technology and approaches always get better. Whether this is from Sci-fi movies or just because people get a new iPhone every year, maybe a combination of both. Approaches can be highly problem or domain-specific and not generalize to other problem areas or at least not generalize well. We don’t have an all-powerful single AI approach to everything. Almost nobody today would allow an LLM to drive their car. However, some have hooked them up to their bank accounts. Yikes!
But you can detect an underlying sense of give it time in people’s discussions on this topic. Whenever you point out issues you usually get, well GPT-5 is gonna… This goes without saying that ChatGPT is based on a large language model, and large language models are trained on what people write, not even what they actually think in certain cases. They perform best on generative tasks. On the other hand, tasks like operating a car have nothing to do with language. Sure, you could tell the car a destination, but every other operation has nothing to do with language. It’s true that LLMs can also generate code, but do you want your car to generate and compile code while driving it? Let me answer that. Hell no. Heed my words, maybe not this use case, but something in the same order of stupid is coming.
Developing buggy software in the hopes that improvements are on the way and outside your control is not a great strategy for reliable software development.
Developing buggy software in the hopes that improvements are on the way and outside your control is not a great strategy for reliable software development. I’ve heard multiple stories from dev teams that they continue to run buggy code with LLM functionality and make excuses for apparent failures because of sunken costs.
The hype has led to a new form of software development that appears to be more like casting a spell than developing software. The AI hustle bros want you to believe everything is so simple and money is just around the corner.
Now’s a good time to remind everyone that fantasy sells far better than reality. Lord of the Rings will always sell more books than one titled Eat Your Vegetables. Trust me, as most of my posts are along the lines of Eat Your Vegetables posts, I make no illusions that every AI hustler’s Substack making nonsensical and unfounded predictions is absolutely crushing me in page views.
Engineering Amnesia
In a development context, we may forget that better methods exist or allow ourselves to reintroduce known issues that cause cascading failures and catastrophic impacts on our applications. This isn’t without precedent.
The LAND attack came back in Windows XP after it was known and already mitigated in previous Windows OSs. ChatGPT plugins are allowed to execute in the context of each other’s current domains, even though we’ve seen time and time again how this violates security. The Corrupted Blood episode was a failure to understand how the containment of a feature could cause catastrophic damage to an application, so much so that it forced a reset. And, of course, don’t even get me started on the Web3 space. I mean, who wouldn’t want tons of newly minted developers creating high-risk financial products without knowledge of known security issues? It was fascinating to see security issues in high-impact products for which standard, boring, and known security controls would have prevented them. These are just a couple off the top of my head, and there are many more.
As new developers learn to use LLMs to perform common tasks for which we have better, more reliable methods, they may never become aware of these methods because their method just kind of works.
Avoiding Issues
The perplexing part of all of this is that these issues are pretty easy to avoid, mainly by thinking carefully about your application’s architecture and the features and components you are building. Let me also state that these issues won’t be solved by writing better prompts.
Reliability and visibility issues won’t be solved by writing better prompts
There’s the perception that using an LLM to figure everything out is easier than other methods. On the surface, it may appear that there’s some truth to that. It’s also easier to spend money on a credit card than to make the money to pay the bill. So, it’s the case that you may be kicking the can down the road. Avoiding these issues isn’t hard, and a bit of thought about your application and its features will go a long way.
Look at your application’s features. Break these features down into functional modules. The goal of breaking down these features into smaller components is to evaluate the intended functionality to determine the best approach for the given feature. At a high level, you could ask a few questions with the goal of determining the right tool for the processing task.
Does the function require a generative approach?
Are there existing, more reliable methods to solve the problem?
How was the problem solved before generative AI? (Potential focusing question if necessary)
Is there a specific right or wrong answer to the problem?
What happens if the component fails?
These questions are far from all-encompassing, but they are meant to be simple and provide some focus on individual component functionality and the use case. After all, LLMs are a form of generative AI, and therefore, they are best suited to generative tasks. Asking if there’s a specific right or wrong answer is meant to focus on the output of the function and consider if a supervised learning approach may be a better fit for the problem.
We have reliable ways of formatting data, so it’s perplexing to see people using LLMs to perform data formatting and transformations, especially since you’ll have to perform those transformations every time you call the LLM. Asking these questions can help avoid issues where improperly formatted data can cause a cascading issue.
Example
Let’s take a simple example. You want a system that parses a stream of text content looking for mentions of your company. If your company is mentioned, you want to evaluate the sentiment around the mention of your company. Based on that sentiment, you’d like to write some text addressing the comment and post that back to the system. We break this down into the following tasks below.
For parsing, analysis, and text generation steps, it would be tempting to collapse all of them together and send them to an LLM for processing and output. This would be maximizing the LLM functionality in your application. You could technically construct a prompt with context to try and perform these three activities in a single shot. That would look like the following example.
In this case, you have multiple points of failure that could easily be avoided. You’d also be sending a lot of potentially unnecessary data to the LLM in the parsing stage since all data, regardless of whether the company was mentioned, would be sent to the LLM. This can substantially increase costs and increase network traffic, assuming this was a hosted LLM.
You are also counting on the LLM to parse the content given properly, then properly analyze and then, based on the two previous steps, properly generate the output. All of these functions happen outside of your visibility, and when failures happen, they can be impossible to troubleshoot.
So, let’s apply the questions mentioned in the post to this functionality.
Parsing
Does the function require a generative approach? No
Are there existing, more reliable methods to solve the problem? Yes, more traditional NLP tools or even simple search features
Is there a specific right or wrong answer to the problem? Yes, we want to know for sure that our company is mentioned.
What happens if the component fails? In the current LLM use case, the failure feeds into the following components outside the visibility of the developer, and there’s no way to troubleshoot this condition reliably.
Analysis
Does the function require a generative approach? No
Are there existing, more reliable methods to solve the problem? Yes, more traditional and mature NLP tasks for sentiment analysis
Is there a specific right or wrong answer to the problem? Yes
What happens if the component fails? In the current LLM use case, the failure feeds into the following text generation component outside the developer’s visibility, and there’s no way to troubleshoot this condition reliably.
Text Generation
Does the function require a generative approach? Yes
Are there existing, more reliable methods to solve the problem? LLMs appear to be the best solution for this functionality.
Is there a specific right or wrong answer to the problem? No, since many different texts could satisfy the problem
What happens if the component fails? We get text output that we don’t like. However, since the previous steps happen beyond the developer’s visibility, there’s no way to troubleshoot failures reliably.
Revised Example
After asking a few simple questions, we ended up with a revised use case. This one uses the LLM functionality for the problem it’s best suited for.
In this use case, only the text generation phase uses an LLM. Only confirmed mentions of the company, along with the sentiment and the content necessary to write the comment, are sent to the LLM. Much less data flows to the LLM, lowering cost and overhead. By using more robust methods, much less can go wrong as well, and less likely to have cascading failures affecting downstream functions. When something does go wrong in the parsing or analysis stages, troubleshooting is much easier since you have more visibility into those functions. So, breaking down this functionality in such a way means that failures can be more easily isolated and addressed, and you can improve more reliably as the application matures.
Now, I’m not claiming that this is a development utopia. A lot can still go wrong, but it’s a far more consistent and reliable approach than the previous example.
After talking with developers about this, some of the questions I’ve received are along the lines of, “There are better methods for my task, so if we can’t cut corners, then why use an LLM at all?” Yes, that’s a good question, a very good question, and maybe you should reevaluate your choices. This is my surprised robot face when I hear that.
LLMs Aren’t Useless
Once again, I’m not saying that LLMs are useless or that you shouldn’t use them. LLMs fit specific use cases and classes of functionality that applications can take advantage of. For many tasks, there’s the right tool for the job or at least a righter tool for the job. However, this right tool for the right job approach isn’t what’s being proposed in countless online forums and tutorials. I’m concerned with a growing movement of using LLMs as some general-purpose application functionality for tasks that we already have much more reliable ways of performing.
Conclusion
Will we inhabit a sprawling landscape of digital decay where everything rests on crumbling foundations? Probably not. But there will be a noticeable shift in the applications we use on a daily basis. But it doesn’t have to be. By being choosy and analyzing functionality where LLMs are best suited, you can make more reliable and robust applications, and the environment will also thank you.
I wanted to start my refreshed blog with a post on Deepfakes, but probably not highlighting the threat you expect. For the past couple of years, I’ve said the real threat from Deepfakes is different from the one discussed most of the time. There’s a lot of handwaving and hype focused on one specific threat, but this can create a distraction from some profound and lasting issues. Let’s look at a couple of other threats posed by DeepFakes and examine why these have a more lasting impact.
Narrative Evidence
When you think of the danger from Deepfakes, you are probably thinking about their ability to convince people something happened that didn’t. This threat is something I call narrative evidence because you are using the content in an attempt to show evidence in support of some larger story. It’s this issue that steals all of the oxygen on the topic. The threat’s stated impact is that it tears at the fabric of reality, and people will believe things because they see and hear it. Although this impact isn’t false, it doesn’t take into account certain actualities.
The fabric of reality is already torn. If anything proves this, it should be the events of 2020. We’ve seen people burn down 5G towers and believe that a major company was shipping children in their furniture. At this moment in the United States, millions of people believe something happened that didn’t with no evidence and no proof. These falsehoods are all perpetuated without the benefit of Deepfakes.
Let’s consider an example In 2019, there was an altered video of Speaker Nancy Pelosi making the rounds on social media. This video was slowed, making her seem as though she was slurring her speech and intoxicated. No high-tech tools were used. Now, how would a Deepfake have changed this? The reality is, it probably would have made little difference. People who wanted it to be true would share it, while others would not.
Thankfully, the creators of fake content are rarely subtle. Someone generating content for Speaker Pelosi would have her saying something about how she enjoys the nourishing effects of child blood or something equally as ridiculous. This ridiculousness is an indicator of future use. In the future, Deepfakes won’t be a tool used to convince people an event happened but instead used to excite a particular group’s existing biases, in much the same way fake content and memes do today. This is because provenance and reality don’t matter in this context.
In the future, Deepfakes won’t be a tool used to convince people an event happened but instead used to excite a particular group’s existing biases.
As resources become more available and tools get easier to use, Deepfakes technology will remove the friction in creating fake content, but this also has a downside for its purveyors. Increased availability and simplification will generate a deluge of fake content, but this increase will normalize the content and make people tune it out. So, this fake content won’t be a tool to expand a particular viewpoint to new people, mostly keep the current crop engaged. While the technology catches up, there is a good bet we’ll see an expansion of services offering Deepfakes as a Service (DFaaS).
The fact of the matter is, we underestimate people’s biases when they evaluate content, and people have gotten pretty good at pwning themselves.
Deepfakes in Attacks
What about the Deepfakes used in attacks? It’s true, there are a couple of instances of Deepfakes being used in attacks, but these are exceptions and not the rule. In general, humans aren’t good at envisioning threats that haven’t happened yet, but once they happen, they do adapt. This adaptation will be the same for these attacks. The success of these attacks only work while the novelty is high, and the novelty wears off quickly.
The success of these attacks only work while the novelty is high.
What About Evidence of a Crime?
I mentioned the word evidence, so what about Deepfakes being used in a court of law? It’s unlikely that this would become a real issue in criminal court. It’s unlikely because there’s usually not a single piece of evidence in a case, so corroborating details wouldn’t exist. Also, techniques are getting better to detect manipulations that wouldn’t survive the scrutiny faced in a court of law. Is it impossible? Certainly not depending on the situation, but it is doubtful that this would become some widespread issue.
Still a Threat?
In the short term, narrative evidence attacks still pose a threat and are something we should be conscious of, so I’m not suggesting we write this threat off. The novelty value is still relatively high. However, I consider narrative evidence attacks more of a short term threat and won’t be the most impactful and long-lasting effect of Deepfakes. In short, the risk is overhyped, not non-existent, and my goal is to get people to focus on some of the more long-lasting problems.
Lasting Problems
There are several threats from DeepFakes, but the two of the most lasting and impactful fall under the following categories:
Reality Denial
Harassment
Reality denial is the opposite of the threat most people claim. The mere existence of Deepfakes is enough for people to question legitimate content. Anytime someone sees evidence of something they don’t like, they can just claim it’s a DeepFake. This situation can have massive ripple effects. I mean, how do you get a fair trial by a jury if the jury is willing to mentally throw out legitimate evidence?
Weaponizing backlash against legitimate content is also much easier to engineer because it takes no effort at all. All of this conducted with no technology, no constructions, and no time. The impact is everyone from friends to nation states can merely raise the question of the content’s provenance, and for many who are biased in that direction, it will be enough. This is the threat that should scare people, but it’s not the only threat. There’s another that can affect you personally.
Harassment
Deepfakes have the ability to cause harm in instances where provenance and reality aren’t important. Here’s a question to ponder, does it matter whether the fake nudes of you shared online are real or fake? Deepfakes have the ability to take bullying and harassment to the next level since you can steal someone’s likeness and put them in all manner of situations. These situations include pictures, audio, and video. In most cases, it doesn’t matter whether the content is real or not. The impact is the same.
In October of 2020, I reviewed and provided feedback on a report before publication on Automating Image Abuse. The report detailed a Telegram channel where you could strip the clothes off of individuals. The original incarnation of this software was called DeepNude, and that term has stuck to all manner of technology concerning the removal of clothing.
Harassment will be the real legacy of Deepfakes.
Harassment will be the real legacy of Deepfakes. Consider how ease of use and availability of tools makes harassment and bullying much easier. In the near future, anyone who wants to generate this kind of content will have an outlet for doing so.
This is an area where the legal system can help, and we are starting to see some anti-Deepfake laws, but unfortunately, they are focusing on issues of narrative evidence and not harassment. This issue is something I think will change over the next few years, but the legal system moves slowly. Online platforms and social media companies can help as well, by building tools and punishing users spreading harmful content. Unfortunately, short of legal assistance and cooperation of social media companies, harassment may be one of those cultural issues we have to learn to live with for quite some time.
The Entertainment Industry
The entertainment industry is who should be worried about the technology powering Deepfakes. The disruption caused will be particularly impactful to actors and actresses, meaning they may be out of a job in the future. It would be a mistake to think that the generated content of the future will resemble the CGI of the past.
As an example, the creators of South Park made a Deepfakes television show called Sassy Justice and can be viewed on YouTube. The show features a cast of celebrities (all fake) and, like most things the South Park creators do, is entertaining and educational, performed in an over the top fashion.
In the future, availability and advancements will make it easier for regular people to generate their own worlds, people, monsters, etc. It may very well be that in the not too distant future, people are begging you to watch their feature film like a lot of artists do about their songs today. So it’s not all doom and gloom, depending on your perspective.
In a post-Covid19 world where social distancing and other environmental concerns impact real film shoots, a generated alternative could prove lucrative and allow movie studios and amateurs alike to increase the content.
Conclusion
Genies rarely fit back into bottles, and we need to come to grips with the fact that the technology is here to stay. Focusing only on the narrative evidence aspect of Deepfakes takes attention away from the long-lasting threats. This lack of awareness is apparent in the anti-Deepfakes laws being drafted. We need to make sure we highlight the other threats, such as harassment, so they get more attention from lawmakers and social media companies.
