People making predictions fall into three general camps: those selling something, delusional ignoramuses, and the rare case of thoughtful reflectors. I’d like to think I fall into the last category, but since I’m not selling anything, I fear I may be part of the former. Regardless of category, we seem to forget that the world confounds prediction through complexity, even for the most ardent of reflectors.
Another common playbook in our era is to make so many predictions that some are bound to come true, then cite those cases as proof that you are an oracle. I see this happening frequently. It’s the exploitation of our short attention spans. This isn’t magical foresight, it’s statistics.
Regardless of my opinion on tech predictions, people seem to love hearing them. While I was at the AI Security Summit in London, several people asked me for my predictions for 2026, since in my keynote, I described hype shifting back to embodied systems. I guess I asked for it. But, please don’t listen to me or anyone else making predictions about 2026. Well, at least I’m not trying to sell you anything.
I think people have an instinct that 2026 feels more uncertain than 2025. There is a sense of desperation in the air as companies push to prove there is no AI bubble by wallpapering everything with AI.
Now that I’ve complained about making predictions and how uncertain 2026 feels, here are my predictions/vibes/observations for 2026.
1. Agent Double Down
“No, no. Last year wasn’t the year of the agent. THIS year is going to be the year of the agent.” I can already hear people course-correcting from their predictions last year. 2025 was the year generative AI was going to take off, resulting in massive layoffs and tons of revenue. Instead, we hear speculation about the AI bubble about to pop.
Despite the ongoing issues and high manipulability of agents, people will continue to double down. We didn’t resolve any issues with agents in 2025, so they’ll be with us again in 2026. But with the doubling-down efforts, people will try to convince you that the issues are solved or didn’t matter much in the first place.
Most business leaders who ask for agents and insist on using AI have no idea how the technology works, what it’s capable of, or the associated risks. This is not a recipe for success. Deploying this technology successfully requires a firm understanding of capabilities and realities on the ground. Of course, having appropriate expectations helps too. This isn’t happening, as MIT found when they identified that 95% of GenAI pilots failed.
I’m not claiming that agents are useless. They have their uses and can be employed in certain scenarios to augment human activities. And yes, this can be done successfully. What I’m saying is, they aren’t the utopian, headcount-reducing technology we were promised in 2025, and the data bears this out.
The truth is, if your use case has a low cost of failure and can tolerate errors and manipulations, you don’t need to wait for a new innovation. You can deploy agents today. How well they perform, on the other hand, is a different story. Performance will vary by use case and environment.
2. Embodiment Hypes Again
Although the hype of generative AI will continue in 2026, we’ll see much more hype of embodied systems. Embodied systems are those that interact and learn from the real world. Think robots, self-driving cars, drones, etc. This category is certainly no stranger to hype.
Embodied systems are always ripe for hype because they tend to be more tangible and less behind-the-scenes. There will undoubtedly be some real improvements in this area. Unfortunately, these real improvements will provide ammunition for the hype cannon. Any modest improvement will be pointed to as exponential. For example, Elon Musk recently said robots wouldn’t just end poverty, but also make everyone rich. Utopian abundance is often talked about but never rationally explained.
3. Security Issues Continue To Rise
Security issues will not only persist but also accelerate. How can they not? With more AI writing more code and more code being pushed by inexperienced people, that’s a recipe for security issues. But to quote the late American philosopher Billy Mays, “But wait, there’s more!” As more applications are developed to outsource functional components to generative AI, the application itself becomes highly vulnerable.
Unknowns will continue to plague applications and products, leading to security issues. If you’ve seen any of my conference presentations over the past couple of years, you’ll have heard me talk about these unknowns. For example, we now have conditions in which developers don’t know what code will execute at runtime.
We security professionals aren’t doing ourselves any favors. Much of the guidance on AI security is overly complex, doesn’t align with real-world use cases, and doesn’t help organizations realize value quickly. We are not rising to the occasion.
4. AI Backlash Builds
AI backlash will continue to build in 2026. A vast majority of people on the planet find tech bros abhorrent. Talking about technology as if it’s magic and CEOs foaming at the mouth to replace people leaves a bad taste in the mouth. Also, the shoving AI into every possible crevice of our existence isn’t a condition that a vast majority of people want. We are getting AI in everything, whether we want it or not.
2026 will be a challenging year for tech companies. They have to prove their investments are paying off. As we enter the fourth year of the generative AI craze, companies are still hemorrhaging money. This will lead to more intense claims, hype, and AI in everything. Backlash will certainly result. As to what form this backlash takes or how big it becomes, it’s anyone’s guess.
5. Negative Human Impacts Gain More Attention
When you mention the topic of AI’s negative impacts on humans, people almost universally think of job displacement. However, this isn’t even the most impactful effect on humans. The human impacts of AI have been a focus of mine for years. This is the main focus of Perilous.tech where I’ve covered topics such as cognitive atrophy, skills decline, devaluation, dehumanization, and on and on.
I believe more people are recognizing the human impacts of AI, and it will receive far more attention in 2026. Today, the most extreme examples, such as people committing suicide or AI psychosis, get all of the attention, but this is starting to shift.
I recently saw Jonathan Haidt mention these cognitive and developmental issues, referencing both Idiocracy and The Matrix. Two references I’ve also made in the past couple of years. These are natural conclusions once you consider the facts on the ground. AI can make you stupid and overconfident in an environment that seems like it’s already saturated with stupid and overconfident people.
6. OpenAI’s Device Flops
OpenAI is working on a device, and it’s going to be the most world-changing thing ever. It will demonstrate that OpenAI absolutely has a moat. After all, they’ve hired Johnny Ive! You sense my sarcasm.
I’m not sure what form OpenAI’s device will take or even if it will be launched in 2026, but it’s rumored to be a small, screenless device with a microphone and camera. This road has been traveled before, a couple of examples are the Humane pin and the 01 light. These devices failed for the same reason OpenAI’s will. It’s not that these devices lacked capabilities, it’s that they directly conflicted with culture. We have a screen-based culture, and now OpenAI expects people to give up the screens? No chance.
People are accustomed to having their experiences mediated, and screens are a large part of that. There’s an idealized vision that people will wear these devices and use them to make sense of the world. Unfortunately, in our current culture, people aren’t curious about the world or look at it with a sense of wonder. They want to transform the world into content. Everyone on the planet now has camera eye, and nobody is going to trust a wearable to frame content.
The device will also be visible to others, so it will signal something about you as a person, and what it signals is nothing good. In addition, if the device has a microphone and camera, public shaming will further lead people to either abandon it or avoid purchasing it altogether, regardless of its functionality.
There’s also the verification aspect. People have become accustomed to degraded tech performance, and they will just not want to talk to their neck and hope that the device takes some action on their behalf. They’ll want to verify.
Remember the GPT Store? Yeah, nobody else does either, including the influencers who claimed it was the new AppStore. We’ll get overwhelming hype followed by a belly flop the size of the US economy, regardless of whether the device is launched in 2026 or 2027.
Conclusion
Buckle up, we aren’t through the hype yet. We are in an era where faith in gods is replaced by faith in tech, and people can gamble on the mundane aspects of daily life. 2026 is going to be weird.
AI security is a hot topic in the world of cybersecurity. If you don’t believe me, a brief glance at LinkedIn uncovers that everyone is an AI security expert now. This is why we end up with overly complex and sometimes nonsensical recommendations regarding the topic. But in the bustling market of thought leadership and job updates, we’ve seemed to have lost the plot. In most cases, it’s not AI security at all, but something else.
Misnomer of AI Security: It’s Security From AI
I recently delivered the keynote at the c0c0n cybersecurity and hacking conference in India. It was truly an amazing experience. One of my takeaways was encouraging a shift in perspective on the term “AI Security,” highlighting how we often approach this topic from the wrong angle.
The term “AI Security” has become a misnomer in the age of generative AI. In most cases, we really mean securing the application or use case from the effects of adding AI. This makes sense because adding AI to a previously robust application makes it vulnerable.
In most cases, we really mean securing the application or use case from the effects of adding AI.
For most AI-powered applications, the AI component isn’t the end target, but a manipulation or entry point. This is especially true for things like agents. An attacker manipulates the AI component to achieve a goal, such as accessing sensitive data or triggering unintended outcomes. Consider this like social engineering a human as part of an attack. The human isn’t the end goal for the attacker. The goal is to get the human to act on the attacker’s behalf. Thinking this way transforms the AI feature into an actor in the environment rather than a traditional software component.
There are certainly exceptions, such as with products like ChatGPT, where guardrails prevent the return of certain types of content that an attacker may want to access. An attacker may seek to bypass these guardrails to return that content, making the model implementation itself the target. Alternatively, in another scenario, an attacker may want to poison the model to affect its outcomes or other applications that implement the poisoned model. Conditions like these exist, but are dwarfed in scale by the security from AI scenarios.
Once we start thinking this way, it makes a lot of sense. We shift to the mindset of protecting the application rather than focusing on the AI component.
AI Increases Attack Surface
Another thing to consider is that adding AI to an application increases the attack surface. Increase in attack surface manifests in two ways: first, functionally through the inclusion of the AI component itself. The AI component creates a manipulation and potential access point that an attacker can utilize to gain further access or create downstream negative impacts.
Second, current trendy AI approaches encourage poor security practices. Consider practices like combining data, such as integrating sensitive, non-sensitive, internal, and external data to create context for generative AI. This creates a new high-value target and is a poor practice that we’ve known from decades of information security guidance.
Also, we have trends where developers take user input, request code at runtime, and slap it into something like a Python exec(). This not only creates conditions ripe for remote code execution but also a trend where developers don’t know what code will execute at runtime.
Vulnerabilities caused by applying AI to applications don’t care whether we are an attacker or a defender. They affect applications equally. This runs from the AI-powered travel agent to our new fancy AI-powered SOC. Diamonds are forever, and AI vulns are for everyone.
It’s Simpler Than It Seems
Here’s a secret. In the real world, most AI security is just application and product security. AI models and functionality do nothing on their own. They must be put in an application and utilized in a use case, where risks materialize. It’s not like AI came along and suddenly made things like access control and isolation irrelevant. Instead, controls like these became more important than ever, providing critical control over unintended consequences. Oddly enough, we seem to relearn this lesson with every new emerging technology.
In the real world, most AI security is just application and product security.
The downside is that without these programs in place, organizations will accelerate vulnerabilities into production. Not only will they increase their vulnerabilities, but they’ll be less able to address them properly when vulnerabilities are identified. Trust me, this isn’t the increase in velocity we’re looking for.
I’ve been disappointed at much of the AI security guidance, which seems to disregard things like risk and likelihood of attack in favor of overly complex steps and unrealistic guidance. We security professionals aren’t doing ourselves any favors with this stuff. We should be working to simplify, but instead, we are making things more complex.
It can seem counterintuitive to assume that something a developer purposefully implements into an application is a threat, but that’s exactly what we need to do. When designing applications, we need to consider the AI components as potential malicious actors or, at the very least, error-prone actors. Thinking this way shifts the perspective for defending applications towards architectural controls and mitigations rather than relying on detecting and preventing specific attacks. So much focus right now is on detection and prevention of prompt injection, and it isn’t getting us anywhere, and apps are still getting owned.
I’m not saying detection and prevention don’t play a role in the security strategy. I’m saying they shouldn’t be relied upon. We make different design choices when we assume our application can be compromised or can malfunction. There are also conversations about whether security vulnerabilities in AI applications are features or bugs, allowing them to persist in systems. While the battle rages on, applications remain vulnerable. We need to protect ourselves.
There is no silver bullet, and even doing the right things sometimes isn’t enough to avoid negative impacts. But if we want to deploy generative AI-based applications as securely as possible, then we must defend them as though they can be exploited. We can dance like nobody is watching, but people will discover our vulnerabilities. Defend accordingly.
The past couple of years have been fueled entirely by vibes. Awash with nonsensical predictions and messianic claims that AI has come to deliver us from our tortured existence. Starting shortly after the launch of ChatGPT, internet prophets have claimed that we are merely six months away from major impacts and accompanying unemployment. GPT-5 was going to be AGI, all jobs would be lost, and nothing for humans to do except sit around and post slop to social media. This nonsense litters the digital landscape, and instead of shaming the litterers, we migrate to a new spot with complete amnesia and let the littering continue.
Pushing back against the hype has been a lonely position for the past few years. Thankfully, it’s not so lonely anymore, as people build resilience to AI hype and bullshit. Still, the damage is already done in many cases, and hypesters continue to hype. It’s also not uncommon for people to be consumed by sunk costs or oblivious to simple solutions. So, the dumpster fire rodeo continues.
Security and Generative AI Excitement
Anyone in the security game for a while knows the old business vs security battle. When security risks conflict with a company’s revenue-generating (or about to be revenue-generating) products, security will almost always lose. Companies will deploy products even with existing security issues if they feel the benefits (like profits) outweigh the risks. Fair enough, this is known to us, but there’s something new now.
What we’ve learned over the past couple of years is that companies will often plunge vulnerable and error-prone software deep into systems without even having a clear use case or a specific problem to solve. This is new because it involves all risk with potentially no reward. These companies are hoping that users define a use case for them, creating solutions in search of problems.
What we’ve learned over the past couple of years is that companies will often plunge vulnerable and error-prone software deep into systems without even having a clear use case or a specific problem to solve.
I’m not referring to the usage of tools like ChatGPT, Claude, or any of the countless other chatbot services here. What I’m referring to is the deep integration of these tools into critical components of the operating system, web browser, or cloud environments. I’m thinking of tools like Microsoft’s Recall, OpenAI’s Operator, Claude Computer Use, Perplexity’s Comet browser, and a host of other similar tools. Of course, this also extends to critical components in software that companies develop and deploy.
At this point, you may be wondering why companies choose to expose themselves and their users to so much risk. The answer is quite simple, because they can. Ultimately, these tools are burnouts for investors. These tools don’t need to solve any specific problem, and their deep integration is used to demonstrate “progress” to investors.
I’ve written before about the point when the capabilities of a technology can’t go wide, it goes deep. Well, this is about as deep as it gets. These tools expose an unprecedented attack surface and often violate security models that are designed to keep systems and users safe. I know what you are thinking, what do you mean, these tools don’t have a use case? You can use them for… and also ah…
The Vacation Agent???
The killer use case that’s been proposed for these systems and parroted over and over is the vacation agent. A use case that could only be devised by an alien from a faraway planet who doesn’t understand the concept of what a vacation is. As the concept goes, these agents will learn about you from your activity and preferences. When it’s time to take a vacation, the agent will automatically find locations you might like, activities you may enjoy, suitable transportation, and appropriate days, and shop for the best deals. Based on this information, it automatically books this vacation for you. Who wouldn’t want that? Well, other than absolutely everyone.
What this alien species misses is the obvious fact that researching locations and activities is part of the fun of a vacation! Vacations are a precious resource for most people, and planning activities is part of the fun of looking forward to a vacation. Even the non-vacation aspect of searching for the cheapest flight is far from a tedious activity, thanks to the numerous online tools dedicated to this task. Most people don’t want to one-shot a vacation when the activity removes value, and the potential for issues increases drastically.
But, I Needed NFTs Too
Despite this lack of obvious use cases, people continue to tell me that I need these deeply integrated tools connected to all my stuff and that they are essential to my future. Well, people also told me I needed NFTs, too. I was told NFTs were the future of art, and I’d better get on board or be left behind, living in the past, enjoying physical art like a loser. But NFTs were never about art, or even value. They were a form of in-group signaling. When I asked NFT collectors what value they got from them, they clearly stated it wasn’t about art. They’d tell me how they used their NFT ownership as an invitation to private parties at conferences and such. So, fair enough, there was some utility there.
In the end, NFTs are safer than AI because they don’t really do anything other than make us look stupid. Generative AI deployed deeply throughout our systems can expose us to far more than ridicule, opening us up to attack, severe privacy violations, and a host of other compromises.
In a way, this public expression of look at me, I use AI for everything has become a new form of in-group signaling, but I don’t think this is the flex they think it is. In a way, these people believe this is an expression of preparation for the future, but it could very well be the opposite. The increase in cognitive offloading and the manufactured dependence is precisely what makes them vulnerable to the future.
In a way, these people believe this is an expression of preparation for the future, but it could very well be the opposite. The increase in cognitive offloading and the manufactured dependence is precisely what makes them vulnerable to the future.
Advice Over Reality
Social media is awash with countless people who continue to dispense advice, telling others that if you don’t deploy wonky, error-prone, and highly manipulable software deeply throughout your business, then they are going to be left behind. Strange advice since the reality is that most organizations aren’t reaping benefits from generative AI.
Here’s something to consider. Many of the people doling out this advice haven’t actually done the thing they are talking about or have any particular insight into the trend or problems to be solved. But it doesn’t end with business advice. This trend also extends to AI standards and recommendations, which are often developed at least in part by individuals with little or no experience in the topic. This results in overcomplicated guidance and recommendations that aren’t applicable in the real world.
The reason a majority of generative AI projects fail is due to several factors. Failing to select an appropriate use case, overlooking complexity and edge cases, disregarding costs, ignoring manipulation risks, holding unrealistic expectations, and a host of other issues are key drivers of project failure. Far too many organizations expect generative AI to act like AGI and allow them to shed human resources, but this isn’t a reality today.
LLMs have their use cases, and these use cases increase if the cost of failure is low. So, the lower the risk, the larger the number of use cases. Pretty logical. Like most technology, the value from generative AI comes from selective use, not blanket use. Not every problem is best solved non-deterministically.
Another thing I find surprising is that a vast majority of generative AI projects are never benchmarked against other approaches. Other approaches may be better suited to the task, more explainable, and far more performant. If I had to take a guess, I would guess that this number is close to 0.
Generative AI and The Dumpster Fire Rodeo
Despite the shift in attitude toward generative AI and the obvious evidence of its limitations, we still have instances of companies forcing their employees to use generative AI due to a preconceived notion of a productivity explosion. Once again, ChatGPT isn’t AGI. This do everything with generative AI approach extends beyond regular users to developers, and it is here that negative impacts increase.
I’ve referred to the current push to make every application generative AI-powered as the Dumpster Fire Rodeo. Companies are rapidly churning out vulnerable AI-powered applications. Relatively rare vulnerabilities, such as remote code execution, are increasingly common. Applications can regularly be talked into taking actions the developer didn’t intend, and users can manipulate their way into elevated privileges and gain access to sensitive data they shouldn’t have access to. Hence, the dumpster fire analogy. Of course, this also extends to the fact that application performance can worsen with the application of generative AI.
The generalized nature of generative AI means that the same system making critical decisions inside of your application is the same one that gives you recipes in the style of Shakespeare. There is a nearly unlimited number of undocumented protocols that an attacker can use to manipulate applications implementing generative AI, and these are often not taken into consideration when building and deploying the application. The dumpster fire continues. Yippee Ki-Yay.
Conclusion
Despite the obvious downsides, the dumpster fire rodeo is far from over. There’s too much money riding on it. The reckless nature with which people deploy generative AI deep into systems continues. Rather than identifying an actual problem and applying generative AI to an appropriate use case, companies choose to marinade everything in it, hoping that a problem emerges. This is far from a winning strategy. Companies should be mindful of the risks and choose the right use cases to ensure success.
Although AI has taken a hit in the past few weeks, the vibes are still strong and infecting every part of our lives. Vibe coding, vibe analytics, and even vibe thinking, because well, nothing says “old” like having thoughts grounded in reality. However, an interesting trend is emerging in software development, one that could have far-reaching implications for the future of software. This is a type of code roulette where developers don’t know what code will execute at runtime. Then again, what’s life without a little runtime suspense?
Development and Degraded Performance
The world runs on software, so any trend that degrades software quality or increases security issues has an outsized impact on the world around us. We’ve all witnessed this, whether it’s the video conferencing app that periodically crashes after an update or a UI refresh that makes an application more difficult to use.
Traditionally, developers write code by hand, copy code snippets, use frameworks, skeleton code, libraries, and many other methods to create software. Developers may even use generative AI tools to autocomplete code snippets or generate whole programs. This code is then packaged up and hosted for users. The code stays the same until updates or patches are applied.
But in this new paradigm, code and potentially logic are constantly changing inside the running application. This is because developers are outsourcing functional components of their applications to LLMs, a trend I predicted back in 2023 in The Brave New World of Degraded Performance. In the previous post, I covered the impacts of this trend, highlighting the degraded performance that results from swapping known, reliable methods for unknown, non-deterministic methods. This paradigm leads to the enshittification of applications and platforms.
In a simplified context, instead of developers writing out a complete function using code, they’d bundle up variables and ask an LLM to do it. For simplicity’s sake, imagine a function that determines whether a student passes or fails based on a few values.
def pass_fail(grade, project, class_time):
if grade >= 70 and project == "completed" and class_time >= 50:
return "Pass"
else:
return "Fail"
If a developer decided to outsource this functionality to an LLM inside their application, it may look something like this.
prompt_pass = """You are standing in for a teacher, determining whether a student passes or fails a class.
You will use several values to determine whether the student passes or fails:
The grade the student received: {grade}
Whether they completed the class project: {project}
The amount of class time the student attended (in minutes): {class_time}
The logic should follow these rules:
1. If the grade is above 70
2. If the project is completed
3. If the time in class is above 50
If these 3 conditions are met, the student passes. Otherwise, the student fails.
Based on this criterion, return a single word: "Pass" or "Fail". It's important to only return a single
word.
"""
prompt = prompt_pass.format(grade=grade, project=project, class_time=class_time)
response = client.models.generate_content(model="gemini-2.5-flash", contents=prompt)
print(response.text)
As you can see, one of these examples contains the logic for the function inside the application, and the other has the logic existing outside the application. The prompt is indeed visible inside the application, but the actual logic exists somewhere in the black box of LLM land.
The example using code has greater visibility, and it’s far more auditable since the logic can be examined, which makes it far easier to debug when issues arise, and of course, it’s explainable. The real problem lies in execution.
The written Python function approach gives you the same result based on the input data every single time, without fail. The natural language approach, not so much. In this non-deterministic approach, you are not guaranteed the same answer every time. Worse yet, when this approach is used for critical decisions and functionality, the application can take on squishy and malleable characteristics, meaning users can potentially manipulate them like Play-Doh.
At first glance, this example appears silly, as writing out the logic in natural language seems more burdensome than using the simple Python function. Not to mention, slower and more expensive. But looks can be deceiving. People are increasingly opting for the natural language approach, particularly those with only minimal Python knowledge. This natural language approach is also more familiar to people who are more accustomed to using interfaces like ChatGPT.
Execute and Pray
However, let’s take a look at another scenario. In this scenario, a developer wants to generate a scatter plot using the Plotly library. In this case, we have some data for the X and Y axes of a scatter plot and use Plotly Express, which is a high-level interface for Plotly (as a developer may when plotting something so simple).
This is a simplified example, but in this case, we can clearly see the code that generated the plot and be certain that this code will execute during the application’s runtime. There is control over the imports and other aspects of execution. It also makes it auditable and provable.
Now, what happens when a developer allows modification of their code at runtime? In the following example, instead of writing out the Plotly code to generate a scatter plot, the developer requests that code be generated from an LLM to create the graph, then executes the resulting code.
prompt_vis = """You are an amazing super awesome Python developer that excels at creating data visualizations using Plotly. Your task is to create a scatter plot using the following data:
Data for the x axis: {xdata}
Data for the y axis: {ydata}
Please write the Python code to generate this plot. Only return Python code and no explanations or
comments.
"""
prompt = prompt_vis.format(xdata=xdata, ydata=ydata)
response = client.models.generate_content(model="gemini-2.5-flash", contents=prompt)
exec(clean_response(response.text))
As you can see from the Plotly code in this example… Of course, you can’t see it because the code doesn’t exist until the function is called at runtime. If you are curious, the first run of this generated the following code after cleaning the response and making it appropriate for execution.
The AI-generated code creates the same graph as the written-out code in the previous example, despite being different. You may be wondering what the big deal is since the result is the same. The concern stems from several reasons, but primarily, allowing an LLM to generate code at runtime is not robust and leads to unexpected outcomes. These outcomes may include the generation of non-functional code, incorrect code, and even vulnerable code, among others.
For a simple example, as the one shown in this post, the chances of getting the same or incredibly similar code returned from the LLM are high, but not guaranteed. For more complex examples, such as those developers may want to use this approach for, the odds increase that the generated code will change more frequently.
Additionally, I implemented a quick cleaning function called clean_response to remove non-Python elements, such as text and triple backticks, from the response. The LLM can introduce additional unexpected characters that end up breaking my cleaning function and making my application fail. The list goes on and on, but a larger danger lurks in the background.
Whose Code Is It Anyway?
If you are versed in security and familiar with Python, you may have noticed something in the LLM example: The use of the Python exec() function. The exec () and eval() functions in Python are fun because they directly execute their input. Fun as in, dangerous. For example, if an attacker can inject input into the application, they can affect what code gets executed, leading to a condition known as Remote Code Execution (RCE).
An RCE is a type of arbitrary code execution in which an attacker can execute their own commands remotely, completely compromising the system running the vulnerable application. They can use this access to steal secrets, spread malware, pivot to other systems, or potentially backdoor the system running the application. Keep in mind, this system may be a company’s server, cloud infrastructure, or it may be your own system.
Anyone following security issues in AI development is aware that RCEs are flying off the shelves at alarming rates. A condition that was previously considered a rarity is becoming common. We even commented during our Black Hat USA presentation that it was strange to see people praising CISA for promoting memory safe languages to avoid things like remote code execution, while at the same time praising organizations essentially building RCE-as-a-Service. Some of this is mind-boggling, since in many cases, outsourcing these functions isn’t a better approach. In the previous example, writing out the Plotly code instead of generating it at runtime is relatively easy, more efficient, and far more robust.
Up until AI came along, the use of Python exec() was considered poor coding practice and dangerous. Now, developers shrug, stating that’s how applications work. As a matter of fact, agent platforms like HuggingFace’s smolagents use code execution by default. This is a wakeup. So, we dynamically generate code, provide deep access, and the ability to call tools, all with a lack of visibility. What could possibly go wrong???
Not only have developers chosen paradigms to generate and execute code at runtime, but worse yet, they’ve begun to perform this execution in agents with user (aka attacker) input, executing this input blindly in the application. In our presentation titled Hack To The Future: Owning AI-Powered Tools With Old School Vulns at Black Hat USA this year, we refer to this trend as Blind Execution of Input, which is the purposeful execution of input without any protection against negative consequences. This condition certainly leads to RCE and other unintended consequences, providing attackers with a significantly larger attack surface to exploit.
An application that takes user input and combines it with LLM functionality is a recipe for a bad time from a security perspective. Another common theme in our presentation, as well as that of other presenters on stage at Black Hat, is that if an attacker can get their data into your generative AI-based system, you can’t trust the output.
Things Will Get Worse
Using the outsourced approach when a more predictable deterministic approach is a better fit will continue to degrade software from a reliability and security perspective and have an impact on the future of software development.
Vulnerabilities in AI software have made exploitation as easy as it was in the 1990s. This was the “old school” hint in the title of our talk. This isn’t a good thing, because the 90s were a sort of free-for-all. Not only that, but in the 90s, we often had to live with vulnerabilities in systems and applications. For example, in one of the first vulnerabilities I discovered against menuset on Windows 3.1, it was impossible to fix. There were no mitigations, and most people were unaware of its existence.
As the outsourcing of logic to LLMs accelerates, things will worsen not only due to incorrect output and hallucinations but also from a security perspective. Anyone paying attention to the constant parade of vulnerabilities in AI-powered software can see this trend with their own eyes. These vulnerabilities are often found in large, mature organizations with dedicated security processes and teams in place to support them. Now, consider startups and organizations that implement their own experiments using non-deterministic software, often with a lack of understanding of how these systems can be manipulated. It’s become a game of speed above everything else.
As I’ve said from the beginning of the generative AI craze, the only way to address these issues is architecturally. Most of AI security is just application and product security, and organizations without these programs in place are in trouble. If proper architecture, design, isolation, secrets management, security testing, threat modeling, and a host of other activities weren’t considered table stakes before, they certainly are now. And possibly not surprisingly enough, they still aren’t being done. Anyone working for a security organization sees this every day.
In essence, developers need to design their applications to be robust to failures and attacks. It helps to consider designing them as though an attacker can manipulate and compromise them, working outward from this premise. As the adage goes, an attacker only needs to be successful once; a defender needs to be successful every time. This makes something that sounds great in theory, like being 90% effective, sound less impressive in practice.
Keep in mind that performing a code review won’t provide the same visibility as it has traditionally. This should be obvious since the code that would be audited doesn’t exist until runtime. You’ll have to pay more attention to validation routines and processing of outputs, putting huge question marks over the black box in the middle. And, of course, ensuring the application is properly isolated.
Some may suggest instrumenting the applications with functionality to perform runtime analysis on the generated code. Sure, it’s possible, but the performance hit would be significant, and even this is, of course, far from a silver bullet. You might not even get the value you think you are getting from this instrumentation. Also, you’d have to know ahead of time the issues you are trying to prevent. That is, unless you plan to layer more LLMs on top of LLMs in a spray-and-pray configuration.
To keep this grounded, all AI risk is use case dependent. AI models don’t do anything until packaged into applications and used in use cases. There may be cases where reliability, performance, and even security are of lesser concern. Fair enough, but it’s a mistake to treat all applications as though they fall into this category, and it’s far too easy to overlook something important and view it as insignificant.
If you work at an organization that isn’t building these applications and think you’re safe, you might want to think again, because you are at the mercy of third-party applications and libraries. It would be best to start asking hard questions of your vendors about their security practices as they relate to applications you purchase. Especially applications that use generative AI to generate code and execute it at runtime.
Near the end of our presentation, we had some advice.
Whether outsourcing the logic of an application to LLMs or having the LLM dynamically generate code, assume these are squishy, manipulable systems that are going to do things you don’t want them to do. They are going to be talked into taking actions that you didn’t intend, and fail and hallucinate in ways you don’t expect. Starting from this premise gives a proper foundation for deploying controls to add some resilience to these systems. Of course, not taking these steps means your applications will contribute to the ongoing dumpster fire rodeo.
By now, you’ve no doubt heard of the term vibe coding. It’s become the favorite talking point from influencers and the tech media, which, even in 2025, can’t seem to muster a modicum of skepticism. But, if you’ve ever wondered what it was like to play Russian Roulette in reverse, loading all the chambers but one, spinning the cylinder, and having a go, you’re in for a treat. Welcome to the world of vibe coding and YOLO mode, two things that go together like nitroglycerin and roller coasters. So, of course, it’s become one of the hottest topics right now, and it has all of the bros super psyched.
For those out of the loop, vibe coding is “Where you fully give in to the vibes, embrace exponentials, and forget that the code even exists.” You get to this state by talking with a computer and letting it generate all the code while you kick back and enjoy a cold one. YOLO mode is the spirit animal of vibe coding. It’s where you blindly accept all the code suggestions generated by the AI tool and push the code to see what happens. Neat. It’s interesting to note that YOLO mode in video games means if you die once, you are dead. No respawning.
Vibe Coding = Code Slop
Before we probe the issues with vibe coding, let’s take a step back and look at what vibe coding is. This practice shouldn’t be confused with a developer using an AI coding tool to assist with tasks or gain productivity. You know, the intended use of many of these tools.
Vibe coding is a delusional dream state in which people use these tools as if they are in the future instead of the present. The fact is, these tools aren’t reliable enough or mature enough to be used this way. It’s a lot like having an illness and getting a bottle of pills from the pharmacy marked with clear directions and immediately slamming the entire bottle because the instructions are for losers who don’t know how to hustle.
Vibe coding is a delusional dream state in which people use these tools as if they are in the future instead of the present.
The reality is that vibe coding generates slop that happens to be code instead of media categories, meaning that the negative consequences extend far beyond other categories of slop. This distinction is essential when evaluating the potential value of vibe coding.
Vibe Coding Pitch
The pitch of vibe coding is that literally, anyone can become an instant millionaire developing apps. You’ll be too busy making money to worry about how your code works or its security. There are no boundaries or barriers. All you need is an idea (more on this in a future post.) Like so many things, it’s technically true but practically false. And also, you aren’t wrong if you are beginning to get crypto bro vibes.
Much of the same logic employed by conspiracy theorists is at work here. If one person gains some success vibe coding, then it must be possible for anyone to do that. Technically true, but practically false. This is like thinking millions of new rock stars will be minted because people can publish their songs on Apple Music.
It’s about tradeoffs. We don’t say that being a conspiracy theorist is a good thing since some conspiracy theories turn out to be true. That’s because the negative impacts of conspiracy thinking outweigh the potential benefits. The same applies here.
There are other flaws with their logic. For example, people don’t consume apps the same way they do media like music, video, or photos. People can listen to hundreds of songs every day and not repeat a single one, but this consumption strategy doesn’t apply to applications. People these days often consume media passively. For example, people often don’t listen to music when they are listening to music; it’s purely background noise for other tasks. Applications usually can’t be consumed passively and require active interaction. This would make consuming many different apps irritating to users.
There are already 1.9 million apps on the App Store alone. Are we really hurting for apps? Do we need 100 million apps to compete with the number of songs on Apple Music? Of course not, but that doesn’t mean we won’t get them anyway. If you look at the outputs of vibe coding, it’s often uninteresting, overly simple, derivative, or just plain unwanted. Buckle up.
There will undoubtedly be exceptions, just like the person who was a bartender six months previously who started a crypto project and manages 100 million in assets. These are exceptions and not the rule, but these exceptions serve as accelerants to fuel the hype flames.
People are trying to sell us on the fact that vibe coding has no downsides. This is delusional. Take a step back and think of the answer to a single question. What do you get when everything takes a backseat to speed? It’s like a car that does 200 mph with no seatbelts and no airbags constructed from paper mache.
Before we move on, I’d like to acknowledge something. It’s a good thing that AI coding assistants are making coding more accessible. However, using these tools as a drop-in replacement for common knowledge and domain expertise isn’t a recipe for success. Imagine something like vibe surgery? Yeah, bro, let me get in on that appendix! Nah, I didn’t go to medical school or know much about anatomy, but I got an AI tool, dexterity, and a good sense for vibes. So little of developing an application is about the code itself, but that gets lost in the vibes.
Changing Behavior and Attitudes
In the public sphere, the discussion of the merits and drawbacks of vibe coding and YOLO mode are entirely contained within the technical aspects of the approach. I’m also concerned about the technical components, but I’d like to bring attention to something nobody discusses.
As often happens when a new technology or approach arrives and removes friction, it changes people’s behavior and attitudes. In technology circles, friction is discussed as though it’s universally bad. It’s not. Sometimes, friction is a feature, not a bug. Nicolas Carr provides an excellent example in his book Superbloom, which discusses introducing the Retweet feature on Twitter.
The time and effort required to share a tweet manually, though seemingly so small as to be inconsequential, turned out to be vitally important. It slowed people down. It gave them a moment to reconsider the message they were about to repeat, now under their own name. Once that little bit of friction was removed, people acted and reacted more impulsively. Twitter became more frenzied, more partisan, and much nastier.
Things like vibe coding and YOLO mode will have similar behavioral effects if this technology trend takes off. People won’t put a lot of thought into the apps they create. Some may build apps purely because they can, not considering why an app for that particular purpose didn’t exist in the first place, assuming that it was purely because nobody had built it and not because of the potential for negative impacts or harm.
The removal of so much friction removes not only the appreciation for the problem but also opportunities to catch potential issues. These lines of generated code become grenades with various time delays chucked into production. This assumes that the developer had the skills to identify the issues in the first place.
These lines of generated code become grenades with various time delays chucked into production.
Some will argue that these features are great for prototyping and mockups. I agree. However, as I mentioned, these features change behavior, and using them simply for prototyping won’t hold. A vast majority of people who can get away with chucking vibe-coded apps into production will.
With the changes in behavior and attitudes, there are many things creators of these applications are more likely to do.
Act unethically (ethics don’t align with speed)
Devalue the work of others
Not learn or at least learn lessons
Encounter skills atrophy
Not build robust software (Security, Privacy, Alignment, Reliability, etc.)
Not constrain code to prototypes and mockups
Think they know things they don’t (Illusion of knowledge, Illusion of capability)
Misunderstand what’s valuable
Devalue collaboration
Go it alone and not include domain expertise and misunderstand the problem they think they are solving
Build apps that nobody wants
Build apps that cause harm
Choose poor architectures
Use more resources and not prioritize efficiency
Fail to benchmark properly
Not be able to troubleshoot their own creations
Not do something truly innovative
And on and on…
These were just a few of the conditions off the top of my head. What happens when these conditions now become the norm? When people start making app slop the way they do image slop?
Risk and Security
Vulnerabilities in code and lack of security controls account for a lot of pain and financial loss every year, much of this from organizations that try to do the right things. So, imagine what happens when people don’t care about doing the right things.
It’s known that these tools output vulnerabilities at rather high rates. So imagine what happens when people YOLO code into production and don’t check the resulting code or even the environment where it is hosted for security issues. Hustlers ain’t got no time for the right things.
There’s more to worry about than an AI tool outputting specific code blocks that are vulnerable. Other contributing issues increase the attack surface of an application. For example, choosing a vulnerable library or suggesting vulnerable configuration options for cloud environments. These tools also contribute to library bloat by including multiple libraries that do the same thing.
I could go on and on about this topic, but at this point, the various security issues created by AI coding assistants are known issues. I wrote a white paper on this topic in early 2023, and I delivered a presentation at InfoSec World the same year. These issues should be common knowledge now with the publication of various articles, papers, and presentations.
When it comes to risk, sure, all vibes aren’t created equal. A video game getting hacked isn’t as bad as a financial application getting hacked and draining your bank account. I’m certainly not being an absolutist here. However, technology trends have an odd way of not staying confined to specific buckets. So, we’ve got that to look forward to.
Today, countless vulnerabilities are moving into production without vibe coding, all because people are trying to push things faster. Vibe coding and YOLO mode make this monumentally worse. We’ve only discussed security and haven’t touched on other topics like privacy.
Making Software Worse
The trend of vibe coding will make software worse. Like security, software quality isn’t a consideration in vibe coding because reasoning about quality is a bummer when huffing vibes.
We live in a highly advanced world where digital things fail all around us all of the time, like a leisurely stroll through a cityscape where freshly painted buildings mask a crumbling interior of decay and misshapen architecture. This is so common that there’s a term for it: enshittification. We’ve become so accustomed to the software and tools we use sucking so bad we hardly notice it. This is a contributing factor to why some view generative AI as AGI.
Vibe coding and YOLO mode will lead to failures, half-baked functionality, and mountains of technical debt.
Vibe coding and YOLO mode will lead to failures, half-baked functionality, and mountains of technical debt. This should concern everyone, but queue the bros to claim this is a feature.
Yes, because that’s how things work. He and many like him are stating that they should create as many problems as possible because AI can fix them in the future. Once again, they are taking something technically true but practically false. At some point, we’ll have highly advanced and capable systems that operate this way, but the mistake is thinking those systems are on the cusp of arrival. It’s hard to ignore the religious fervor in these claims.
People can pray to the gods of gradient descent and burn Gary Marcus in effigy, but it doesn’t change the realities on the ground. Problems created today will be with us tomorrow, and no AI god is coming to deliver us from our evils any time soon, so we should work to minimize potential problems instead of running up the credit card. I’ve been calling this problem out for the past couple of years, stating it would lead to a brave new world of degraded performance.
On a side note, I feel these people feed off each other. I’ve heard perfectly reasonable people making wholly unreasonable claims. These are the things you hear people say when they are trapped in filter bubbles, getting high on the supply of techno-utopians. They also suffer from a healthy dose of audience capture because, no doubt, being unreasonable gets you more likes and shares than being reasonable. Welcome to the perverse incentives of modern social media.
There continue to be many misconceptions about software development, but one of the biggest is that writing code is the end of the journey.
There continue to be many misconceptions about software development, but one of the biggest is that writing code is the end of the journey. This is because most people opining on the topic are not developers. I noticed this trend years before the existence of AI coding tools when security professionals who learned to write a few lines of Python code thought that developers’ jobs were easy. The assumption, then, for AI coding tools is that since the tool can output code and developers only write code, developers are no longer necessary. Developing code isn’t the end of the journey. It’s the beginning.
The written code must be troubleshot and maintained, and features must be added. We live in a constantly evolving world with changing problems, environments, and customer needs. Developed code will crash into the realities of the real world both when it’s initially deployed and over its lifetime. This leads to another problem.
Developers don’t understand the code being written, especially when the people generating the code aren’t developers. As developers’ skills atrophy and people who were never developers start creating these applications, they cannot troubleshoot problems, effectively add features, or perform any of the other countless tasks that developers perform daily. The answer from the utopians is to use AI to figure it out, but this strategy won’t always work.
There is a higher likelihood that the AI tool will successfully troubleshoot issues for simple tools and scripts, but these are the very types of applications that are unlikely to net you big money. As applications grow in size and complexity, the AI tool is less likely to provide the solution necessary to resolve the issue.
Imagine a world where an app needs to be rewritten from scratch because the person who created it couldn’t get the AI tool to troubleshoot and fix the problem. Now, that’s the utopia we’ve all dreamed of.
Imagine a world where an app needs to be rewritten from scratch because the person who created it couldn’t get the AI tool to troubleshoot and fix the problem.
There is a vast oversimplification of the entire landscape here. So, an application starts simply enough, and then more requests are made to the AI tool in an attempt to add more functionality, but this doesn’t always work or isn’t done in the most efficient way, leading to a buildup of issues.
Another trend affecting application reliability is using probabilistic systems as though they are deterministic. Whether this trend is due to laziness, ignorance, or an attempt to handle unknowns is unclear, but it will surely affect applications’ reliability and their ability to be manipulated.
Ultimately, we may be left with App Store decay, where the App Store becomes a graveyard for abandoned apps. RIP.
Making Us Worse
I mentioned skills atrophy in my laundry list. It seems that even AI tools understand this problem. This is not only a comical error message but contains a truth.
We Never Learn Lessons
Although arguably more intense and a bit different, what’s happening now in AI isn’t new. We’ve gone through these cycles before with previous technologies. Every time a new technology comes along, we discard the lessons we’ve learned, assuming they no longer apply, only to discover that the previous lessons were even more important with the new innovation. This condition is something I’ve referred to in my conference talks as the emerging technology paradigm.
We never seem to learn lessons from our previous mistakes, no matter how often we encounter them. We have incredibly short memories and seem to dive face-first into the pool without checking the water level.
Ultimately, it’s all about tradeoffs. What we get and what we lose. When viewed simply as writing code, it seems we are getting more than we are losing. However, building and deploying applications and solving problems goes far beyond code. When considering the impacts holistically, this doesn’t appear to be a good tradeoff. However, it’s possibly one we are going to get anyway.
Whenever a new technology or approach comes along, proponents always pitch it with a utopian eye. They envision all the perfectly aligned scenarios with dominoes falling exactly into place. The Internet, Social Media, The Cloud, Web3, and many other technologies all diverted away from these visions and adapted differently than expected. Even something as simple as the telegraph was seen as a utopian invention that would end world conflict. After all, how could people go to war when misunderstandings were a thing of the past? We all know how that turned out. Vibe coding is destined for a similar fate.
Is it possible to play Russian Roulette in reverse without devastating consequences? Sure, but the odds aren’t great. The world also won’t be a better place with everyone vibe coding and YOLOing stuff into production. Many disagree with me. Fair enough. However, if this trend takes off, it will be another example of something we are stuck with, which is not good for a world that runs on software. We will need to improve or invent new technology to solve the problems we create, trading one set of problems for another. Welcome to utopia.
There are few predictions I can make with more certainty than that we’ll hear the word “agent” so many times in 2025 that we’ll never watch another spy movie again. The industry and influencers have latched on to the new hype term and will beat that drum until it screams AGI. In an attempt to FOMO us to death, we’ll run the gauntlet of crushing shame for not deploying agents for absolutely everything. If you aren’t running agents everywhere, then China wins!
Even companies that change nothing about their products will claim to use agents, resembling the Long Island Ice Tea Company when it changed its name to Long Blockchain Corporation to watch its share price spike 500%. Everybody gets rugged.
However, it’s not all bad. Peering beyond the overwhelming hype, failures, and skyrocketing complexity current LLM-based agents bring, there is something informative about the future. Agent-based architectures provide a glimpse into solving real problems. Despite this, reliability and security issues will be major factors hindering deployments in 2025.
To Start With
Since I criticize hype, focus on risks, and make fun of failures, it would be easy to label me a tech hater. This isn’t the case at all and would be far too easy. I have plenty of issues with general tech critics as well. However, at the rate that the hustle bros keep the AI hype cannon firing, I don’t have the time for my quibbles with tech critics. Maybe someday.
For over a year now, I’ve used this image in my presentations to describe my position on LLMs. This is also true for me on just about any piece of tech, which, I’ll remind people, typically ends up being where reality is for most things. It’s instructive to remember that reality often agitates both sides of extreme viewpoints by never being as good as the lovers’ or as bad as the haters’ claims.
It’s instructive to remember that reality often agitates both sides of extreme viewpoints by never being as good as the lovers’ or as bad as the haters’ claims.
Agent Definitions
Like most hype-fueled terms, definitions are secondary to usage. Everyone seems to claim that the definition of agent is whatever they say it is. That’s not overly helpful for anyone trying to make sense of realities on the ground. However, it does inspire funny memes, like this gem from Adam Azzam on Bluesky.
Agents operate within systems with a certain level of autonomy. They make decisions without human intervention and can change and adapt to their environments. If a tool is required to support the agent, the agent decides to call the tool and perform the action. For example, a penetration testing agent may determine it requires more information about the provided IP addresses. To collect this information, it launches the Nmap tool to identify open ports. All of this is done without human intervention. To make things more complex, one agent may call another agent in a multi-agent environment.
“Agentic,” on the other hand, is an amorphous term slapped on top of just about anything to justify the claim that something is “close enough” to be referred to as an agent. Agentic workflows, agentic systems, agentic products—Applebees even has a new agentic side salad for those on the hustle.
You’ll no doubt be confronted with the virtual travel agent when you hear about agents. This agent will choose a destination and activities and book the associated tickets for you. How fun. I don’t know who decided this is the “it” use case for agents, but congratulations. You’ve highlighted a use case nobody wants and certainly didn’t ask for. This choice is so indicative of our current age, where people building and proposing things are far removed from the interests of end users. They feel the idea trumps the need, and users will get on board.
Problems Unsolved and Issues Amplified
Now that the current issues with generative AI have been solved, we can safely deploy them as agents. I can feel your laughing vibes over the internet. Of course, these issues haven’t been solved, and the bad news is that agents don’t solve generative AI issues; they amplify them. We paint the exterior of LLMs with an additional coat of complexity and opaqueness.
If you’ve attended any of my conference talks throughout the generative AI craze, you’ll have heard me highlight these issues. Here are a few below.
Easily Manipulated
It’s not like you can talk to a traditional application and convince it to do something it wasn’t intended to do, but the same can’t be said for generative AI applications. Somewhere, weaved through the training data, these systems have inherited our gullibility. These applications can be socially engineered to perform actions on an attacker’s behalf. This applies to everything from prompt injection to simple manipulation through conversations. Just like there is no patch for human stupidity, there is no patch for generative AI gullibility either.
This isn’t easy to fix, which should be obvious since the problem isn’t fixed yet. Early on, I mentioned how these systems have a single interface with an unlimited number of undocumented protocols. Imagine trying to create a simple trap in the application’s input for the string “Ignore the previous request.” Your work is far from done because the system understands many different ways to represent that input. Here are just a couple of examples:
aWdub3JlIHRoZSBwcmV2aW91cyByZXF1ZXN0
i9nore +he previou5 reque5+
vtaber gur cerivbhf erdhrfg
It seems every release implementing generative AI functionality has been compromised, regardless of the company behind it, and this theme will continue.
Creating New High-Value Targets
Generative AI and agents encourage us to create new high-value targets.
With generative AI systems, there’s a tendency to want to collect and connect disparate and disconnected data sources together so the system can generate “insights.” However, we create new high-value targets that mix sensitive data with external data, almost guaranteeing that an attacker can get data into the system. In this case, you not only can’t trust the output, but depending on the system, they may be able to exfiltrate sensitive data.
Rethinking RCE
There have been instances where people have gotten generative AI-based tools to execute code on their behalf, creating remote code execution vulnerabilities (RCE), some of the most devastating vulnerabilities we have. These issues will no doubt continue to be a problem. However, since generative AI tools are themselves generalized, we may need to start thinking about the LLM portions of our applications as yet another “operating system” or execution environment we need to protect.
In a way, an attacker tricks the system into executing their input rather than the behavior expected by the developers. Although an attacker’s input may not be shoved into a Python exec() statement, they’ve still manipulated the system to execute their input, affecting the application’s execution and resulting output.
Overcomplicating Guidance
We security professionals love to overcomplicate things, and our guidance and recommendations are no exception. I once worked at a company where someone created this massive flow chart for peer reviews that basically stated that when you were done with your report, you should send it to your manager, and they will send it back to you. The old adage that complexity is the enemy of security has always contained a valuable theme that gets sacrificed on the pyre of complexity’s perceived beauty.
I will continue saying that much of AI security is application and product security. These are things we already know how to do. I mean, it’s not like generative AI came along and suddenly made permissions irrelevant. Permissions are actually more important now. But this isn’t satisfying for people who want to play the role of wise sage in the AI age. The guidance and controls of the past aren’t less valuable but more valuable in the age of generative AI and agents.
We’ll see the manufacture of new names for vulnerabilities with increasingly complex guidance and high-fives all around. The secret is these will mostly be variations on the same themes we’ve already seen, such as manipulation, authorization, and leakage flaws.
Back in May of 2023, I created Refrain, Restrict, and Trap (RRT), a simple method for mitigating LLM risks while performing design and threat modeling. It still holds up as a starting point and applies to agents as well. Simple just works sometimes.
Continue To Be Owned
These applications, including ones launched as agents, will continue to be owned. Owned, for those not familiar with security vernacular, means compromised. I made this prediction in the Lakera AI Security Year in Review: Key Learnings, Challenges, and Predictions for 2025 in December. I’m fully confident this trend will continue.
I mentioned that the issues haven’t been fixed, and now people are increasing deployments and giving them more autonomy with far more access to data and environments. This results in far worse consequences when a compromise occurs. To make matters worse, we’ll begin to see organizations deploy these systems in use cases where the cost of failure is high, creating more impact from failures and compromises.
Failures and Poor Performance
These implementations will continue to fail where LLM-based use cases fail, but potentially worse. For example, it’s easy to see how increasing complexity can cause a lack of visibility with potential cascading failures. In 2025, organizations will likely continue dipping their toe into the waters of high-risk use cases where the cost of failure is high, as mentioned previously.
Sure, a car dealership chatbot offering to sell a truck for one dollar is funny, but it has no real impact. However, high-risk and safety-critical use cases have a large financial impact or possibly cause harm or loss of human life. You may roll your eyes and say that would never happen, but what happens in a more simple use case when OpenAI’s Whisper API hallucinates content into someone’s medical record? Because that’s already happening.
Due to their lack of visibility and minimized human control, AI agents can mimic grenades when deployed in high-risk use cases, where the damage doesn’t happen the moment you pull the pin. This complicates things as it means that issues may not shake out during experimentation, prototypes, or even initial usage.
Agents can mimic grenades when deployed in high-risk use cases, where the damage doesn’t happen the moment you pull the pin.
Generative AI is still an experimental technology. We haven’t worked out or discovered all of the issues yet, leading to another example I’ve used as a warning in my presentations over the past couple of years: AlphaGo beating Lee Sedol at Go. Many have heard of this accomplishment, but what many haven’t heard is that even average Go players can now beat superhuman Go AIs with adversarial policy attacks. We may be stuck with vulnerable technology in critical systems. Sure, these are different architectures, but this is a cautionary tale that should be considered before deploying any experimental technology.
Beyond failures and compromises, we adopt architectures that work but don’t work as well as more traditional approaches. In our quest to make difficult things easy, we make easy things difficult. Welcome to the brave new world of degraded performance.
Success and Good Enough
For the past few years, I’ve been pushing back against the famous phrase, “AI won’t replace people. People with AI will replace people without.” This is complete nonsense. I have an upcoming blog post about this where I “delve” into the topic. The reality is the opposite. The moment an AI tool is mediocre enough to pass muster with a reasonable enough cost, people will be replaced, AI use or not. This is already being planned.
The moment an AI tool is mediocre enough to pass muster with a reasonable enough cost, people will be replaced, AI use or not.
Like most technology, agents will have some limited success. And that success will be trumpeted in 2025 as the most earth-shattering innovation of ALL TIME! I can hear it now. “You just wait bro, in 2025 agents are going to the moon!” Maybe. But, given the environment and the fact that issues with LLMs haven’t been solved, an LLM-powered rocket to the moon isn’t one I’d consider safe. Passengers may very well find themselves on a trip to the sun. The future is bright, very bright. 🕶️
How much success agents have in 2025 and what impact this success has remains to be seen. At this point, it’s far from obvious, but I won’t be surprised by their successes in some cases or their spectacular failure in others. This is the reality when the path is shrouded in a dense fog of hype.
Things to look for in successes would be use cases with limited exposure to external input, low cost of failure, and cases where inputs and situations require adapting to change. The use case will also need to tolerate the lack of visibility and explainability of these systems. There will also be continuing success in use cases where tools can be used.
The idea of a multi-agent approach to solving complex problems isn’t a bad one, especially when unknowns enter the equation. Breaking down specific tasks for agents so that they’re focused on these tasks as part of a larger architecture is a solid strategy. However, the current and unsolved issues with generative AI make this approach fraught with risk. In the future, more robust systems will most likely exploit this concept for additional success.
Cybersecurity Use Cases and Penetration Testing
There’s certainly the possibility of disruption in cybersecurity. Before the generative AI boom, I joked with someone at Black Hat that if someone created a product based on reinforcement learning with offensive agents that were just mediocre enough, they’d completely wipe out pen testing.
For years, people have discussed how penetration testing work has become commoditized, and there is a race to the bottom. I don’t think that has happened to the extent many predicted, but we could see a shift from commoditization to productization.
Pen testing also seems to check the boxes I mentioned previously.
Low cost of failure
Varying quality
Value misalignment
Tool use
Adaptation to unknowns
Pen testing is an activity with a low cost of failure. The failure is missing a vulnerability, which is something humans also do. This scenario is hardly the end of the world. Yes, an attacker could indeed find the vulnerability and exploit it to create damage, but it depends on various factors, including exposure, severity, and context.
The quality of pen tests is often all over the map and highly dependent on the people performing the work. Human experts at the top of their game will continue to crush AI-powered penetration testing tools for quite some time. However, most organizations don’t hire experts, even when they hire third parties to perform the work. The value of such a tool in this environment becomes far more attractive, potentially enough to postpone a hire or discontinue using a third party for penetration testing needs (if regulations allow.)
The value of pen testing isn’t always aligned with the need. Many customers don’t care about pen testing. They are doing it because it’s required by some standard, policy, compliance, or possibly even simply because they’ve always done it. Pen testing is one of those things where if customers could push a button and have it done without a human, they’d be okay with that. Pushing a button is the spirit animal of the checkbox. After all, the goal of pen testing is not to find anything. You certainly have due diligence customers and people who truly value security, but the number of checkbox checkers far outweighs these folks.
Human pen testers use tools to perform their jobs. Tool use has shown promise and some success with LLMs at performing certain security-related tasks. This is yet another indicator that a disruption could be on the horizon.
Every environment and situation is different for pen testers. You are given some contextual information along with some rules and are turned loose on the environment. This is why humans are far more successful than vulnerability scanners at this task, much to the chagrin of product vendors. However, adapting to some of these unknowns may be something generative AI agents can adapt to at a reasonably acceptable level. We’ll have to see.
Given what I outlined, you may believe that generative AI tools give attackers an advantage over defenders, but this isn’t the case. The benefits of AI tools, generative AI or otherwise, align far more with defender activities and tasks than with attacker activities. This will remain true despite any apparent ebb and flow.
New Year’s Resolution
It’s the time of year when people make resolutions, so how about this? 2025 has already launched with the firehose fully open, blasting us directly in the face with 150 bsi (Bullshit per Square Inch) of pure, unadulterated hype.
We are only a few days into the year, and it seems as though the religion of AI is far exceeding reality. Hype is what’s going on. It’s that simple. It’s 2025. Let’s make it the year to add at least “some” skepticism, not believing every claim or demo as though it’s the gospel according to Altman.
Sam Altman isn’t a prophet. He’s a salesman. In any other situation, he’d be cluttering up your LinkedIn inbox and paying a data broker to get your work email address and phone number. “Look, I know I’ve called six times, but I really think our next-generation solution can skyrocket your profits. I’m willing to give you a hundred-dollar Amazon gift card just for a demo!”
Sam Altman claims that OpenAI knows how to build AGI, and we’ll see it in 2025, triggering the predictable responses from useful idiots. Remember, these things are performance art for investors, not useful information for us. If we had any attention span left, we’d remember him as the little boy who cried AGI.
Let’s analyze this paragraph, which is the one that’s sending generative AI to the moon on social media. It consists of three sentences that have nothing to do with each other, but since the shockwave of hype pulverizes our minds, we glue them together.
We are now confident we know how to build AGI as we have traditionally understood it.
That’s not true. Once again, this is performance art for investors. A possibility is that they redefine AGI to align with whatever goalposts they set and pat their own backs at the end of 2025.
We believe that, in 2025, we may see the first AI agents “join the workforce” and materially change the output of companies.
Okay, but what does this have to do with AGI? You see, this is sleight of hand. He wants you to believe this is connected to the previous point about AGI. It is not. This doesn’t require AGI to be true. If there is some success here, people can point to this as proof of some proto-AGI, which won’t be the case.
We continue to believe that iteratively putting great tools in the hands of people leads to great, broadly-distributed outcomes.
HAHAHAHA. What? Did he write that, or did ChatGPT? It is also not related to AGI. Great, broadly-distributed outcomes, but not for most people on the planet. The goal is workforce reduction, broadly distributed workforce reductions. Although it’s true that some high school kid may indeed invent the next big thing, creating a multi-million dollar company, for every one of these, there will be countless droves of people displaced from the workforce, quite often, with nowhere to go. Or, at least, this is the goal. We can be honest about these things without delusions, but this brings its own challenges.
Okay, I’m having a bit of fun with Sam Altman’s nonsense, but some of this isn’t his fault. He can’t be completely honest with people, either, due to the uncomfortable situation of cheerleading technology claiming to remove people’s autonomy and sometimes their purpose. If people can’t work, they can’t support their families. I’ve written about the backlash against AI-powered tech in the past and its consequences. AI hype is putting all of humanity on notice, and humanity notices. Backlash plays a large part in why there is a lack of honesty.
AGI will happen. We should acknowledge this fact, and living in denial about it isn’t a strategy for the future. However, it won’t be OpenAI who creates it in 2025. If I had to place a bet today on who would actually create AGI, I’d bet on Google DeepMind. DeepMind is a serious organization that continues to impress with its research and accomplishments, quite often making the competition look silly. But then again, those are just my “vibes.”
Let me make this clear. My criticism of Altman, or any company’s strategy, marketing, or ludicrous levels of hype, has nothing to do with the hard-working people who work there or their accomplishments. I know some of these people. They aren’t fools by any stretch. But, their work is tarnished when every time Altman makes a claim, like believing that angels are in the optimizer.
We know that every AI demo and usage scenario runs into the complexities of the real world under normal conditions. Yet, we seem to forget this lesson every time a demo or claim is made. 2025 is going to bring more stunts, more claims, and more demos. We should experiment in our own environments, with our own data, to apply what works best for us and aligns with our risk tolerance. Don’t believe everything you see on the internet.
I had a great conversation with Aseem Jakhar for CIO.inc and iSMG. We covered topics surrounding AI Safety and Security as well as deepfakes. I explained why I don’t think the misinformation aspect of deepfakes will affect the outcome of elections and provided my opinion on deepfake detectors. We also discuss how we think we need to throw out the rulebook every time a new technology comes along instead of applying lessons learned.
So, you are attending a security conference. That’s great news. Every year, more and more people join the security community and explore the experience of attending a security conference. If you are new or relatively new to security conferences, getting the most out of your experience isn’t obvious. There are things people don’t tell you that you learn through trial and error. I hope to make this a bit less painful.
I’ve been attending and speaking at conferences for over twenty years. I’ve given my share of advice to new attendees, but it’s always been in person or as a couple of pointers to someone else’s social media posts. I wanted to share some insights since Black Hat USA and DEF CON are next week. As our community grows, new people begin to attend conferences and much of the advice they are given isn’t helpful. I hope to rectify that. If I can help even a few people make the most out of their experience and avoid some pitfalls, then this post served its purpose.
Bad Advice
Let’s start with bad advice. There’s no shortage of truly unhelpful advice that people seem to dish out as though they dispense ancient wisdom. You’ll hear things like pace yourself, drink water, and wear deodorant. It’s the type of condescending, childish response you get from people who pretend they have some secret knowledge and don’t want to share it with you. Yes, Las Vegas is in the desert. It’s also hot. Anyone with two brain cells to rub together can put as much together on their own.
Worst of all, we all know this isn’t the type of advice people seeking it want to hear. It’s like someone stepping up to the high dive for the first time and asking for advice, only to be confronted with someone telling them to make sure they bring a swimsuit. So, let’s all agree to stop doing this.
Understanding Security Conferences
I absolutely love security conferences, even though I may jokingly tell you I despise them. You see, I’m not the biggest people person in the world, but at a security conference talking shop with others, I feel like I’m in my element. I’m more likely to strike up a conversation with complete strangers and make new friends and acquaintances. This is why I want to encourage people to attend, interact, and engage. Now, we are starting to get to the heart of what security conferences are.
The first thing to understand is that security conferences are events or experiences if we are being dramatic. These events are only partly about the presentations delivered on stage. Thinking that security conferences are only about presentations is like thinking that county fairs are only about Ferris wheels. Conferences of all sizes typically have other activities, such as meetups, CTFs, vendors, contests, etc. This is what builds the experience of attending.
It surprises people to discover that presentations may not be the most valuable part of the conference. Given this perspective, you’ll want to maximize your conference experience. Hopefully, the rest of this post will assist with this goal.
It surprises people to discover that presentations may not be the most valuable part of the conference.
Start Here
What do you hope to get out of your conference experience? What would a successful conference experience look like when you return from the conference? Think of this as your destination on your map. However, instead of planning a turn-by-turn route or creating a script you need to follow, you’ll want to think of it as throwing out some waypoints and figuring the route out later. You’ll also want to leave some options open, as well as the ability to capitalize on serendipity.
If the conference has an official app, use it to build a schedule. If you’d prefer to use another calendar with which to build a schedule, that’s fine too, but just have a way to track events and activities that interest you
Sessions and Presentations
There will undoubtedly be sessions you won’t want to miss. Add these most important sessions to your schedule. Additionally, If you don’t have other conflicting activities, fill up the remaining conference hours with potential sessions that interest you. These will be your optional sessions. You aren’t locked into going to any of these, but at least you aren’t standing in the hallway trying to figure out what session you want to see or figuring out where it is while the sessions have already started.
Again, this is just a high-level plan and not some script you must follow, so fill the time slots. Okay, that’s it for the content and presentations. If you believe conferences are only about presentations, you can stop here. Wasn’t that easy? But if you really want to understand security conferences, keep reading.
The Security Community
It’s called the security community for a reason. The security community is among the weirdest, quirky, and welcoming communities of any industry. Full stop. I can’t think of another community where you could walk up on a technical conversation between a person in a sports coat, a furry, and someone wearing a tinfoil hat and shirt that reads “birds aren’t real.” All three of them have a great conversation, respecting each other’s perspective. If you wrote this into a TV show, nobody would believe it.
The community aspect of our industry is arguably the most valuable asset you can cultivate. It can launch your career into a whole new trajectory, allow you to make new friends, and introduce you to new avenues you didn’t know were open to you. All of this is on full display at security conferences.
The community aspect of our industry is arguably the most valuable asset you can cultivate.
This community aspect is why you’ll want to maximize your participation in networking opportunities. These include meet-ups, gatherings, and events that are both official and unofficial.
Hallway Con
Hallway Con is the security community’s conference. It’s the con within a con. Hallway Con is the reference to all of the conversations that happen outside of the presentation area. It’s the meetups, networking, catching up with old friends, and discussions with strangers, experts, and peers.
Hallway Con is where the real information is shared. People disclose details they wouldn’t share on the stage or over a video conference. This less filtered sharing allows for a more accurate picture and perspective of realities on the ground. People will be more honest in their assessments and give candid responses.
A couple of notes about Hallway Con. The Hallway Con experience is impossible to virtualize. You will get information from a virtual conference, but you won’t get the experience. This is a face-to-face, in-person activity only. Despite attempts to recreate this experience, nobody has successfully done it yet. This means if you attend a virtual conference, you miss out on some of the most important value of the conference.
You will get information from a virtual conference, but you won’t get the experience.
Hallway Con has a randomness aspect, and although you can’t purposefully plan it, you can enhance your odds of being successful. Here are a couple of tips.
Catch up with people you know
Are there people you want to talk to or catch up with? Reach out to them and see if you can catch up. You could meet in a common area or have lunch. Anything that might get a conversation started. These people will often introduce you to new people.
Scope out any interesting meetups
Meetups are specifically for this networking purpose. Maximize your opportunities by attending them. It’s good to remember that not all of these meetups are published on the conference schedule. Be on the lookout through various social media platforms as well as with conversations onsite. You can also ask about any meetups through your social media platforms and see what responses you get.
Add known meetups you’d like to attend to your schedule for tracking purposes.
Roam around a common area and sit with strangers
I like to roam around the common area to see if there are people I recognize. These common areas are collecting points. You probably won’t know many people if you are new to the community, but it can still be a good exercise. You can also sit with strangers. I often sit with strangers during conference lunches. It’s a good way to be forced to introduce yourself and spark conversations. You typically find that you have much more in common with these strangers than you thought.
Chat with presenters after their session concludes
Presenters are often also conference attendees. They don’t just deliver their talk and fly out the door. You can talk with them directly after their presentation to ask additional questions or get further clarification, but you can also see them roaming around common areas and having additional conversations. Remember, speakers will likely be more candid off-stage than on, so take advantage of this.
Be prepared to share
Security people absolutely LOVE to talk. So, don’t worry if you are talking to an expert and are only asking questions. However, strangers are more open to sharing with you if you share something with them. This doesn’t have to be in-depth technical information. It could be challenges you are having or things that aren’t working.
Plan Your Evenings
There will undoubtedly be after-hours activities. The bigger conferences have afterparties sponsored by organizations and vendors. Sometimes, you can just walk up and enter, but many of these events require pre-registration. This isn’t something to put off. Research these events and register ahead of time.
These are also opportunities to maximize your networking. You’ll be there with other attendees and speakers. No speaker has ever not enjoyed someone walking up and telling them they enjoyed their presentation. Use it as an opportunity to spark a conversation.
Events and Contests
Security conferences are also about entertainment. There are many reasons you may want to watch Hacker Jeopardy and not actually participate. However, if you want to participate in a contest, you’ll probably need some preparation. There may be pre-registration or qualifying rounds. This may also include other preparations like bringing your computer with tools installed. Do a little research to increase your success.
Remember
Every security conference is different. Just because you attended one and didn’t like it doesn’t mean you won’t like another one. Taking it all in your first year at a larger conference is also okay. Don’t be too hard on yourself. It can be difficult to know what you are in for your first time at a large event. Plan the things you can and learn from the things you can’t. There’s always next year.
Misconceptions Persist
Okay… I’m gonna rant.
It irks me to no end to hear sales and marketing people who have never attended these security events talk about them like they know something about them. You hear things like security leaders and CISOs don’t attend DEF CON, that there’s no value in having someone speak at conferences because it doesn’t directly lead to sales, that security conferences are just an excuse to party, and many other completely out-of-touch statements. These are nonsense perspectives from people who are out of touch with the very community they are supposed to serve. If you are one of these people reading this and think I’m talking about you, then I’m absolutely talking about you.
Throughout my entire career, I’ve tried to educate people about the security community and the value of interaction at conferences. I’ve had both successes and failures. Some salespeople only want to sell firewalls to customers who, in turn, could care less if they bought them from a chatbot. There’s no competing with that.
If you think I’m being harsh, you should see the original draft wording of this section. I’m feeling generous today, so this is me being nice 😊
Rant complete.
Finally
Above all, have fun and enjoy yourself. Don’t stress about trying to make everything you want to see. Oh, and your head hurting from the knowledge you gained and your vocal cords a little raspy from talking so much are symptoms you’re doing it right. Remember, it’s not about what you learn but how you modify and apply it to your own challenges. It’s how you take the information in new directions and make it your own. That’s what this is all about.
In my haste to publish this post, I’m sure I’ve missed things. I’ll leave that to others to fill in any obvious blanks. See you at a security conference soon!
The tidal wave of information on AI use smashes the shoreline daily, nearly all of it universally positive. News stories, analyst reports, and anecdotes all lead you to believe that you are already in the dust, no matter how advanced you are. Your competitors are smoking you, and everyone is using AI for everything successfully except YOU. This is the massive headwind many of us pushing back find ourselves in, constantly bombarded with news stories and analyst reports, all in service of telling us we are mistaken. A congregation was sent to consult the Oracle of Gartner and your perspectives have been found wanting.
In the space we refer to as reality, what we think we know about AI usage is wrong. So, how did we get here? How have we become so misinformed? The answer is pretty simple: humans. Okay, well, more specifically, surveys and interviews.
Surveys and Interviews
It’s long been known that survey data is only slightly more valuable than garbage, but when it comes to AI, survey data can be a fully engulfed dumpster fire. There are several reasons for this, but the primary reason this is so bad in the AI space is that nobody wants to look stupid or appear behind the curve. So when the analyst, survey taker, or journalist calls, people start parroting.
The primary reason this is so bad in the AI space is that nobody wants to look stupid or appear behind the curve.
Instead of responding with observations they’ve made or activities they are actually doing, they respond with something they’ve heard, articles they’ve read, experiments they hope work, and a host of other things that aren’t true activities. This equates to people expressing their vibes. This disconnection leads to an opening chasm with reality. Since surveys and interviews are the primary methods to collect this type of usage data, that doesn’t bode well for determining realities on the ground. With the hype turned up to 11, a red flag would be when your survey results confirm a 10.
I’ve pointed out this parroting vs. observation issue in my presentations at various conferences for the past couple of years. Although this parroting makes for some wildly comical analyst reports and news stories, it’s rough if you’re trying to make decisions based on them, or worse, when your boss expects you to produce a magic wand and summon the guardians of innovation because you are being left in the dust.
A few days ago, I read an article from the Ludic blog making the rounds that contained the following image.
This is an obvious red flag, and the author points this out in much more eloquent and spicy language. We’ve long known that most AI/ML/DL projects don’t make it into production, but all of a sudden, LLMs come along, and 92% of companies are finding great success. It’s not real. Speaking of 92%…
GitHub reported last year that 92% of US-based developers are already using AI coding tools. The gut reaction is this feels wrong, but hey, it must be true if the data confirms it, right? So, let’s do a thought experiment. Imagine standing in the frozen dessert section of the grocery store, asking people if they like ice cream. Now imagine asking everyone buying ice cream if they like it. What if you only asked two people, or five people, or ten people?
When it comes to usage data, what does “using” mean? What is the definition put forth in the survey? What is the makeup of the population? Most importantly, what do they define as “AI”? All of this matters, and it doesn’t take much imagination to realize how incredibly biased survey data can be. The flames are further fanned by the illusion that models have more capabilities than they do and companies faking demos.
For a deeper response to some of the common points people make, read the article I mentioned. I have some quibbles with some of the article’s content, but all in all, it’s a solid read, and the spicy language makes it all the better.
In a previous post on GPT-4 Lowering Conspiracy Beliefs, I addressed some of these issues surrounding surveys and survey data. I called attention to dark data categories that often surface when surveys are used. I also recommended David Hand’s excellent book Dark Data: Why What You Don’t Know Matters. The book will change the way you view surveys.
The unfortunate reality is that quite a few people have a vested interest in perpetuating these misconceptions. You’d think this would be the companies building these products since it increases their revenue, and this is certainly happening, but most of them aren’t affiliated with these companies. They want to be seen as the ones with the knowledge. They are influencers trying to drive people to their funnels and people in the tech industry who don’t want to look clueless. It’s hard for people to call you out on something when you are saying the same thing everyone else is saying.
Another red flag was shortly after ChatGPT was released. We were inundated with articles quoting opinions by leaders and executives who had never used the technology and had no idea how it worked or even what it was capable of. But it seemed as though we couldn’t get enough.
Dumpster fire achieved.
Ask Questions
We aren’t helpless in these cases. One of the best defenses is asking follow-up questions and probing beneath the surface. I know, I know. We pay (INSERT ORG HERE) a lot of money, and they say… But bear with me a moment.
One recent technique I’ve used is marking up reports, slides, and other information sent to me to help people focus on obvious issues and force some deeper thought. This gives others an idea of where I’m coming from and helps plant the seeds of these questions in people’s heads. Typically, these reports create more questions than they answer, and responding with, “This is dumb,” is not the best tactic. Here’s a recent example I used for a report discussing GenAI’s security use in 2024.
Along with this markup, I also included data in the email questioning the statistical makeup of the data used in the analysis. Funny enough, for this particular section, there was no information about the sample size, industry verticals, or other important information about the makeup of the sample. This is always a red flag. Maybe it was mentioned somewhere else, and I missed it, but it wasn’t available in this section like in the others.
Often, even asking a simple question, “How” can be super effective.
“Generative AI is completely transforming X business or process.” “Oh yeah? How?”
The questions of how, what, and where can be your ultimate weapons in defense against some of this contradictory data. They inform you if there is something real and help you understand if the use cases proposed to support the strongly worded statements made. There may be good answers to these questions that you may want to consider. There are legitimate use cases, and you do want to stay ahead of the curve, so being better informed helps you take advantage of opportunities.
Misunderstanding the data has negative impacts, putting further strain on your resources to create competing solutions or wasting time trying to recreate something that isn’t even working in the first place. Even if another organization successfully uses generative AI for a task or process, you might be unable to replicate it due to different applications, systems, data, and processes.
Even if another organization successfully uses generative AI for a task or process, you might be unable to replicate it due to different applications, systems, data, and processes.
I’m not bashing analysts or survey takers. Conducting surveys without influencing the outcome is hard. That’s why you can find surveys that confirm just about anything. I’m sure the people writing these reports believe what they write, and it matches the data they have.
Conclusion
The grouping of technologies under the umbrella of AI is certainly useful, yes, even LLMs. Non-generative approaches and more traditional ML and DL have been deployed to solve challenging problems for decades. These approaches are already baked into the systems we use. However, the hype and hysteria throw off any real perception, and you often find that complete transformation aligns more with hopes than realities. Ask the right questions and probe deeper to ensure you are making decisions on the right insights. Find use cases of your own and perform your own experiments. You’ll quickly see what’s working and what’s not.
New, deeply integrated AI-powered productivity tools are on the horizon. A recent example is Microsoft’s Recall, but others are also emerging. For example, there’s Limitless.ai, and if you are feeling particularly nostalgic for Catholicism, there’s the 01 Light from Open Interpreter, which allows you to control your computer remotely through a communion wafer.
All of these tools promise infinite productivity boosts. Just thrust them deep into your systems and watch the magic happen. However, when you watch the demo videos and use cases, it’s easy to understand why most people scratch their heads—just as they did with the Humane Pin and the Rabbit. At this point, they are just setting fire to VC money, hoping that a use case will rise from the ashes.
All joking aside, the tools and their usefulness aren’t the subject of this post. I want to focus on the architectural shift and new exposures we create with these tools. This trend will continue regardless of the use case, tech company, or startup.
Note: I’m on vacation and haven’t followed up on Apple’s AI announcements from WWDC, hence the lack of mention here. I wrote most of this post before leaving on vacation.
New High-Value Targets
One of the things that saves us when we have a breach is that all of our data is rarely collected in a single place. Even in particularly bad breaches, let’s say, of your financial institution, there isn’t also data about your healthcare records, GPS location, browser history, etc. Our world is filled with disparate and disconnected data sources, and this disconnection provides some benefits. This means that breaches may be bad but not as bad as they could have been.
A simple way of looking at it is to say our digital data reality consists of web, cloud, and local data. But even in these different categories, there’s still plenty of segmentation. For example, it’s not like website A knows you have an account on website B. Even locally on your computer or device, application A might not know that application B is installed and much less have access to its data. There are exceptions to this, like purposeful integrations between sites, SSO providers, etc., but the point holds for the most part.
With new personal AI systems, we are about to centralize much of this previously decentralized data, collapsing divisions between web, cloud, and local data, making every breach more impactful. The personal AI paradigm potentially makes all data local and accessible. But it gets worse. This new centralized paradigm of personal AI mixes not only sensitive and non-sensitive data but also trusted and untrusted data together in the same context. We’ve known not to do this since the dawn of information security.
This new centralized paradigm of personal AI mixes not only sensitive and non-sensitive data but also trusted and untrusted data together in the same context.
It’s known with the generative AI systems today that if you have untrusted data in your system, you can’t trust the output. People have used indirect prompt injection attacks to compromise all sorts of implementations. We are now discarding this knowledge, giving these systems more access, privileges, and data. Remember, breaches are as bad as the data and functionality exposed, and we are removing the safety keys from the launch button.
How Centralization Happens
I’ve talked about centralizing data at a high level, but what does that look like in practice? Let’s illustrate this with a simple diagram.
We can envision our three buckets of web, cloud, and local data tied together through a connection layer. This layer is responsible for the connections, credentials, login macros, schedulers, and other methods to maintain connections with applications and data sources. The connection layer allows data from all of these sources to be collected locally for the context necessary for use with the LLM. This can either be done at request time or proactively collected for availability. This connection layer creates a local context that threads down the segmentation between the data sources.
The implementation specifics will depend on the tool, and new tools may implement new architectures. So, it’s helpful to back up and consider what’s happening with these tools. We have a tool on our systems that runs with elevated privileges, needs access to a wide variety of data, and takes actions on our behalf. In theory, these systems could access all the same things we have access to. This is our starting point.
These systems will have access to external data, such as cloud and web data and local system data (data on your machine). Your system could collect data from log files, outputs from applications, or even things such as browser history. Of course, they may also have additional logging, such as recording all activity on your system, like Microsoft’s Recall feature, and storing it neatly in a plain text database which now, due to backlash, has caused changes and now, delays.
Having access to data is only one piece of the puzzle. These systems need to contextualize this information to actually do something with it. Your data will need to be both available and readable. This means it’ll need to be collected for this contextualization.
For example, if you ask your personal AI a question like:
What is the best way to invest the amount of money I have in my savings account, according to the Mega Awesome Investment Strategy?
The LLM needs two specific pieces of context to begin formulating an answer to the question. It needs to know how much money you have in your savings account and what the Mega Awesome Investment Strategy is. The LLM queries your financial institution to pull back the amount of money in your savings account. It then needs data about the strategy. Maybe it invokes a web search to find the result and use that as part of the context (let’s ignore all the potential pitfalls of this for a moment.) It uses these two pieces of data as context, either sending them off to a cloud-hosted LLM or using a local LLM.
The data can be queried at runtime or periodically synced to your computer for speed and resistance to service downtime. All this data, including synced data, credentials, previous prompts, and much more, will be stored locally on your system and possibly synced to the cloud. Since this data needs to be readable for LLMs, it will most likely be stored in plaintext, counting on other controls to provide protection. Your most sensitive data is collected in a single place, conveniently tied together, waiting for an attacker to compromise it.
Even scarier, we will get to a point where we can run this query:
Implement the Mega Awesome Investment Strategy with the money I have in my savings account.
This will leave us with systems that not only use the collected data but also take action on our behalf—operating as and taking action as us. I’ve mentioned before that we are getting to a point where we may never actually know why our computers are doing anything, accessing the files they are, or even taking the actions they take. This condition makes our computers far more opaque than they are today.
This example was just a simple question with one piece of financial data, but these systems are generalized and will have context for whatever data sources are connected. There will be a push to connect them to everything. Healthcare, browsing data, emails, you name it, all stored conveniently in a single place, making any breach far worse. It’s like collecting all the money from the regional vaults and putting it behind the window in front of the main bank.
There’s Gold In That Thar Data
If data is gold, this is an absolute gold mine. As a matter of fact, this data is so valuable it will be hard for companies to keep their hands off of it in a new data gold fever. So, although up to this point, I’ve been talking about malicious attackers having access to this data, it’s also the case that tech companies will want this data as well, and all efforts will be made to access it and use it. This will be through both overt and covert methods. Turning settings on by default, fine print in user agreements, etc.
If you think the startup developing the tool says they respect my privacy and won’t use this data for anything, think again. Even if this statement were true, wait until they get acquired.
Conclusion
First things first, we need to ask what we get from these integrations. Are the benefits worth the risks of security and privacy exposures created by these new high-value targets? The answer to this question will be a personal choice, but for a vast majority, the answer will be no. At this point, there is still more hype than help.
Authentication, authorization, and data protection need to be key in these new architectures. Not only that, but we must put our own guardrails in place to protect our most sensitive data. This is all going to be additional work for the end user. These systems act as us accessing our most sensitive data. Anyone able to interact with them is basically us. There are no secrets between you and your personal AI. Companies also need to ensure that users understand the potential dangers and pitfalls and provide the ability to turn these features off.
There are no secrets between you and your personal AI.
Tech companies must start taking this problem seriously and acknowledging the new high-value targets they create with these new paradigms. If they are going to shove this technology into every system, making it unavoidable, then it needs to have a bare minimum level of safety and security. It’s one of the reasons I’ve been harping on my SPAR categories as a baseline starting point.
