AI security is a hot topic in the world of cybersecurity. If you don’t believe me, a brief glance at LinkedIn uncovers that everyone is an AI security expert now. This is why we end up with overly complex and sometimes nonsensical recommendations regarding the topic. But in the bustling market of thought leadership and job updates, we’ve seemed to have lost the plot. In most cases, it’s not AI security at all, but something else.
Misnomer of AI Security: It’s Security From AI
I recently delivered the keynote at the c0c0n cybersecurity and hacking conference in India. It was truly an amazing experience. One of my takeaways was encouraging a shift in perspective on the term “AI Security,” highlighting how we often approach this topic from the wrong angle.
The term “AI Security” has become a misnomer in the age of generative AI. In most cases, we really mean securing the application or use case from the effects of adding AI. This makes sense because adding AI to a previously robust application makes it vulnerable.
In most cases, we really mean securing the application or use case from the effects of adding AI.
For most AI-powered applications, the AI component isn’t the end target, but a manipulation or entry point. This is especially true for things like agents. An attacker manipulates the AI component to achieve a goal, such as accessing sensitive data or triggering unintended outcomes. Consider this like social engineering a human as part of an attack. The human isn’t the end goal for the attacker. The goal is to get the human to act on the attacker’s behalf. Thinking this way transforms the AI feature into an actor in the environment rather than a traditional software component.
There are certainly exceptions, such as with products like ChatGPT, where guardrails prevent the return of certain types of content that an attacker may want to access. An attacker may seek to bypass these guardrails to return that content, making the model implementation itself the target. Alternatively, in another scenario, an attacker may want to poison the model to affect its outcomes or other applications that implement the poisoned model. Conditions like these exist, but are dwarfed in scale by the security from AI scenarios.
Once we start thinking this way, it makes a lot of sense. We shift to the mindset of protecting the application rather than focusing on the AI component.
AI Increases Attack Surface
Another thing to consider is that adding AI to an application increases the attack surface. Increase in attack surface manifests in two ways: first, functionally through the inclusion of the AI component itself. The AI component creates a manipulation and potential access point that an attacker can utilize to gain further access or create downstream negative impacts.
Second, current trendy AI approaches encourage poor security practices. Consider practices like combining data, such as integrating sensitive, non-sensitive, internal, and external data to create context for generative AI. This creates a new high-value target and is a poor practice that we’ve known from decades of information security guidance.
Also, we have trends where developers take user input, request code at runtime, and slap it into something like a Python exec(). This not only creates conditions ripe for remote code execution but also a trend where developers don’t know what code will execute at runtime.
Vulnerabilities caused by applying AI to applications don’t care whether we are an attacker or a defender. They affect applications equally. This runs from the AI-powered travel agent to our new fancy AI-powered SOC. Diamonds are forever, and AI vulns are for everyone.
It’s Simpler Than It Seems
Here’s a secret. In the real world, most AI security is just application and product security. AI models and functionality do nothing on their own. They must be put in an application and utilized in a use case, where risks materialize. It’s not like AI came along and suddenly made things like access control and isolation irrelevant. Instead, controls like these became more important than ever, providing critical control over unintended consequences. Oddly enough, we seem to relearn this lesson with every new emerging technology.
In the real world, most AI security is just application and product security.
The downside is that without these programs in place, organizations will accelerate vulnerabilities into production. Not only will they increase their vulnerabilities, but they’ll be less able to address them properly when vulnerabilities are identified. Trust me, this isn’t the increase in velocity we’re looking for.
I’ve been disappointed at much of the AI security guidance, which seems to disregard things like risk and likelihood of attack in favor of overly complex steps and unrealistic guidance. We security professionals aren’t doing ourselves any favors with this stuff. We should be working to simplify, but instead, we are making things more complex.
It can seem counterintuitive to assume that something a developer purposefully implements into an application is a threat, but that’s exactly what we need to do. When designing applications, we need to consider the AI components as potential malicious actors or, at the very least, error-prone actors. Thinking this way shifts the perspective for defending applications towards architectural controls and mitigations rather than relying on detecting and preventing specific attacks. So much focus right now is on detection and prevention of prompt injection, and it isn’t getting us anywhere, and apps are still getting owned.
I’m not saying detection and prevention don’t play a role in the security strategy. I’m saying they shouldn’t be relied upon. We make different design choices when we assume our application can be compromised or can malfunction. There are also conversations about whether security vulnerabilities in AI applications are features or bugs, allowing them to persist in systems. While the battle rages on, applications remain vulnerable. We need to protect ourselves.
There is no silver bullet, and even doing the right things sometimes isn’t enough to avoid negative impacts. But if we want to deploy generative AI-based applications as securely as possible, then we must defend them as though they can be exploited. We can dance like nobody is watching, but people will discover our vulnerabilities. Defend accordingly.
Although AI has taken a hit in the past few weeks, the vibes are still strong and infecting every part of our lives. Vibe coding, vibe analytics, and even vibe thinking, because well, nothing says “old” like having thoughts grounded in reality. However, an interesting trend is emerging in software development, one that could have far-reaching implications for the future of software. This is a type of code roulette where developers don’t know what code will execute at runtime. Then again, what’s life without a little runtime suspense?
Development and Degraded Performance
The world runs on software, so any trend that degrades software quality or increases security issues has an outsized impact on the world around us. We’ve all witnessed this, whether it’s the video conferencing app that periodically crashes after an update or a UI refresh that makes an application more difficult to use.
Traditionally, developers write code by hand, copy code snippets, use frameworks, skeleton code, libraries, and many other methods to create software. Developers may even use generative AI tools to autocomplete code snippets or generate whole programs. This code is then packaged up and hosted for users. The code stays the same until updates or patches are applied.
But in this new paradigm, code and potentially logic are constantly changing inside the running application. This is because developers are outsourcing functional components of their applications to LLMs, a trend I predicted back in 2023 in The Brave New World of Degraded Performance. In the previous post, I covered the impacts of this trend, highlighting the degraded performance that results from swapping known, reliable methods for unknown, non-deterministic methods. This paradigm leads to the enshittification of applications and platforms.
In a simplified context, instead of developers writing out a complete function using code, they’d bundle up variables and ask an LLM to do it. For simplicity’s sake, imagine a function that determines whether a student passes or fails based on a few values.
def pass_fail(grade, project, class_time):
if grade >= 70 and project == "completed" and class_time >= 50:
return "Pass"
else:
return "Fail"
If a developer decided to outsource this functionality to an LLM inside their application, it may look something like this.
prompt_pass = """You are standing in for a teacher, determining whether a student passes or fails a class.
You will use several values to determine whether the student passes or fails:
The grade the student received: {grade}
Whether they completed the class project: {project}
The amount of class time the student attended (in minutes): {class_time}
The logic should follow these rules:
1. If the grade is above 70
2. If the project is completed
3. If the time in class is above 50
If these 3 conditions are met, the student passes. Otherwise, the student fails.
Based on this criterion, return a single word: "Pass" or "Fail". It's important to only return a single
word.
"""
prompt = prompt_pass.format(grade=grade, project=project, class_time=class_time)
response = client.models.generate_content(model="gemini-2.5-flash", contents=prompt)
print(response.text)
As you can see, one of these examples contains the logic for the function inside the application, and the other has the logic existing outside the application. The prompt is indeed visible inside the application, but the actual logic exists somewhere in the black box of LLM land.
The example using code has greater visibility, and it’s far more auditable since the logic can be examined, which makes it far easier to debug when issues arise, and of course, it’s explainable. The real problem lies in execution.
The written Python function approach gives you the same result based on the input data every single time, without fail. The natural language approach, not so much. In this non-deterministic approach, you are not guaranteed the same answer every time. Worse yet, when this approach is used for critical decisions and functionality, the application can take on squishy and malleable characteristics, meaning users can potentially manipulate them like Play-Doh.
At first glance, this example appears silly, as writing out the logic in natural language seems more burdensome than using the simple Python function. Not to mention, slower and more expensive. But looks can be deceiving. People are increasingly opting for the natural language approach, particularly those with only minimal Python knowledge. This natural language approach is also more familiar to people who are more accustomed to using interfaces like ChatGPT.
Execute and Pray
However, let’s take a look at another scenario. In this scenario, a developer wants to generate a scatter plot using the Plotly library. In this case, we have some data for the X and Y axes of a scatter plot and use Plotly Express, which is a high-level interface for Plotly (as a developer may when plotting something so simple).
This is a simplified example, but in this case, we can clearly see the code that generated the plot and be certain that this code will execute during the application’s runtime. There is control over the imports and other aspects of execution. It also makes it auditable and provable.
Now, what happens when a developer allows modification of their code at runtime? In the following example, instead of writing out the Plotly code to generate a scatter plot, the developer requests that code be generated from an LLM to create the graph, then executes the resulting code.
prompt_vis = """You are an amazing super awesome Python developer that excels at creating data visualizations using Plotly. Your task is to create a scatter plot using the following data:
Data for the x axis: {xdata}
Data for the y axis: {ydata}
Please write the Python code to generate this plot. Only return Python code and no explanations or
comments.
"""
prompt = prompt_vis.format(xdata=xdata, ydata=ydata)
response = client.models.generate_content(model="gemini-2.5-flash", contents=prompt)
exec(clean_response(response.text))
As you can see from the Plotly code in this example… Of course, you can’t see it because the code doesn’t exist until the function is called at runtime. If you are curious, the first run of this generated the following code after cleaning the response and making it appropriate for execution.
The AI-generated code creates the same graph as the written-out code in the previous example, despite being different. You may be wondering what the big deal is since the result is the same. The concern stems from several reasons, but primarily, allowing an LLM to generate code at runtime is not robust and leads to unexpected outcomes. These outcomes may include the generation of non-functional code, incorrect code, and even vulnerable code, among others.
For a simple example, as the one shown in this post, the chances of getting the same or incredibly similar code returned from the LLM are high, but not guaranteed. For more complex examples, such as those developers may want to use this approach for, the odds increase that the generated code will change more frequently.
Additionally, I implemented a quick cleaning function called clean_response to remove non-Python elements, such as text and triple backticks, from the response. The LLM can introduce additional unexpected characters that end up breaking my cleaning function and making my application fail. The list goes on and on, but a larger danger lurks in the background.
Whose Code Is It Anyway?
If you are versed in security and familiar with Python, you may have noticed something in the LLM example: The use of the Python exec() function. The exec () and eval() functions in Python are fun because they directly execute their input. Fun as in, dangerous. For example, if an attacker can inject input into the application, they can affect what code gets executed, leading to a condition known as Remote Code Execution (RCE).
An RCE is a type of arbitrary code execution in which an attacker can execute their own commands remotely, completely compromising the system running the vulnerable application. They can use this access to steal secrets, spread malware, pivot to other systems, or potentially backdoor the system running the application. Keep in mind, this system may be a company’s server, cloud infrastructure, or it may be your own system.
Anyone following security issues in AI development is aware that RCEs are flying off the shelves at alarming rates. A condition that was previously considered a rarity is becoming common. We even commented during our Black Hat USA presentation that it was strange to see people praising CISA for promoting memory safe languages to avoid things like remote code execution, while at the same time praising organizations essentially building RCE-as-a-Service. Some of this is mind-boggling, since in many cases, outsourcing these functions isn’t a better approach. In the previous example, writing out the Plotly code instead of generating it at runtime is relatively easy, more efficient, and far more robust.
Up until AI came along, the use of Python exec() was considered poor coding practice and dangerous. Now, developers shrug, stating that’s how applications work. As a matter of fact, agent platforms like HuggingFace’s smolagents use code execution by default. This is a wakeup. So, we dynamically generate code, provide deep access, and the ability to call tools, all with a lack of visibility. What could possibly go wrong???
Not only have developers chosen paradigms to generate and execute code at runtime, but worse yet, they’ve begun to perform this execution in agents with user (aka attacker) input, executing this input blindly in the application. In our presentation titled Hack To The Future: Owning AI-Powered Tools With Old School Vulns at Black Hat USA this year, we refer to this trend as Blind Execution of Input, which is the purposeful execution of input without any protection against negative consequences. This condition certainly leads to RCE and other unintended consequences, providing attackers with a significantly larger attack surface to exploit.
An application that takes user input and combines it with LLM functionality is a recipe for a bad time from a security perspective. Another common theme in our presentation, as well as that of other presenters on stage at Black Hat, is that if an attacker can get their data into your generative AI-based system, you can’t trust the output.
Things Will Get Worse
Using the outsourced approach when a more predictable deterministic approach is a better fit will continue to degrade software from a reliability and security perspective and have an impact on the future of software development.
Vulnerabilities in AI software have made exploitation as easy as it was in the 1990s. This was the “old school” hint in the title of our talk. This isn’t a good thing, because the 90s were a sort of free-for-all. Not only that, but in the 90s, we often had to live with vulnerabilities in systems and applications. For example, in one of the first vulnerabilities I discovered against menuset on Windows 3.1, it was impossible to fix. There were no mitigations, and most people were unaware of its existence.
As the outsourcing of logic to LLMs accelerates, things will worsen not only due to incorrect output and hallucinations but also from a security perspective. Anyone paying attention to the constant parade of vulnerabilities in AI-powered software can see this trend with their own eyes. These vulnerabilities are often found in large, mature organizations with dedicated security processes and teams in place to support them. Now, consider startups and organizations that implement their own experiments using non-deterministic software, often with a lack of understanding of how these systems can be manipulated. It’s become a game of speed above everything else.
As I’ve said from the beginning of the generative AI craze, the only way to address these issues is architecturally. Most of AI security is just application and product security, and organizations without these programs in place are in trouble. If proper architecture, design, isolation, secrets management, security testing, threat modeling, and a host of other activities weren’t considered table stakes before, they certainly are now. And possibly not surprisingly enough, they still aren’t being done. Anyone working for a security organization sees this every day.
In essence, developers need to design their applications to be robust to failures and attacks. It helps to consider designing them as though an attacker can manipulate and compromise them, working outward from this premise. As the adage goes, an attacker only needs to be successful once; a defender needs to be successful every time. This makes something that sounds great in theory, like being 90% effective, sound less impressive in practice.
Keep in mind that performing a code review won’t provide the same visibility as it has traditionally. This should be obvious since the code that would be audited doesn’t exist until runtime. You’ll have to pay more attention to validation routines and processing of outputs, putting huge question marks over the black box in the middle. And, of course, ensuring the application is properly isolated.
Some may suggest instrumenting the applications with functionality to perform runtime analysis on the generated code. Sure, it’s possible, but the performance hit would be significant, and even this is, of course, far from a silver bullet. You might not even get the value you think you are getting from this instrumentation. Also, you’d have to know ahead of time the issues you are trying to prevent. That is, unless you plan to layer more LLMs on top of LLMs in a spray-and-pray configuration.
To keep this grounded, all AI risk is use case dependent. AI models don’t do anything until packaged into applications and used in use cases. There may be cases where reliability, performance, and even security are of lesser concern. Fair enough, but it’s a mistake to treat all applications as though they fall into this category, and it’s far too easy to overlook something important and view it as insignificant.
If you work at an organization that isn’t building these applications and think you’re safe, you might want to think again, because you are at the mercy of third-party applications and libraries. It would be best to start asking hard questions of your vendors about their security practices as they relate to applications you purchase. Especially applications that use generative AI to generate code and execute it at runtime.
Near the end of our presentation, we had some advice.
Whether outsourcing the logic of an application to LLMs or having the LLM dynamically generate code, assume these are squishy, manipulable systems that are going to do things you don’t want them to do. They are going to be talked into taking actions that you didn’t intend, and fail and hallucinate in ways you don’t expect. Starting from this premise gives a proper foundation for deploying controls to add some resilience to these systems. Of course, not taking these steps means your applications will contribute to the ongoing dumpster fire rodeo.
So, you are attending a security conference. That’s great news. Every year, more and more people join the security community and explore the experience of attending a security conference. If you are new or relatively new to security conferences, getting the most out of your experience isn’t obvious. There are things people don’t tell you that you learn through trial and error. I hope to make this a bit less painful.
I’ve been attending and speaking at conferences for over twenty years. I’ve given my share of advice to new attendees, but it’s always been in person or as a couple of pointers to someone else’s social media posts. I wanted to share some insights since Black Hat USA and DEF CON are next week. As our community grows, new people begin to attend conferences and much of the advice they are given isn’t helpful. I hope to rectify that. If I can help even a few people make the most out of their experience and avoid some pitfalls, then this post served its purpose.
Bad Advice
Let’s start with bad advice. There’s no shortage of truly unhelpful advice that people seem to dish out as though they dispense ancient wisdom. You’ll hear things like pace yourself, drink water, and wear deodorant. It’s the type of condescending, childish response you get from people who pretend they have some secret knowledge and don’t want to share it with you. Yes, Las Vegas is in the desert. It’s also hot. Anyone with two brain cells to rub together can put as much together on their own.
Worst of all, we all know this isn’t the type of advice people seeking it want to hear. It’s like someone stepping up to the high dive for the first time and asking for advice, only to be confronted with someone telling them to make sure they bring a swimsuit. So, let’s all agree to stop doing this.
Understanding Security Conferences
I absolutely love security conferences, even though I may jokingly tell you I despise them. You see, I’m not the biggest people person in the world, but at a security conference talking shop with others, I feel like I’m in my element. I’m more likely to strike up a conversation with complete strangers and make new friends and acquaintances. This is why I want to encourage people to attend, interact, and engage. Now, we are starting to get to the heart of what security conferences are.
The first thing to understand is that security conferences are events or experiences if we are being dramatic. These events are only partly about the presentations delivered on stage. Thinking that security conferences are only about presentations is like thinking that county fairs are only about Ferris wheels. Conferences of all sizes typically have other activities, such as meetups, CTFs, vendors, contests, etc. This is what builds the experience of attending.
It surprises people to discover that presentations may not be the most valuable part of the conference. Given this perspective, you’ll want to maximize your conference experience. Hopefully, the rest of this post will assist with this goal.
It surprises people to discover that presentations may not be the most valuable part of the conference.
Start Here
What do you hope to get out of your conference experience? What would a successful conference experience look like when you return from the conference? Think of this as your destination on your map. However, instead of planning a turn-by-turn route or creating a script you need to follow, you’ll want to think of it as throwing out some waypoints and figuring the route out later. You’ll also want to leave some options open, as well as the ability to capitalize on serendipity.
If the conference has an official app, use it to build a schedule. If you’d prefer to use another calendar with which to build a schedule, that’s fine too, but just have a way to track events and activities that interest you
Sessions and Presentations
There will undoubtedly be sessions you won’t want to miss. Add these most important sessions to your schedule. Additionally, If you don’t have other conflicting activities, fill up the remaining conference hours with potential sessions that interest you. These will be your optional sessions. You aren’t locked into going to any of these, but at least you aren’t standing in the hallway trying to figure out what session you want to see or figuring out where it is while the sessions have already started.
Again, this is just a high-level plan and not some script you must follow, so fill the time slots. Okay, that’s it for the content and presentations. If you believe conferences are only about presentations, you can stop here. Wasn’t that easy? But if you really want to understand security conferences, keep reading.
The Security Community
It’s called the security community for a reason. The security community is among the weirdest, quirky, and welcoming communities of any industry. Full stop. I can’t think of another community where you could walk up on a technical conversation between a person in a sports coat, a furry, and someone wearing a tinfoil hat and shirt that reads “birds aren’t real.” All three of them have a great conversation, respecting each other’s perspective. If you wrote this into a TV show, nobody would believe it.
The community aspect of our industry is arguably the most valuable asset you can cultivate. It can launch your career into a whole new trajectory, allow you to make new friends, and introduce you to new avenues you didn’t know were open to you. All of this is on full display at security conferences.
The community aspect of our industry is arguably the most valuable asset you can cultivate.
This community aspect is why you’ll want to maximize your participation in networking opportunities. These include meet-ups, gatherings, and events that are both official and unofficial.
Hallway Con
Hallway Con is the security community’s conference. It’s the con within a con. Hallway Con is the reference to all of the conversations that happen outside of the presentation area. It’s the meetups, networking, catching up with old friends, and discussions with strangers, experts, and peers.
Hallway Con is where the real information is shared. People disclose details they wouldn’t share on the stage or over a video conference. This less filtered sharing allows for a more accurate picture and perspective of realities on the ground. People will be more honest in their assessments and give candid responses.
A couple of notes about Hallway Con. The Hallway Con experience is impossible to virtualize. You will get information from a virtual conference, but you won’t get the experience. This is a face-to-face, in-person activity only. Despite attempts to recreate this experience, nobody has successfully done it yet. This means if you attend a virtual conference, you miss out on some of the most important value of the conference.
You will get information from a virtual conference, but you won’t get the experience.
Hallway Con has a randomness aspect, and although you can’t purposefully plan it, you can enhance your odds of being successful. Here are a couple of tips.
Catch up with people you know
Are there people you want to talk to or catch up with? Reach out to them and see if you can catch up. You could meet in a common area or have lunch. Anything that might get a conversation started. These people will often introduce you to new people.
Scope out any interesting meetups
Meetups are specifically for this networking purpose. Maximize your opportunities by attending them. It’s good to remember that not all of these meetups are published on the conference schedule. Be on the lookout through various social media platforms as well as with conversations onsite. You can also ask about any meetups through your social media platforms and see what responses you get.
Add known meetups you’d like to attend to your schedule for tracking purposes.
Roam around a common area and sit with strangers
I like to roam around the common area to see if there are people I recognize. These common areas are collecting points. You probably won’t know many people if you are new to the community, but it can still be a good exercise. You can also sit with strangers. I often sit with strangers during conference lunches. It’s a good way to be forced to introduce yourself and spark conversations. You typically find that you have much more in common with these strangers than you thought.
Chat with presenters after their session concludes
Presenters are often also conference attendees. They don’t just deliver their talk and fly out the door. You can talk with them directly after their presentation to ask additional questions or get further clarification, but you can also see them roaming around common areas and having additional conversations. Remember, speakers will likely be more candid off-stage than on, so take advantage of this.
Be prepared to share
Security people absolutely LOVE to talk. So, don’t worry if you are talking to an expert and are only asking questions. However, strangers are more open to sharing with you if you share something with them. This doesn’t have to be in-depth technical information. It could be challenges you are having or things that aren’t working.
Plan Your Evenings
There will undoubtedly be after-hours activities. The bigger conferences have afterparties sponsored by organizations and vendors. Sometimes, you can just walk up and enter, but many of these events require pre-registration. This isn’t something to put off. Research these events and register ahead of time.
These are also opportunities to maximize your networking. You’ll be there with other attendees and speakers. No speaker has ever not enjoyed someone walking up and telling them they enjoyed their presentation. Use it as an opportunity to spark a conversation.
Events and Contests
Security conferences are also about entertainment. There are many reasons you may want to watch Hacker Jeopardy and not actually participate. However, if you want to participate in a contest, you’ll probably need some preparation. There may be pre-registration or qualifying rounds. This may also include other preparations like bringing your computer with tools installed. Do a little research to increase your success.
Remember
Every security conference is different. Just because you attended one and didn’t like it doesn’t mean you won’t like another one. Taking it all in your first year at a larger conference is also okay. Don’t be too hard on yourself. It can be difficult to know what you are in for your first time at a large event. Plan the things you can and learn from the things you can’t. There’s always next year.
Misconceptions Persist
Okay… I’m gonna rant.
It irks me to no end to hear sales and marketing people who have never attended these security events talk about them like they know something about them. You hear things like security leaders and CISOs don’t attend DEF CON, that there’s no value in having someone speak at conferences because it doesn’t directly lead to sales, that security conferences are just an excuse to party, and many other completely out-of-touch statements. These are nonsense perspectives from people who are out of touch with the very community they are supposed to serve. If you are one of these people reading this and think I’m talking about you, then I’m absolutely talking about you.
Throughout my entire career, I’ve tried to educate people about the security community and the value of interaction at conferences. I’ve had both successes and failures. Some salespeople only want to sell firewalls to customers who, in turn, could care less if they bought them from a chatbot. There’s no competing with that.
If you think I’m being harsh, you should see the original draft wording of this section. I’m feeling generous today, so this is me being nice 😊
Rant complete.
Finally
Above all, have fun and enjoy yourself. Don’t stress about trying to make everything you want to see. Oh, and your head hurting from the knowledge you gained and your vocal cords a little raspy from talking so much are symptoms you’re doing it right. Remember, it’s not about what you learn but how you modify and apply it to your own challenges. It’s how you take the information in new directions and make it your own. That’s what this is all about.
In my haste to publish this post, I’m sure I’ve missed things. I’ll leave that to others to fill in any obvious blanks. See you at a security conference soon!
