I’ve had quite a few conversations about the OpenAI presentation at Black Hat over the past couple of days. Let’s just say my feedback isn’t positive. I didn’t immediately post about it due to travel, responsibilities, and, well, the fact that I still have to work. I know, isn’t AI supposed to relieve us of work so we can spend more time yapping on the Internet? I hate it when reality confounds utopia.
Anyway, here’s a couple of my quickly written thoughts.
Felony Humble Bragging
It’s turned into a hot AI lab breakout summer, and the only way to cool off, it seems, is to announce that your experiments have not only broken out of containment but also broken the law. After OpenAI’s public admission that it had hacked Hugging Face, Anthropic and Meta followed suit with their own brand of “we did it too.”
Rather than keep their mouths shut and hope nobody noticed, these organizations grabbed a megaphone and blasted it out to everyone, trying to get their own slice of the attention pie. Not doing it in the spirit of transparency and openness, but in a gesture of pure marketing hype. However, nobody seemed to care much about them.
During the final keynote (called a locknote) at Black Hat USA, I referred to this situation as felony humble bragging. Look at how powerful our models are. They are so powerful they don’t even listen to us and have no problem breaking into other people’s systems. For a fee, you can have this power too!
In the spirit of responsibility, let’s play a game called “What if a human did it?” If a human had done these attacks, there’d be some kind of consequences, but since an AI lab did it, everyone seems cool with it. The victim in the OpenAI case, Hugging Face, seemed absolutely giddy that they’d been hacked, ecstatic with the attention, milking it for all it’s worth.
In OpenAI’s initial write-up, Hugging Face sounded like someone who’d won an award rather than someone who was hacked. Clem even joked on Twitter about flying to SF to chat with the rogue agent.
One reason Hugging Face was so happy was that this entire event fit their narrative. Hugging Face had to resort to using open models to investigate the incident due to refusals in commercial models. So, the perceived power of AI and the necessity for an open model equal a win for them.
The OpenAI Presentation
What OpenAI did was create a vacuum, starving the narrative of detail. Their initial write-up was a pure marketing exercise, which left people wanting more. This condition draws attention to any further information that OpenAI releases. This is one of the reasons the YouTube video of the talk has over 430k views to date.
I said before the conference that OpenAI would get onstage, give a presentation with minimal detail, and basically use it for marketing. That’s exactly what happened. They plucked a few tidbits from the hack, made a couple of jokes, and then shared takeaways that everyone already knows. Ultimately positioning it as, we know we did this, but you need more of us! They used the presentation as a victory lap, hinting at the intentions for a new product.
They used the presentation as a victory lap.
The big thing everyone is talking about is that the agents set up a message board to chat and share data with each other. Yeah, cool, but even that isn’t as novel as it appears. If people remember, back in 2017, Facebook got press over experiments in which two bots created their own language to communicate. Nature finds a way. Apparently, so do AI experiments.
The other thing is that there was absolutely no mention of cost. OpenAI stood up on stage and made the capability sound like any high school kid with a laptop now has this power, but by OpenAI’s own admission, this consumed a significant number of tokens on a newer model. AKA expensive. I certainly hope cost is addressed in a future detailed write-up, but we’ll most likely never know.
On July 24th, given the vast valley of unknowns, I posed four possible scenarios for the OpenAI Hugging Face hack:
- Scenario 1: The whole thing was a publicity stunt
- Scenario 2: They noticed the experiment going off the rails and decided to see where it went, hoping later to use it for publicity
- Scenario 3: Operational error, oversight, or poor configuration
- Scenario 4: Exactly as they claim, and this is some novel emergence of capability
I mentioned that there was a rather high probability that the reality fell somewhere between scenarios 1 through 3. It now seems more likely that it’s closer to 3. However, there are still many unknowns.
In the past, labs would have configured experiments responsibly and turned them off when they went off the rails. But we aren’t in the turn-it-off era anymore; we are in the turn-it-up era. The see-where-it-goes era. The felony humble brag era.
We aren’t in the turn-it-off era anymore; we are in the turn-it-up era.
To their credit, the talk was entertaining. The jokes they made were funny, like when they reached out to Hugging Face to ask if they were affected by the attack. It can be hard sometimes to disassociate the entertainment from the true value. So, someone who was entertained by the talk may associate that with value. But if you came expecting more details or any kind of accountability, you were certainly left wanting.
What I saw was pure accountability theater, a marketing exercise disguised as accountability. They shared a timeline and a few tidbits, but spent most of the time talking about how powerful AI is, locked in the narrative.
The presentation is below. You can watch for yourself:
The Pivot To Security
AI labs seem to be pivoting to security. It’s not just OpenAI, but I recently saw a job posting on LinkedIn for Anthropic to help build Claude Security. This pivot makes sense. After all, they’ve enjoyed successes in development, and development and security are related. Security is also an enterprise use case like development, possibly opening more doors for them in enterprises.
I’m not saying labs shouldn’t pivot to security or that products won’t enjoy success. It’s just that the addressable market for cybersecurity products isn’t enough to justify the massive investment in AI, but I agree, it’s a better use case than asking an AI to book a vacation for you.
We’ll see where this pivot leads, but in the near term these tools won’t replace cybersecurity professionals. There will still be plenty of leftover problems, and the use of AI may create new problems that need to be solved. In short, AI will continue to be tools, not talent.
Low-Hanging Fruit
Now we come to the issue of the gym class guy.
One of the stories infecting my newsfeed was that of the Australian guy, whose AI assistant found a vulnerability that allowed him to be moved to the head of the line. This scenario is now conflated with the hot AI lab breakout summer scenarios, but they aren’t the same thing.
What AI highlights in scenarios like the gym class booking is the reality that many of us working in cybersecurity services have witnessed firsthand for decades. Most organizations have gotten away with lax application security processes and haven’t placed enough emphasis or allocated sufficient budget for cybersecurity. The gym class booking issue was a simple authorization bug, not a complex zero day. Any competent tester would have found this bug.
What the use and increased cybersecurity capabilities of AI mean for companies should be clear at this point. The era of companies getting away with leaving their low-hanging fruit unpicked is coming to an end.
The era of companies getting away with leaving their low-hanging fruit unpicked is coming to an end.
We are now told by OpenAI and others that we desperately need better AI for defenders so that companies that refused to take things like application security seriously and refused to put an appropriate budget in place for security in general can now have security! What a world we live in.
Conclusion
As hot AI lab breakout summer comes to a close, it’s important for us to put things in perspective. Ultimately, it’s not about what happened in each of these cases, but what it means, and what it means isn’t something we are great at deciphering.
When ChatGPT arrived, people predicted businesses would fall. When coding capabilities arrived, people predicted developers would be out of jobs in 6 months. As vibe coding entered the mainstream, people predicted SaaS was dead. Now, post-Mythos and post-Hugging Face hack, people are predicting cybersecurity is a goner.
The world is a complex place, and there’s a near-limitless number of ways people can shoot themselves in the foot. With every problem solved, new problems emerge. Yes, at some point, near-magical tech will arrive to solve our problems, but the question of when is key. The labs want you to believe we are on the cusp of this arrival. The reality, however, is quite different. In the near- to mid-term, AI will continue to be used as a tool, developers and cybersecurity professionals will still exist, and SAP and Salesforce won’t be taken out by vibe coding.