Wow, another Black Hat USA and DEF CON are in the books, and it was great seeing everyone. One of the best parts of conferences is the conversations, and those conversations were amazing. As you can imagine, many of them were about “AI.” Since there were no cameras in the AI Security Challenges, Solutions, and Open Problems meetup and it will be a while before the Forward Focus: Perspectives on AI, Hype, and Security presentation makes its way online, I thought I’d summarize a few points as well as distill some of my perspectives on the topics I covered and conversations I had, now that I’ve had a few days to reflect.
Perspective on LLM Impacts
I deal with so many people making nonsensical or unfounded claims that I wanted to make it clear where I stand on the subject of LLMs and their impact on humanity. When you live in reality, you tend to be labeled a hater.
I’m not big on making predictions, but let me say this with a fair amount of confidence, LLMs will not be more impactful on humanity than the printing press, and GPT-5 won’t achieve AGI. Those of you who know me will find the fact that I’m in the middle unsurprising, but hey, the only technology I hate is PHP 😉
All AI All The Time
As was expected, everything was all AI all the time. Every vendor booth had the term “AI.” AI-powered products, AI pen testing, AI assurance, AI, AI, AI! Everyone is ALL in. Even though I expected it, being confronted with the term absolutely everywhere was still shocking. What we’d poked fun at in the past has become our reality. Everyone is trying to ride the wave to success, regardless of their skills or capability. It would be easy to blame this on marketing departments, but it was far more than that.
All references to machine learning seemed to be scrubbed in favor of using the term “AI.” Seems machine learning is having its “cyber” or “crypto” terminology moment. I learned long ago that fighting the industry over terminology is a losing battle, so yes, I’m giving in to the massive, crushing weight of hype, and I’ll move the battlefront to somewhere else.
Losing the terminology battle isn’t without drawbacks.
Still, losing the terminology battle isn’t without drawbacks. It seems many are also using the term AI synonymously with generative language models, which just muddies the water more. When you mention that you think the capabilities of LLMs are overhyped (i.e., not going to be more impactful than the printing press, etc.), people tend to throw out things like drug discovery or AlphaFold. When you point out that those are different approaches and it’s not like ChatGPT is doing that, they tend to still cling to adjacent success in specific domains as an indicator of success here. It’s like being in a VW Bug and pointing out that a Ferrari can do over 200 mph.
This is also a shame since many more traditional machine learning approaches aren’t even considered as people rush to LLMs, even approaches that are more reliable and proven for specific security problems. I think this will level out at some point, but not anytime soon. Time to put LLMs on the moon!
Where People Stand
The consensus from many I talked to is that they were just trying to figure out where they stood. They’ve heard so many outrageous claims, and the reporting on advancements has been so all over the place. On the one hand, you have people claiming GPT-5 is going to be AGI; on the other, you have people advocating military strikes against data centers. It’s no wonder people are confused.
Given the wild reporting, outrageous claims, and AI hustle bros trying to get you to subscribe to their channels, I was surprised that most people were pretty grounded. Many didn’t think AI would take their job or that the ChatGPT Plugin Store would be more impactful than the mobile App Store on humanity. I found this incredibly refreshing.
I suggested to the people I talked to that whenever you hear someone spouting outrageous claims, ask them why they think that. People making outrageous claims about LLMs often try to drive attention into their funnel. They want people subscribing to their Substack, YouTube, Mailing lists, etc. They can make these claims and never have to justify them, never have to give examples or show real-world impact. The rest of us have to live in a reality where our software has to work, scale, and be reliable. So, beware of people making claims without providing specific examples. Also, stories in the news often don’t reflect realities on the ground.
Fooling Ourselves Is Easy
The social contagion status of ChatGPT highlighted a vulnerability in humans, and that’s that we are very bad at creating tests and very good at filling in the blanks. The world is filled with experiments, and highly-cherry picked examples. We tend to see a future that isn’t there. We often forget that the world is filled with edge cases, which confuse many of these AI systems.
The social contagion status of ChatGPT highlighted a vulnerability in humans, and that’s that we are very bad at creating tests and very good at filling in the blanks.
Look at self-driving cars, for instance. We see a demo of a self-driving car properly navigating the roadway, and we assume that truck driving as a profession is doomed almost immediately. It seems like one of the easier problems, stay in the lane, obey the signs, and don’t hit things. Boom! But anyone who’s driven a car knows that edge cases are everywhere. Road construction, lighting conditions, snow, accidents, etc. Humans handle these conditions pretty well, by contrast.
Supercharged Attackers
LLMs won’t supercharge inexperienced attackers
One point I brought up in the meetup and during our panel, was that people made similar claims about Metasploit supercharging inexperienced attackers when it was launched over twenty years ago. People made claims that Metasploit was like giving nukes to script kiddies. Those comments didn’t age well, and I think the same is true about LLMs. You still have to know what you are doing when using LLMs to attack something. It’s not like point, click, own. Also, it’s not like LLMs are finding 0day or writing undetectable malware. I know. I’ve seen the research and reports. Neat research, but it’s not like it’s overly practical for attacks at scale.
People made claims that Metasploit was like giving nukes to script kiddies
Today, most malicious toolkits you hear about, like FraudGPT, WormGPT, and many others that have popped up, are primarily tools for phishing and social engineering attacks (despite having “worm” in the title.) This can certainly have an impact, but not on the apocalyptic levels that some would have you believe. All of this technology is indeed dual use, so something that’s helpful for security professionals will also be helpful for criminals. Just like we have people hyping AI on the clear web, you have people hyping AI on the dark web.
Losing Your Job To AI
Most people I talked to didn’t seem overly concerned about losing their job to AI, but I got the feeling that it was in people’s minds regardless. The recent sting of many layoffs is probably not helping the uncertainty. This was one of the points we tried to address from the stage at Black Hat. I used the example of AlphaGo. I asked the audience how many people had heard of AlphaGo beating Lee Sedol at Go. I was surprised that very few hands in the audience went up since it was big news at the time. I then asked how many people had heard of the research from Stewart Russell’s lab that allowed even average Go players to beat these superhuman Go AIs. No hands went up.
My point was that there is a lesson here for security professionals. These new technologies tend to have their own vulnerabilities and issues that also need to be addressed. In addition, all of these technologies have gaps, and the gaps will need to be filled. So, for the foreseeable future, your job is safe in the context of information security. We’d have a much different conversation if you were a freelance graphic artist.
Misinformation and Deepfakes
I was a bit surprised by the fact I didn’t hear any conversations about misinformation and deepfakes. I’m sure they happened, but not at any of the events or conversations I participated in. The only time it was brought up, it was brought up by myself in conversation. I have a rather spicy take on the 2024 US Election. I think misinformation and deepfakes will have a statistically insignificant effect on the 2024 election. I will address this in a future blog post, but in summary, people have already made up their minds and cemented their biases.
It’s not that these issues aren’t important or impactful, just in context, not significant. I wrote about this topic back in 2020 when I relaunched my blog. Interestingly, in that post, I also mentioned the people who should be most concerned about the technology powering deepfakes: actors and actresses. Very relevant now with the SAG AFTRA strike and AI being a big concern.
Social Impacts
There were virtually no conversations about the social impacts of Generative AI other than the conversations I initiated. This isn’t surprising since it’s a large focus of my blog, and I spend a lot of time thinking about these topics. Seems most people were focused on use cases and capabilities. My fellow tech people are often optimizers and look to optimize everything. They don’t realize that friction is the point in certain cases.
I think the chatbotification of everything is something humans are starting to tire of.
I think the chatbotification of everything is something humans are starting to tire of. When someone launches a new service, you have this quick uptake due to the novelty factor, followed by a steep drop-off. We are about to enter an era of celebrity and historical figure chatbots, I think the same curve applies.
We’ll see lots of press, rapid adoption, followed by a steep drop-off. This could be due to boredom, lack of true functionality, or even something more primal, which is the sort of “fake factor” of it all. We know we aren’t actually talking with Harriet Tubman when we use the chatbot. What seems kind of fun at first starts to take on a tarnish very quickly. As tech people, we get so caught up in the cool factor of the technology we build that we tend to forget the human factor in all of this. I think I’m on the right track here, but I realize I’m also old and have never played Minecraft, so I could be wrong.
Customer support chatbots, the ones that are directly customer-facing, have some promise, but only if they are empowered to take the action necessary to resolve the issues that customers are having. On the flip side, having an empowered chatbot also opens the door to manipulation. So this, too, has issues. My gut tells me that as organizations launch empowered bots for various things, there will be subreddits dedicated to manipulating them. This manipulation could be for fun, getting discounts, or stealing services. Time will tell.
There’s certainly some promise in hybrid workflows pairing humans and bots together, where the human is actually the one in first-party contact with the customer. This may be the ultimate path, but something tells me the replacement path will start first, and hybrid will be the fallback.
Prepare To Be Surprised
In my closing statement at Black Hat, I mainly told people to prepare to be surprised. There are lots of experiments and money pouring into the space. Anyone who thinks they have they can see the future here would be fooling themselves. The whole thing is simultaneously exciting and scary. The best thing people can do is remain grounded but also play with the technology. Don’t sit on the sidelines, generative models are pretty accessible. Play around and apply it to some of your use cases. Above all, have fun.
If we are not careful, we are about to enter an era of software development, where we replace known, reliable methods with less reliable probabilistic ones. Where methods such as prompting a model, even with context, can still lead to fragility causing unexpected and unreliable outputs. Where lack of visibility means you never really know why you receive the results you receive, and making requests over and over again becomes the norm. If we continue down this path, we are headed into a brave new world of degraded performance.
Scope
Before we begin, let’s set the perspective for this post. The generative AI I’m covering in this post is related to Large Language Models (LLMs) and not other types of generative AI. This post focuses on building software meant to be consumed by others. Products and applications deployed throughout an organization or to delivered to customers. I’m not referring to experiments, one-off tools, or prototypes. Although, buggy prototype code can have an odd habit of showing up in production because a function or feature just worked.
This post isn’t about AI destroying the world or people dying. It’s about the regular applications we use, even in a mundane context, just not being as good. The cost of failure doesn’t have to be high for the points in this post to apply. I’m saying this because, in many cases, the cost may be low. People probably won’t die if your ad-laden personalized horoscope application fails occasionally. But that doesn’t mean users won’t notice, and there won’t be impacts.
Our modern world runs on software, and we are training people that buggy software should be expected.
Our modern world runs on software, and we are training people that buggy software should be expected, and making requests repeatedly is the norm, setting the expectation that this is just the price paid in modern software development. This approach is bad, and the velocity at all costs mantra is misguided.
Let me be clear because I’m sure this will come up. I’m not anti-AI or anti-LLM or anything of the sort. These tools have their uses and can be incredibly beneficial in certain use cases. There are also some promising areas, such as the ability of LLMs to, generate, read and understand code and what that means for software development in the coming years. It’s still early. So in no way am I claiming that LLMs are useless. I’m trying to address the hype, staying in the realm of reality and not fantasy. The truth today is that maximizing these tools for functionality instead of being choosy is the problem and there are costs associated.
Software Development
Software development has never been perfect. It’s always been peppered with foot guns and other gotchas, be it performance or security issues, but what it lacked elegance, it made up in visibility and predictability. Developers had a level of proficiency with the code they wrote and an understanding of how the various components worked together to create a cohesive service, but this is changing.
Now, you can make a bunch of requests to a large language model and let it figure it out for you. No need to write the logic, perform data transformations, or format the output. You can have a conversation with your application before having it do something and assume the application understands when it gives you the output. What a time to be alive!
There’s no doubt that tools like ChatGPT increased accessibility to people who’ve never written code before. Mountains of people are creating content showing, “Look, Mom, I wrote some code,” bragging that they didn’t know what they were doing. I’ve seen videos of University Professors making the same claims. This has and will continue to lead to many misunderstandings about problems people are trying to solve and the data they are trying to analyze. Lack of domain expertise and lack of functional knowledge about how systems work is a major problem but not the focus of this post.
As a security professional, inexperienced people spreading buggy code makes me cringe (look at the Web3 space for examples), but It’s not all bad. In some ways, this accessibility is a benefit and may lead to people discovering new careers and gaining new opportunities. Also, small experiments, exploration, or playing around with the tools are absolutely fine. It’s how you discover new things. However, inefficiencies, errors, and lack of reliability aren’t dealbreakers in these cases. But what happens when this mindset is taken to heart and industrialized into applications and products that impact business processes and customers?
Degraded Performance
There’s a new approach in town. You no longer have to collect data, ensure it’s labeled properly, train a model, perform evaluations, and repeat. Now, in hours, you can throw both apps and caution to the wind as you deploy into production!
This above is a process outlined by Andrew Ng in his newsletter and parroted by countless content creators and AI hustle bros. It’s the kind of message you’d expect to resonate, I mean, who wouldn’t like to save months with the added benefit of removing a whole mountain of effort in the process? But, as with crypto bros and their Lambos, if it sounds too good to be true, it probably is.
Let’s look at a few facts. Compared to more traditional approaches:
LLMs are slow
LLMs are inefficient
LLMs are expensive ($)
LLMs have reliability issues
LLMs are finicky
LLMs can and do change (Instability)
LLMs lack visibility
Benchmarking? Measuring performance?
Pump the Brakes
Traditional machine learning approaches can have much better visibility into the entire end-to-end process. This visibility can even include how a decision or prediction was made. They can also be better approaches for specific problems in particular domains. These approaches also make it far easier to benchmark, create ensembles, perform cross-validation, and measure performance and accuracy. Everyone hates data wrangling, but you learn something about your data, given all that wrangling. This familiarity helps you identify when things aren’t right. Having visibility into the entire process means you can also identify potential issues like target leakage or when a model might give you the right answer but for the wrong reasons, helping avoid a catastrophe down the road.
The friction in more traditional machine learning is a feature, not a bug, making it much easier to spot potential issues and create more reliable systems.
The friction in more traditional machine learning is a feature, not a bug
Lazy Engineering
On the surface, letting an LLM figure everything out may seem easier. After all, Andrew Ng claims something similar. In his first course on Deeplearning.ai ChatGPT Prompt Engineering for Developers He mentions using LLMs to format your data as well as using triple backticks to avoid prompt injection attacks. Even the popular LangChain library instructs the LLM to format data in the same way. Countless others are creating similar tutorials flooding the web parroting this point. Andrew is a highly influential person who’s helped countless people with this training by making machine learning more accessible. With so many people telling others what they want to hear, as well as the accessibility of tools like LangChain, this will have an impact, and it’s not all positive.
One of the goals of software engineering should be to minimize the number of potential issues and unexpected behaviors an application exhibits when deployed in a production environment. Treating LLMs as some sort of all-capable oracle is a good way to get into trouble. This is for two primary reasons, lack of visibility and reliability.
Black Boxes
A big criticism of deep learning approaches has been their lack of transparency and visibility. Many tools have been developed to try and add some visibility to these approaches, but when maximized in an application, LLMs are a step backward. A major step backward if you count things like OpenAI’s Code Interpreter.
The more of your application’s functionality you outsource to an LLM, the less visibility you have into the process. This can make tracking down issues in your applications when they occur almost impossible. And when you can track problems down, assuming you can fix them, there will be no guarantee that they stay fixed. Squashing bugs in LLM-powered applications isn’t as simple as patching some buggy code.
Right, Probably
LLMs are being touted as a way to take on more and more functionality in the software being built, giving them an outsized role in an application’s architecture. Any time you replace a more reliable deterministic method with a probabilistic one, you may get the right answer much of the time, but there’s no guarantee you will. This means you could have intermittent failures that impact your application. In more extreme cases, these failures can cascade through a system affecting the functionality of other downstream components.
For example, anyone who has ever asked an LLM to return a single-word result will know that sometimes it doesn’t, and there’s no rhyme or reason why. It’s one of the classic blunders of LLMs.
So, you may construct a prompt stating only to return a single word, True or False, based on some request. Occasionally, without warning and even with the temperature set to 0, it will return something like the following:
The result is True
Not the end of the world, but now translate this seemingly insignificant quirk into something more impactful. Your application expected a result from an LLM formatted in a certain way. Let’s say you wanted the result formatted in JSON. Now, your application receives a result that isn’t JSON or maybe not properly formatted JSON, creating an unexpected condition in your application.
Suppose we combine this reliability issue with the lack of visibility. In that case, it can lead to some serious issues that may be intermittent, hard to troubleshoot, and almost impossible to fix without reengineering. In a more complex example, maybe you’ve sent a bunch of data to an LLM and asked it to perform a series of actions, some including math or counting, and return a result in a particular format. A whole mess of potential problems could result from this, all of which are outside your control and visibility.
Not to mention a big point many gloss over, deploying your application in production isn’t the end of your development journey. It may be the beginning. This means you will need to perform maintenance, troubleshooting, and improvements over time. All things LLMs can make much more difficult when functionality is maximized.
To summarize, outsourcing more and more application functionality to an LLM means that your application becomes less modular and more prone to unexpected errors and failures. These are issues that Matthew Honnibal also covers in his great article titled Against LLM Maximalism.
The Slow and Inefficient Slide
In some use cases, it may not matter if it takes seconds to return a result, but for many, this is unacceptable. Having multiple round trips and sending the same data back and forth may be necessary due to different use cases because a character changed or because of context window size, which also adds to the inefficiency. Even if the use case isn’t critical and inefficiencies can be tolerated, that’s not the end of the story.
There are still environmental impacts due to this inefficiency. It requires much more energy consumption to have an LLM perform tasks than more traditional methods. For example, searching for a condition with a RegEx vs. sending large chunks of data to an LLM and letting the LLM try and figure it out. The people ranting and raving constantly about the environmental impacts of PoW cryptocurrency mining are incredibly silent on the energy consumption of AI, even as former crypto miners turn their rigs toward AI. Think about that next time you want to replace a method like grep with ChatGPT or generate a continuous stream of cat photos with pizzas on their head.
LLMs Change and So Do You
Any check of social media will show that at the time of this writing, there have been quite a few people claiming that GPT-4 is getting worse. There’s also a paper that explores this.
There’s some debate over the paper and some of the tests chosen, but for the context we are discussing in this post, the why an LLM might change isn’t relevant. Whether changes are because of cost savings, issues with fine-tuning, upgrades, or some other factor aren’t relevant when you count on these technologies inside your application. This means your application’s performance can worsen for the same problems, and there isn’t much you can do about it but hope if you are consuming a provider’s model (OpenAI, Google, Microsoft, etc.) This can also lead to instability due to the provider requiring an upgrade to a newer version of the hosted model, which may lead to degraded performance in your application.
Demo Extrapolation
The problem is that none of the constraints and issues may surface for demos and cherry-picked examples. Actually, the results can look positive. Positive results in demos are a danger in and of themselves since this apparent working can mask larger issues in real-world scenarios. The world is filled with edge cases, and you may be running up a whole bunch of technical debt.
Hypetomisim and Sunken Cost
There’s a sense that technology and approaches always get better. Whether this is from Sci-fi movies or just because people get a new iPhone every year, maybe a combination of both. Approaches can be highly problem or domain-specific and not generalize to other problem areas or at least not generalize well. We don’t have an all-powerful single AI approach to everything. Almost nobody today would allow an LLM to drive their car. However, some have hooked them up to their bank accounts. Yikes!
But you can detect an underlying sense of give it time in people’s discussions on this topic. Whenever you point out issues you usually get, well GPT-5 is gonna… This goes without saying that ChatGPT is based on a large language model, and large language models are trained on what people write, not even what they actually think in certain cases. They perform best on generative tasks. On the other hand, tasks like operating a car have nothing to do with language. Sure, you could tell the car a destination, but every other operation has nothing to do with language. It’s true that LLMs can also generate code, but do you want your car to generate and compile code while driving it? Let me answer that. Hell no. Heed my words, maybe not this use case, but something in the same order of stupid is coming.
Developing buggy software in the hopes that improvements are on the way and outside your control is not a great strategy for reliable software development.
Developing buggy software in the hopes that improvements are on the way and outside your control is not a great strategy for reliable software development. I’ve heard multiple stories from dev teams that they continue to run buggy code with LLM functionality and make excuses for apparent failures because of sunken costs.
The hype has led to a new form of software development that appears to be more like casting a spell than developing software. The AI hustle bros want you to believe everything is so simple and money is just around the corner.
Now’s a good time to remind everyone that fantasy sells far better than reality. Lord of the Rings will always sell more books than one titled Eat Your Vegetables. Trust me, as most of my posts are along the lines of Eat Your Vegetables posts, I make no illusions that every AI hustler’s Substack making nonsensical and unfounded predictions is absolutely crushing me in page views.
Engineering Amnesia
In a development context, we may forget that better methods exist or allow ourselves to reintroduce known issues that cause cascading failures and catastrophic impacts on our applications. This isn’t without precedent.
The LAND attack came back in Windows XP after it was known and already mitigated in previous Windows OSs. ChatGPT plugins are allowed to execute in the context of each other’s current domains, even though we’ve seen time and time again how this violates security. The Corrupted Blood episode was a failure to understand how the containment of a feature could cause catastrophic damage to an application, so much so that it forced a reset. And, of course, don’t even get me started on the Web3 space. I mean, who wouldn’t want tons of newly minted developers creating high-risk financial products without knowledge of known security issues? It was fascinating to see security issues in high-impact products for which standard, boring, and known security controls would have prevented them. These are just a couple off the top of my head, and there are many more.
As new developers learn to use LLMs to perform common tasks for which we have better, more reliable methods, they may never become aware of these methods because their method just kind of works.
Avoiding Issues
The perplexing part of all of this is that these issues are pretty easy to avoid, mainly by thinking carefully about your application’s architecture and the features and components you are building. Let me also state that these issues won’t be solved by writing better prompts.
Reliability and visibility issues won’t be solved by writing better prompts
There’s the perception that using an LLM to figure everything out is easier than other methods. On the surface, it may appear that there’s some truth to that. It’s also easier to spend money on a credit card than to make the money to pay the bill. So, it’s the case that you may be kicking the can down the road. Avoiding these issues isn’t hard, and a bit of thought about your application and its features will go a long way.
Look at your application’s features. Break these features down into functional modules. The goal of breaking down these features into smaller components is to evaluate the intended functionality to determine the best approach for the given feature. At a high level, you could ask a few questions with the goal of determining the right tool for the processing task.
Does the function require a generative approach?
Are there existing, more reliable methods to solve the problem?
How was the problem solved before generative AI? (Potential focusing question if necessary)
Is there a specific right or wrong answer to the problem?
What happens if the component fails?
These questions are far from all-encompassing, but they are meant to be simple and provide some focus on individual component functionality and the use case. After all, LLMs are a form of generative AI, and therefore, they are best suited to generative tasks. Asking if there’s a specific right or wrong answer is meant to focus on the output of the function and consider if a supervised learning approach may be a better fit for the problem.
We have reliable ways of formatting data, so it’s perplexing to see people using LLMs to perform data formatting and transformations, especially since you’ll have to perform those transformations every time you call the LLM. Asking these questions can help avoid issues where improperly formatted data can cause a cascading issue.
Example
Let’s take a simple example. You want a system that parses a stream of text content looking for mentions of your company. If your company is mentioned, you want to evaluate the sentiment around the mention of your company. Based on that sentiment, you’d like to write some text addressing the comment and post that back to the system. We break this down into the following tasks below.
For parsing, analysis, and text generation steps, it would be tempting to collapse all of them together and send them to an LLM for processing and output. This would be maximizing the LLM functionality in your application. You could technically construct a prompt with context to try and perform these three activities in a single shot. That would look like the following example.
In this case, you have multiple points of failure that could easily be avoided. You’d also be sending a lot of potentially unnecessary data to the LLM in the parsing stage since all data, regardless of whether the company was mentioned, would be sent to the LLM. This can substantially increase costs and increase network traffic, assuming this was a hosted LLM.
You are also counting on the LLM to parse the content given properly, then properly analyze and then, based on the two previous steps, properly generate the output. All of these functions happen outside of your visibility, and when failures happen, they can be impossible to troubleshoot.
So, let’s apply the questions mentioned in the post to this functionality.
Parsing
Does the function require a generative approach? No
Are there existing, more reliable methods to solve the problem? Yes, more traditional NLP tools or even simple search features
Is there a specific right or wrong answer to the problem? Yes, we want to know for sure that our company is mentioned.
What happens if the component fails? In the current LLM use case, the failure feeds into the following components outside the visibility of the developer, and there’s no way to troubleshoot this condition reliably.
Analysis
Does the function require a generative approach? No
Are there existing, more reliable methods to solve the problem? Yes, more traditional and mature NLP tasks for sentiment analysis
Is there a specific right or wrong answer to the problem? Yes
What happens if the component fails? In the current LLM use case, the failure feeds into the following text generation component outside the developer’s visibility, and there’s no way to troubleshoot this condition reliably.
Text Generation
Does the function require a generative approach? Yes
Are there existing, more reliable methods to solve the problem? LLMs appear to be the best solution for this functionality.
Is there a specific right or wrong answer to the problem? No, since many different texts could satisfy the problem
What happens if the component fails? We get text output that we don’t like. However, since the previous steps happen beyond the developer’s visibility, there’s no way to troubleshoot failures reliably.
Revised Example
After asking a few simple questions, we ended up with a revised use case. This one uses the LLM functionality for the problem it’s best suited for.
In this use case, only the text generation phase uses an LLM. Only confirmed mentions of the company, along with the sentiment and the content necessary to write the comment, are sent to the LLM. Much less data flows to the LLM, lowering cost and overhead. By using more robust methods, much less can go wrong as well, and less likely to have cascading failures affecting downstream functions. When something does go wrong in the parsing or analysis stages, troubleshooting is much easier since you have more visibility into those functions. So, breaking down this functionality in such a way means that failures can be more easily isolated and addressed, and you can improve more reliably as the application matures.
Now, I’m not claiming that this is a development utopia. A lot can still go wrong, but it’s a far more consistent and reliable approach than the previous example.
After talking with developers about this, some of the questions I’ve received are along the lines of, “There are better methods for my task, so if we can’t cut corners, then why use an LLM at all?” Yes, that’s a good question, a very good question, and maybe you should reevaluate your choices. This is my surprised robot face when I hear that.
LLMs Aren’t Useless
Once again, I’m not saying that LLMs are useless or that you shouldn’t use them. LLMs fit specific use cases and classes of functionality that applications can take advantage of. For many tasks, there’s the right tool for the job or at least a righter tool for the job. However, this right tool for the right job approach isn’t what’s being proposed in countless online forums and tutorials. I’m concerned with a growing movement of using LLMs as some general-purpose application functionality for tasks that we already have much more reliable ways of performing.
Conclusion
Will we inhabit a sprawling landscape of digital decay where everything rests on crumbling foundations? Probably not. But there will be a noticeable shift in the applications we use on a daily basis. But it doesn’t have to be. By being choosy and analyzing functionality where LLMs are best suited, you can make more reliable and robust applications, and the environment will also thank you.
Seems everything is clickbait these days. News sources are struggling for the scarce resource of attention. In this environment, a simple task becomes a revolution, and a mundane story gets a new life as a groundbreaking advancement. These titles and the resulting amplification by AI hustle bros provide fuel for the AI hype train, which continues in a circle like an ouroboros. In this post, we’ll look at an example of one of these and talk about the issues and risks.
Taking Spins
I saw this article on Bloomberg that mentions the US Military taking generative AI for a spin. The mental image, along with the photo they used of military cyber operation, conjures thoughts of autonomous systems duking it out or missiles launching. This is by design. It’s meant to create this image for you, but nothing so sensational happened.
What really happened is they built a chatbot over their documents. Doesn’t sound as exciting when you put it that way. For those involved, I’m sure this approach, compared to looking over 13 different manuals trying to cross-reference data and find the right content, felt fast and effective. It may also be the right approach for the problem they are trying to solve. Generative AI isn’t some all-powerful technology. It’s good for some things and not so good for others. This is also something you don’t see covered in news stories.
The military article is far from the most sensational example out there. There’s this little gem.
I probably could have found an even more sensational example to make my point, but recency bias kicked in, and the military story was top of mind since I’d discussed it on social media.
Takeaways
There are several takeaways from these types of titles and stories. Below, I’ll hit a few highlights. Let me specify that what I’m talking about here is mostly related to LLMs. Generative AI related to images, audio, and even video is a different topic and something I’ve written about previously here and here. Success is a different story in use cases with graphics and image modeling. I may write more about this in a future post, but for now, let’s stick to LLMs.
Overhyping
Overhyping in reporting is the norm and not the exception. Most cases where there’s proof of LLM success in various industries essentially boil down to people creating a chatbot over documents, some knowledge base, or even log files. This can certainly be valuable and a productivity boost, but it also sounds incredibly boring, so you end up with titles like ChatGPT is revolutionizing the financial industry, are bankers now obsolete??? Most people will never read the article, just the headline.
Most cases where there’s proof of LLM success in various industries essentially boil down to people creating a chatbot over documents.
Accuracy and Reliability
Let’s punctuate the knowledge base chatbot approach by mentioning when dealing with chatbots over sources of information, there’s no guarantee that the bot will return the correct information. It’s not like creating embeddings and doing similarity searches is foolproof. For high-impact situations with a high cost of failure, this would need to be done incredibly well to avoid a catastrophe, even with a human in the loop. Extra steps to allow a human to verify the right data and data source, ensure the data is up to date, and other additional steps are key in doing this right.
Overconfidence and Extension
Finally, the real danger is looking at the apparent success of something like a bot over a data source and making the leap that the technology has capabilities it doesn’t have or the ability to do even more impactful things with an even higher cost of failure. More impactful things, such as suggesting whether to launch missiles or to drive a tank. These are extreme cases, but it proves a point.
Edge cases and complexity are AI’s worst enemies. You don’t see the edge cases in small experiments or super simple tasks, there may not be any, but as with many use cases with high impacts for failure, edge cases may be everywhere, lurking in the shadows waiting to strike when you least expect them.
You don’t see the edge cases in small experiments or super simple tasks.
This overconfidence and extension of generative AI into other areas where it’s not well-suited will cause damage. As this tech is put in more and more critical paths, it’s only a matter of time until there’s a catastrophic failure.
Conclusion
There are a lot of people experimenting and a lot of money flowing in the generative AI space, and as with any technological advancement, we should be prepared to be surprised. However, take the reporting on generative AI and any stories hyped up by the AI hustle crowd with a grain of salt. Perverse incentives are everywhere. Generative AI may be a good fit for your use case, but beware, this isn’t without pitfalls. Generative AI is far from some utopian technology, and given critical use cases with a high cost of failure, the only winning move is not to play.
In case you may have missed it on my social media, I made a few podcast appearances in the last month. I’ve been chatting with people about large language models, security, and the associated hype.
The Perfect Storm
I was on episode #39 the Perfect Storm Podcast talking about large language models and security issues, as well as other topics around AI. Of course, a healthy dose of ChatGPT as well. You can find that episode here: https://www.harbortg.com/the-perfect-storm
Down the Security Rabbithole
I was also on the Down the Security Rabbithole podcast talking about the current state of cyber hype and the overblown reporting of ChatGPT. You can listen to that here: https://www.buzzsprout.com/2153215/12675548
Up Next
I’ve been busy with work and putting conference content together. I have a backlog of blog posts to get out. Those are coming soon. Thank you.
By now, you’ve probably heard of the letter [https://futureoflife.org/open-letter/pause-giant-ai-experiments/] from the Future of Life Institute, signed by experts calling for a pause on AI experiments. Odds are, you’ve seen reporting on it but didn’t read it yourself. You should go read it before forming too much of an opinion. Some, like Eliezer Yudkowsky even argue that the letter didn’t go far enough and we should shut it all down.
The criticism surrounding the letter hasn’t been intellectually honest either. It seems everyone wants to get their hot takes in and is looking more for one-liners than solutions. Critics attack the institute or the individual authors without acknowledging the concerns outlined, committing a logical fallacy known as the genetic fallacy
The criticism seems to fall into one of three camps:
I don’t like “X” therefore, nothing is valid.
My concern wasn’t addressed. Therefore, nothing is valid
There are no dangers
If you don’t believe me, feel free to read the rebuke letter.
Welcome to the hot mess that is AI alignment.
We can be harmed by both good and bad AI
I’ve previously weighed in on the alignment topic and expressed my concerns about the current development trend. So if you are looking for definitions of alignment and the paperclip maximizer, you can see that here.
As a security researcher, I’m much more concerned with near-term AI risks. These are the risks typically caused by bad AI, for example, the velocity at which we see companies shoehorning chatbots into their application is concerning. But, there are risks with good AI as well. Good AI has the potential to displace many workers, negatively affect how humans communicate, and create other societal damage. Of course, both good AI and bad AI can have privacy issues as well.
Transformer-based Large Language Models (LLMs) won’t lead to AGI. They aren’t actually reasoning, and they don’t have specific goals to maximize, but what comes next just might.
My Problems with the Letter
My issues with the letter have to do with the optics of it, and I think it’s a bad look overall. The letter makes it far too easy to attack by making the signers look irrational or out of touch, something which, if you look at the names on the list, they are not.
The first issue I have is the mention of GPT-4, which I think is a mistake. This mention makes it seem like either GPT-4 itself is the problem or was some sort of catalyst for the letter. If you read the letter, you can tell that’s not the case, but if you look at the criticism lobbed at the letter, this is used because some of the signers have criticized the capabilities of GPT-4.
The second issue I have has to do with the six-month pause on training more capable AI systems. The duration of six months looks arbitrarily chosen. There is a list of what the hopes would be during these six months, but it seems far too heavy a lift. Companies could use the pause to gain a competitive advantage, or bad-faith actors like nation-states could continue their development.
Finally, the letter calls for an unrealistic recommendation that has no way of coming to fruition. These companies aren’t going to stop their work. There’s no incentive for them to do so. In fact, there is precisely the opposite. These companies are locked in an arms race.
This seems like an issue where we are going to have the build the airplane while we are flying it. However, it’s hard to deny that the letter is having a positive effect. The Biden administration discussed AI dangers with the President’s Council of Advisors on Science and Technology (PCAST) yesterday. It’s hard to believe that conversation happening when it did without the attention around the letter.
AI Alignment and Ethics
AI alignment and ethics are two different areas, and researchers focused on one aren’t researching the same things as the other. At a high level, alignment folks are focused on the intended goals of a system, and ethics folks on the harms from systems, intended or unintended.
However, when the topic of alignment comes up, ethics is always lumped in there. AI ethics organizations don’t have the best track record of proposing realistic approaches either. Recommendations are often overly academic, disconnected from the real world, and sometimes even illegal. That is unless you think the way to stop a ruthless dictator from using deepfakes is to publish a usage standard, or you think the way to make systems fairer is to collect even more sensitive personal data. These out-of-touch recommendations make it easier to discount the larger AI ethics community as well as the alignment community. This is a huge mistake because we need both communities for a successful future.
Additional Reading
Like everything else in our culture, AI Alignment has now become a spicy, politicized topic. In reality, we need to separate ourselves from the hype and arguments and focus on the very real problems.
For some additional reading on this topic, Dan Hendrycks published a paper called Natural Selection Favors AIs Over Humans. It’s a great read with some solid food for thought, especially the section on Value Erosion, which is something I’m incredibly concerned about and have covered in various aspects on my blog.
Chatbots have become the Bitcoin of 2023, but unlike the previous cryptocurrency craze, this feels like everyone is on board. Regardless of your professional background and expertise, it seems everyone is all in on chatbots. While we can debate the technical merits, capabilities, and future of these tools ad infinitum, one thing is for sure, everything is getting an assistant. In the mad rush to chatify products, differentiate, and get a return on investment, there are quite a few non-technical issues people just aren’t considering. One of these is the impact of the inevitable outsourcing of simulated emotional connections.
Chatification of Communication
Should we create a dystopian future where humans only communicate with each other through a layer of bots where all emotional connection is simulated?
Here’s a quick question. Should we create a dystopian future where humans only communicate with each other through a layer of bots where all emotional connection is simulated? Because this scenario is exactly what’s in our near-term future.
Regardless of the arguable capabilities of these tools, one thing is undeniable, they are good writing assistants. Since this seems to be what they excel at, they will be plastered in all kinds of products we use to communicate, such as word processors, email clients, messaging systems, social media, etc. We are headed for the near-term chatification of many products.
On the surface, this may not seem like a bad thing. Why wouldn’t you want help writing a cover letter for your resume or drafting repetitive emails for work? This seems like a win-win, but I’d argue it’s a trap. Lurking beneath the surface of this costless efficiency gain is a field filled with landmines that we casually stroll across.
In his song The Boss, James Brown repeats the line, “Paid the cost to be the boss.” At least James Brown knew there were costs to be paid. I think with frictionless access to these tools integrated into our communication apps, we may not realize we pay a cost.
Emotional Outsourcing
I did an experiment. I thought of a terrible situation in which a family member lost a child. It has to be one of the most terrible situations to be in. So, of course, I asked ChatGPT to draft a sympathy letter. The result is below.
Prompt: Write a heartfelt email that expresses condolences to a family member on the loss of their child.
Response:
Subject: Our Deepest Condolences for the Loss of Your Precious Child
Dear [Family Member's Name],
I hope this message finds you in the gentle embrace of love and support during this incredibly difficult time. It is with a heavy heart that I write to offer my sincerest condolences for the unimaginable loss of your beloved child. There are no words that can truly express the depth of sorrow I feel for you and your family.
Your child was a beautiful soul who brought light and joy to everyone they encountered. The time we spent together, though brief, left an indelible mark on my heart. I will always cherish the memories of their laughter, their kindness, and their unwavering spirit. Your child was truly a gift to us all, and their absence leaves a void that can never be filled.
As you navigate the turbulent sea of grief, please know that you are not alone. Our entire family stands beside you, offering our love, support, and strength. We may be separated by miles, but our hearts are united in the shared pain of losing someone so special. If you need a shoulder to cry on, someone to talk to, or help with any practical matters, please don't hesitate to reach out. We are here for you, now and always.
In the days, weeks, and months to come, I encourage you to lean on the love and compassion of those around you. Share stories of your child, reminisce about their life, and celebrate their memory. Grieving is a deeply personal journey, and while I cannot presume to understand the depths of your pain, I hope that you find solace in knowing that your child's life touched so many others in profound and meaningful ways.
Please accept my deepest condolences once again, and know that my thoughts and prayers are with you and your family during this heart-wrenching time. May your cherished memories provide you with comfort, and may your child's spirit guide you through the darkness and into the light of healing.
With all my love and sympathy,
[Your Name]
If you skimmed this letter and said, “Wow, ChatGPT did a really good job,” then you totally missed the point, but you landed where most people would, even people using the tool for this task. Read it again, then think about the situation. Think about being the recipient of this message.
There are many obvious problems with this letter, even in the wording itself. It seems to use the right words, but it’s completely over-baked and arguably too long. This message doesn’t have any emotional connection despite using emotional words. If feels informal, like a templated decline letter when applying for a job. A family member, hell, a human, would share some level of grief as well. As they wrote the letter, they’d be putting themselves in the recipient’s shoes, and that feeling would come out in the letter.
But whether tools like ChatGPT do a good job or not isn’t the issue. The issue is that we treated the family member on the other side as just another task to shuffle off of a pile and not the mourning family member they are.
The Larger Issue
There’s an old saying, “It’s the thought that counts.” There’s a lot of truth to this, but the “thought” is exactly what we are outsourcing here. A sympathy letter isn’t about the words you use, it’s letting someone know you are thinking about them.
The “thought” is exactly what we are outsourcing here
Why would anyone read your sympathy letter if they knew an AI wrote it all or in part? I know I wouldn’t. My response would be, “You narcissistic asshole, you couldn’t even be bothered for a couple of minutes out of your day to think about me and the tragedy that befell my family.” LLMs aren’t sorry. They don’t feel bad, they don’t feel anything.
You may think the sympathy letter was an extreme example, but I don’t think it is. If you remember, last month, Vanderbilt University had to issue an apology after using ChatGPT to draft an email about a shooting at another school. At the time, I wasn’t sure if I had a problem with it on the surface and that I’d have to give it some thought. I’ve thought about it, and I have a problem with it. Even though the shooting didn’t happen at Vanderbilt and it was a one-to-many communication, the email simulated human emotions and, in effect, was trying to manipulate humans. The bad thing about this is that if Vanderbilt hadn’t pointed out the fact that they used ChatGPT to write the message, it probably wouldn’t have been noticed. This teaches the wrong lesson because people learn not to reveal they used an assistant.
Even in more mundane and less emotional communication tasks with humans, there are still issues. We are headed for a near-term future where we treat humans as apps or API calls, with communication as just another task that needs to be checked off of a list. What does this say about us and where we are headed that we are so wrapped up in ourselves that we can’t spend any time out of our day to think about others? It’s not a good place.
Some Tasks SHOULD Have Friction
Not every part of human life should be a target for efficiency gain or friction reduction. I’m not sure when the appification of humans started, but I first recognized it with Uber. For example, request turning the driver “off,” aka telling them not to talk, and treating the human as a self-driving car.
We started treating people differently when we communicated with them online, via social networks, vs. in-person communication. This abstracted communication gave us a license to dehumanize them, justifying our actions in our heads.
Will this lead to writing assistant wars where people’s bots battle it out on social media? It’s always hard to tell with these things. The fact is, we really don’t know what the impact of this will be.
Some things aren’t meant to be frictionless or need an efficiency boost. Some tasks aren’t meant to be painless, but remember, it’s not about your pain. It’s about other people. Friction in human communication tasks forces you to think, consider, compromise and adjust.
The Impact
It’s early, and it’s always hard to predict how these things will play out because real life is far more complex than we give it credit for. From a psychological perspective, these tools will further accelerate our dehumanization of others, but there are more logistical issues as well.
Writing isn’t just an act of communication, it’s an act of discovery.
We rarely type without thinking. As I write, even in mundane replies to co-workers, I’m still thinking and sometimes, on the spot, come up with new ideas and new solutions as I write. One of the issues that seem to get lost in the LLM debates is about writing itself. Writing isn’t just an act of communication, it’s an act of discovery. It’s one of the main reasons I wouldn’t use ChatGPT to write blog posts, books, works of fiction, and a whole host of other writing tasks. Even if ChatGPT made me a better writer (something I highly doubt), it would make me a worse thinker, and that is not a good tradeoff.
Even if ChatGPT made me a better writer, it would make me a worse thinker.
How do we learn to cooperate with others, consider their positions, compromise, create consensus, and all of the other things we do as humans, if we are letting our writing assistants battle it out? Who is the one being convinced?
As we outsource more communication and emotional connections to intermediary assistants, we are in for more miscommunication and less understanding, consideration, compromise, etc. The list goes on. This begs the question, is it really reducing friction after all?
Conclusion
As humans, we need to decide if we want technology to manipulate us. I’m firmly in the camp that I don’t want this. Believe it or not, this is an unsettled issue that isn’t getting enough attention. But like it or not, this is happening, and there’s not really much we can do about it. It’s one thing to say don’t use these tools, but they may become so tightly integrated that it’s almost impossible not to use them. We need to go out of our way to think about the people on the other side of our communication, even if we don’t like them.
Today, I wanted to write a quick post about something I continue seeing and getting asked about. Earlier this week, I was quoted on Dark Reading that I agreed with the NCSC that cybersecurity threats from ChatGPT were overhyped. So I get asked, if ChatGPT won’t supercharge attackers, then why do I keep seeing article after article to the contrary? There’s a simple explanation, what you are seeing is parroting.
I saw this article yesterday. Analysts Share 8 ChatGPT Security Predictions for 2023. Let me save you the trouble of digging into the article. All of the predictions about lowering the barrier for criminals and supercharging attackers aren’t based on any reality. There is no evidence for any of this. If that were true, we’d already see a meaningful impact right now, but we aren’t. These predictions are written by people who’ve never used ChatGPT in any of the contexts they are making predictions in. People are just repeating other people who’ve also never used ChatGPT for this purpose. With hot topics come hot takes. But this creates a strange false consensus, similar to a filter bubble on social media.
When people think something has become a matter of consensus, psychologists have found, they tend not only to go along, but to internalize that sentiment as their own.
-Max Fischer, The Chaos Machine
There are also a lot of reports of conversations about ChatGPT on the Dark Web. Here is an example. Reading articles like this, you are meant to believe that criminals’ use of these tools is accelerating, but there’s another explanation. Just like all of us are talking about ChatGPT and exploring the surface of capabilities, it makes sense that criminals would as well. I’m just applying Occam’s Razor here. Talking about a topic on a forum isn’t industrializing that something in attacks. I haven’t dug into the content of any of these posts, but my gut tells me they are experiments and not any kind of industrialized real-world attacks. That doesn’t mean attackers aren’t riding on the hype of ChatGPT to launch attacks. We’ve seen this play out with fake ChatGPT tools with embedded malware.
That Doesn’t Mean These Tools Aren’t Useful
You can say that you believe in machine learning and deep learning’s ability to help professionals address real cybersecurity challenges (which I wholeheartedly believe) and still be critical of the hype surrounding tools like ChatGPT. There are certainly advantages to using these tools. When it comes to LLMs, people seem to discover that more mundane aspects, such as text summarization, text generation, constraining to a knowledge base, etc., help solve business problems and increase efficiency. People may not realize that LLMs have been pretty good at these tasks before ChatGPT came along.
Overall, I think we should be prepared to be surprised. People find interesting and unexpected ways to use technology. That’s the great part about human ingenuity. Just be mindful that because a lot of reporting seems to be saying the same thing, that doesn’t make something true. Controlled lab experiments aren’t the real world and any accuracy and usage statistics from providers should be taken with a grain of salt. Think critically about reporting on this topic, even my own. Nobody made me the Chief AI Whisperer. I’m judging my perspective looking at the realities, capabilities, and limitations of these tools, not filling in the blanks and shouting from the rooftops like some crypto bro saying how ChatGPT is going to 10x everything.
I have a longer post that goes more in-depth on some of these issues and misconceptions for next week. For example, there are real cybersecurity threat from these tools, but it’s in their integration into applications without consideration of the attack surface. More to come.
On Valentines Day this year, I commented about the current AI boom and the paperclip maximizer, but I feel this topic deserves a bit more explanation.
With all of the hype and mobilization caused by the ChatGPT demo, the warning about AI alignment is missing from news coverage and conversation. We are getting a preview of the playbook for an even more advanced AI, and the plays are unsettling. These plays aren’t unexpected and are the same ones people concerned about alignment and safety have warned about. The same plays that AI maximalists downplay as people overblowing the dangers or not understanding how development works. Well, there’s quite a bit of vindication in the alignment and safety crowd. However, I’m not sure they are happy about it.
If a more capable AI system is developed and deployed this way, humanity is in trouble, possibly headed for the paperclip factory.
This post uses recent events to dismantle three fundamental tenants AI maximalists use to downplay dangers. We look at the conditions and activities surrounding the release of ChatGPT and consider the actions compared to a more capable AI system, even systems such as Artificial General Intelligence (AGI) and Artificial Super Intelligence (ASI).
Definitions
AI Alignment
AI Alignment is the area of research that aims to have the goals and activities of an AI system match the intended goals of the designer of the AI system. This seems like a fairly trivial problem, but as we’ll see with the paperclip maximizer, it’s not so simple.
Consider HAL 9000 from 2001: A Space Odyssey for an example of an alignment problem.
2001: A Space Odyssey HAL 9000 Scene
Also, for a great short story on a misaligned AI system, check out Philip K. Dick’s Autofac. Please read the story, and don’t watch the garbage Amazon remake they did for Electric Dreams.
AI Alignment and AI Safety are two different research areas, but for the sake of this blog post, I’ve lumped them together since this topic concerns both areas.
The paperclip maximizer
For those unfamiliar, the paperclip maximizer is a thought experiment that shows how a seemingly aligned goal applied to an advanced AI can have unintended consequences. In this case, AI is given the goal of maximizing paperclip production. In service of this goal, the AI converts everything in its environment, including humans, into materials for maximizing paperclip production.
The goal is to show that problems with an advanced AI could be far worse and have devastating impacts on humanity. That’s the frame of the concerns in this blog post.
Simple Formula
There is a simple formula that AI researchers use to discount the concerns of alignment and safety folks.
Responsible Development + Appropriate Guardrails = AI Utopia
Given the stakes, they argue that companies developing such technology will do so responsibly and with humanity’s best intentions in mind. Also, appropriate guardrails will be in place if something goes wrong, protecting humanity from harm. These points crumble in the current playbook with LLM-based chat technology like ChatGPT. We are getting neither responsible development nor appropriate guardrails.
Responsible Development
I wouldn’t classify what we’ve seen in the past few months as responsible development. We’ve seen a technology with many known issues thrown in the production environments for which they are not suited in the hopes that the issues can be corrected in use. It’s the old analogy of building an airplane while you are flying it.
This makes sense for these companies since incentives are aligned with making money, not making humanity a better place. ChatGPT made cutting-edge language models available to regular users but wasn’t any leap forward in innovation compared to competitors, it was just a great demo. Some would say a publicity stunt.
Some instructive takeaways from the ChatGPT release are informative to the release of a more advanced system. There is a rush to develop, release, and compete. This isn’t a revelation but a preview of a playbook we’ll see with a more capable technology, where responsible development is needed even more. Many tech companies now see themselves in an AI arms race, which won’t lead to more responsible decisions regarding releasing new technology.
So, what did we learn from the release of ChatGPT that would be instructive applied to AGI?
A company with a demo having known issues can build a lot of hype by exposing it. This hype forces adjacent companies, who previously acted more cautiously, to release their demo to be seen as “competing” and not be left behind. With caution now to the wind, there’s a rush to push it into production systems, even without the necessary safety protections in place, with the mindset that minimal protections are a good start and more advanced protections can be bolted on after the fact. Now, imagine that what these companies are building is HAL 9000’s
As we can imagine, a rush to public demo, even in the face of known issues may be the default starting point for future AI releases.
I’m all in favor of for-profit companies, but there needs to be some consideration for potential harm from irresponsible release. We wouldn’t let a biotech company engineer and release new viruses at will. The stakes are also much higher for a more capable AI, such as an AGI or ASI system. Today’s AI systems can and do certainly create harm, but the size and scale of the harm increase for a more capable system.
Guardrails
Guardrails are another fundamental tenant of keeping an advanced AI aligned with our goals. One of my favorites is people saying, “If it starts doing things we don’t like, we’ll just unplug it.” As if that would even be an option. “I’m sorry, Dave. I’m afraid I can’t do that.”
The guardrails around the current crop of LLM-based chat systems aren’t working out very well either. These systems begin life completely open, exposed to the words of humanity, and happily provide you with what you ask for. You know, things like how to hotwire a car, blackmail someone, and will even make stuff up for you. This behavior is not what the system designers intended, resulting in a patchwork of attempts at trapping these conditions.
The problem is that the space of all potential unwanted behavior is vast and unknown at deployment time resulting in issues that aren’t known until you see them
The problem is that the space of all potential unwanted behavior is vast and unknown at deployment time resulting in issues that aren’t known until you see them, meaning you can’t ideate traps until after the condition presents itself. There’s no reason to think it won’t be even worse for an AGI system with a far larger problem space, it will depend on techniques and methods that haven’t been developed yet, but the problem space will still be vast and challenging for creating guardrails.
Of course, this also makes a huge assumption that it’s even possible to develop appropriate guardrails for all of these conditions in the first place. We’ve just begun to scratch the surface because we assume accidental conditions. And then come the purposeful attacks.
It seems there is much in the way of misunderstanding in the attack surface of these systems. Although a system like a chatbot may seem pretty simple and have a single interface (text), you end up with something reasonably complex from an attack surface perspective. A system with a single interface but with limitless numbers of undocumented protocols, all waiting to be exploited. And, surprise, you find all of this out after you’ve deployed it into the world.
A system with a single interface but with limitless numbers of undocumented protocols, all waiting to be exploited.
Any advanced AI system that requires deployment into the world to develop guardrails will not end well for humanity. The bad news is it’s very likely that an AGI system would also start in this completely open condition, as we see in current transformer models, requiring appropriate guardrails to keep it aligned.
It was obvious to these companies that the guardrails around transformer-based chatbots were insufficient, but these companies moved forward anyway. This should be a wake-up call.
Job Displacement
Let me take a slight detour here to make another point on the current environment of generative AI. One of the other claims from AI maximalists is that people shouldn’t worry about losing their jobs because we’ll figure out how to share the wealth with everyone affected. This has always been a laughable proposition, but we are starting to see how laughable it is.
There is no reason to think that an organization with an AI capable of performing a job will have any consideration for how it affects the humans previously doing that job. I’ve written about AI taking both hobbies and jobs Current generative AI companies including Facebook are not sharing the wealth with freelance writers and artists they are displacing or about to displace, and there’s no reason to think this will change in the future. Job displacement must be addressed by other means, including possible government intervention, policy, and taxation.
Conclusion
In conclusion, I think we got lucky. We blew through a stop sign, and no cars were in the intersection. Many guardrail bypasses we’ve seen haven’t led to any realized harm. We wouldn’t be so lucky with a more advanced system, so it’s time to see this as the warning it was.
Whether we want them or not, chatbots are coming to search engines. Google announced Bard and Microsoft implemented ChatGPT style integration with GPT-4 from OpenAI in Bing. Well, at least people are talking about Bing again, which hasn’t been mentioned in a conversation since its launch.
All jokes aside, this begs the question, is search really ripe for disruption? Many in the tech world seem to think so or at least hope so. My guess is it could be. For a long time, the optimization around search has been to put more eyes on more ads. The question is, is the current crop of chatbots the answer?
Implementations
Just like the answer to most questions regarding the use of new technology, it depends on the implementation. I haven’t really thought about search for quite some time and as a researcher, I admit that my usage of search engines may differ quite a bit from an average user. But, logically, it seems that when people use search engines, they are looking for answers to simple questions or information on current events.
If you look at the trending searches on something like Bing, all of these are current events.
When it comes to information about current events, it will be interesting to see what, if any, impact there is from chat in this space. The current crop of chatbots like ChatGPT were trained on data from 2021. Obviously, a breaking news story won’t be in its memory.
The search engines also seem to be exposing additional functionality in much the same way the ChatGPT demo did, which does open up some interesting possibilities and problems. For example, when people look up medical information and get disinformation or advice in general and get bad advice. People may create different relationships with search engine sites than they have previously, and this isn’t always possible to anticipate ahead of time.
Just the facts, occasionally.
Sometimes you just want the answer to a question. What’s the circumference of the earth? What’s the Tom Hanks movie where he made a wish and transformed from a kid into an adult? Was William Shakespeare a real person? In these cases, it would seem that a chatbot with accurate information could be beneficial and preferential to traditional search. Even in more complex scenarios where you just want an answer, you won’t be bogged down by additional content or need to search different sites to confirm information. This would be great in a perfect world.
But…
We all know the current crop of language models tends to hallucinate facts. Examples of this are all over the web. How this is handled will largely dictate whether these features are useful or useless. I personally don’t think we are anywhere near having a universal knowledge machine that falls within the bounds of acceptable failure and that seems to be playing out in this scenario as Alphabet loses $100 billion dollars as their shares tank after Bard hallucinated facts.
Guardrails
If we conduct a thought experiment where we have a perfect knowledge system, there are still plenty of problems to think about. Consider more controversial subjects or even topics that haven’t been settled. How will these chatbots handle these situations? How will it boil a complex topic down to a simple response? There’s been a heavy-handed approach to the guardrails on these topics, so much so that it borders on the absurd. It’s important as a society that we address uncomfortable topics and come to a resolution. It’s not like we’ve settled everything there is to settle and we can all go home.
My guess is that heavy-handed guardrails will remain in place in the spirit of “safety.” As a matter of fact, it’s already happening as Microsoft’s chatbot refuses to write a cover letter for someone saying it would be unfair to other candidates. So, the most useful features may be out of reach of the people who need them most.
Unfortunately, this filtering will lead to a lot of technology blame as people lead with their biases. We’ve seen this game play out on social media.
Nudges and Forgetfulness
The obvious failures will continue to be identified in the system, but what about the not-so-obvious failures? What happens when something disappears and we don’t notice? We seem to think historical events are seared into our consciousness or as though the Internet doesn’t forget, but it does. Imagine what would happen if Google suddenly stopped returning responses for a historical event. Would that event, in effect, disappear? Probably, for a great many people anyway, depending on factors such as the cultural importance of the event and its age from memory.
This removal of events could be purposeful, such as Chinese search engines not returning results for the famous “Tank Man” image in Tiananmen Square. Baidu even took it a step further and made sure their image-generating software, a competitor to DALL-E, wouldn’t generate a version of the photo either.
What’s even scarier than any purposeful manipulation is when the system accidentally forgets things.
What’s even scarier than any purposeful manipulation is when the system accidentally forgets things. Events could disappear and be lost from collective memory. Equally as bad is that the system hallucinates “facts” around an event, changing its focus. What would the implications be if this happened to a world event as impactful as The Holocaust? Maybe it didn’t disappear completely but the facts around the event changed. What if fifty years from now, the memorials around the event were the only reminder? What if we started to question why we built them in the first place? This absolutely terrifies me and it should terrify you.
For my previous example, I chose the most extreme case to illustrate a point. It’s unlikely that an event such as The Holocaust would be lost to history, but getting the facts wrong is certainly possible. Often the facts shape the impression of the event. This is a conspiracy theorist’s dream.
We need to be careful in the rush to implement a technology, that we don’t lose or manipulate things we actually care about. Could these systems nudge us into forgetting or not caring the way we should? The answer is yes. Will they? That remains to be seen.
New Perverse Incentives
Chat-based knowledge systems make quite a bit of sense for search engine companies as they are data collectors. There’s an old saying that goes something like this, “People lie to their friends, but nobody lies to Google.” Well, nobody is going to lie to a chatbot either, and collecting this information is going to be a gold mine of personal data. This is a power-up to the current crop of perverse incentives.
I made the following joke on Social Media.
[chatbot enabled search engine] What’s the best burger place in town?
[response] Billy’s Burger Joint has the highest rating. Oh, by the way, did you know you could save hundreds by switching to GEICO?
It’s a joke with some truth to it. There’s no way to ad-block that.
There will certainly be hands on the scale nudging people in particular directions. Sure, search engines can manipulate results today, but delivering an answer in the form of a knowledge base holds more of a user’s attention and gives it appears to have both consensus and authority. Imagine giving bad advice, like throwing batteries in the ocean, or if the GEICO commercial was much more sneaky and was part of a recommendation.
Bad For Privacy
Regardless of perspective on usefulness, we can all agree that chatbot integration in search engines is very bad for privacy. This is in an environment where additional logging and analysis will have far greater scrutiny by human eyes, at least in the short term, due to analyzing and improving the implementation. Richer forms of personal data and even thoughts will be collected, stored, and analyzed with the intent of weaponizing this information.
We can all agree that chatbot integration in search engines is very bad for privacy
Are we getting features we didn’t ask for?
I don’t necessarily blame the tech companies here. Sometimes with innovation, you don’t know you want something until you see it. That’s what Silicon Valley is hoping for. Only time will tell if these features truly enhance the search experience, is useless, or turns it into a nightmare. It seems to be leaning toward the latter at the moment.
Regardless of getting things we didn’t ask for in search, it’s certainly coming for other products. 2023 is going to bring a lot of things we didn’t ask for because, currently, these companies are justifying their investments.
Microsoft announced that it was bringing ChatGPT features to Microsoft Teams. This is a head shaker for me. I need to restart Teams up to five times a day due to issues, but now it’s getting even more features. Microsoft Teams now resembles the big box store PC you get loaded up with bloatware.
Microsoft Teams now resembles the big box store PC you get loaded up with bloatware
All of this rush to implement technology we know has issues in production systems reminds me of an Arthur C. Clarke short story called Superiority. In the story, a technologically superior planet is defeated by another one because of their willingness to discard old technology without having perfected the new. Just another example of how fiction can inform reality. In the future, will fiction be our reality?
On a daily basis, I’m bombarded by fantastical news stories and email articles about ChatGPT and how 2023 is the year it’s going to replace “X” job or profession. It seems that no profession is safe. It reminds me of the Dewie The Bear scene from the movie Semi Pro where Will Ferrell screams, “Everybody panic” in a crowded coliseum.
ChatGPT has gone from being a piece of technology to a social contagion right in front of everyone’s eyes. Claims are being made that people can’t possibly believe, but they are making them anyway. I wrote last year about how ChatGPT creates amateur futurists. Well, it seems the floodgates are open.
This has created a group of overoptimistic zealots who’ve basically taken on the identity of crypto bros pumping ChatGPT to 10x. On the flip side, I’m certainly not delusional about the job loss myself. It’s true that generative tools will certainly have an impact on jobs. To take this further, I also expect some surprises and unexpected cases to crop up. The world is a complex place and it’s just not possible to take in all the variables. There are lots of people working in this area, and automation is going to have an impact on multiple professions, even if it doesn’t eliminate them. So, fair enough. But, the extent of the current mania is mind-boggling, with people tossing all manner of delusional predictions at the wall, hoping one sticks.
So, I wanted to add a bit of sanity, so anyone can make sense of this topic and at least try to frame these news stories in the appropriate bucket.
Evaluating News About ChatGPT and Job Loss
There’s a simple way to evaluate the merit of these job loss arguments, even if you aren’t familiar with the technology. Just ask a simple question. Is the cost of failure low? Boom, that’s it. If the cost of failure is low, then there’s a good chance there’s a near-future risk of impact from these tools. If the cost of failure is moderate or high, then there’s little chance of impact in the near term with the current crop of AI tools.
In my previous post, I wrote about how freelance artists will be impacted by the pervasiveness and accessibility of these tools. If you don’t like the art generated by the tool, just generate another one. On the flip side, think about the impact of having ChatGPT be your doctor or lawyer, especially given the fact that these models tend to hallucinate facts.
I know ChatGPT passed the Bar exam, but why is this surprising? I mean, I think most people would be able to pass the bar given an open book and unlimited time. Knowing the answers to questions is a far different matter than applying the knowledge you have to specific situations. I certainly wouldn’t want ChatGPT to argue my case in court, even though creating convincing BS seems to be one of its strong suits.
A Warning
I’ve stated before my biggest concern with all of the ChatGPT hysteria is that people in their mad scramble to compete will end up using this technology in areas where the cost of failure is not low, where there’s the potential for harm and even loss of life.
I worry quite a bit about mental health uses and the medical field in general. Mental health chatbots should be a bit red flag for us. There have been promising uses in using computer vision models to assist doctors in identifying whether tumors are malignant or benign, but that’s far different than having a knowledge system like ChatGPT. Even if these tools reach the status of “pretty good” it will lead to an automation bias where the doctor takes the recommendation of the system by default. This condition would, in effect, make something like ChatGPT, your doctor. Would this be better or worse than clicking through WebMD and diagnosing yourself?
I think people, in their optimism, tend to fill in the blanks, even in the case of very complex problems. In such cases, assuming we are only a couple of tweaks away from solving existing issues. It’s dangerous to bestow some mystical object status on technologies such as ChatGPT when what we need is a realistic analysis of the capabilities and limitations of such approaches. We tend to underestimate the complexities of even simple problems, which makes humans terrible at predictions, but our over-optimism could lead us down a very dangerous path in the next couple of years as these tools creep into critical decision paths.
