Seems everything is clickbait these days. News sources are struggling for the scarce resource of attention. In this environment, a simple task becomes a revolution, and a mundane story gets a new life as a groundbreaking advancement. These titles and the resulting amplification by AI hustle bros provide fuel for the AI hype train, which continues in a circle like an ouroboros. In this post, we’ll look at an example of one of these and talk about the issues and risks.

Taking Spins

I saw this article on Bloomberg that mentions the US Military taking generative AI for a spin. The mental image, along with the photo they used of military cyber operation, conjures thoughts of autonomous systems duking it out or missiles launching. This is by design. It’s meant to create this image for you, but nothing so sensational happened.

What really happened is they built a chatbot over their documents. Doesn’t sound as exciting when you put it that way. For those involved, I’m sure this approach, compared to looking over 13 different manuals trying to cross-reference data and find the right content, felt fast and effective. It may also be the right approach for the problem they are trying to solve. Generative AI isn’t some all-powerful technology. It’s good for some things and not so good for others. This is also something you don’t see covered in news stories.

The military article is far from the most sensational example out there. There’s this little gem.

I probably could have found an even more sensational example to make my point, but recency bias kicked in, and the military story was top of mind since I’d discussed it on social media.

Takeaways

There are several takeaways from these types of titles and stories. Below, I’ll hit a few highlights. Let me specify that what I’m talking about here is mostly related to LLMs. Generative AI related to images, audio, and even video is a different topic and something I’ve written about previously here and here. Success is a different story in use cases with graphics and image modeling. I may write more about this in a future post, but for now, let’s stick to LLMs.

Overhyping

Overhyping in reporting is the norm and not the exception. Most cases where there’s proof of LLM success in various industries essentially boil down to people creating a chatbot over documents, some knowledge base, or even log files. This can certainly be valuable and a productivity boost, but it also sounds incredibly boring, so you end up with titles like ChatGPT is revolutionizing the financial industry, are bankers now obsolete??? Most people will never read the article, just the headline.

Most cases where there’s proof of LLM success in various industries essentially boil down to people creating a chatbot over documents.

Accuracy and Reliability

Let’s punctuate the knowledge base chatbot approach by mentioning when dealing with chatbots over sources of information, there’s no guarantee that the bot will return the correct information. It’s not like creating embeddings and doing similarity searches is foolproof. For high-impact situations with a high cost of failure, this would need to be done incredibly well to avoid a catastrophe, even with a human in the loop. Extra steps to allow a human to verify the right data and data source, ensure the data is up to date, and other additional steps are key in doing this right.

Overconfidence and Extension

Finally, the real danger is looking at the apparent success of something like a bot over a data source and making the leap that the technology has capabilities it doesn’t have or the ability to do even more impactful things with an even higher cost of failure. More impactful things, such as suggesting whether to launch missiles or to drive a tank. These are extreme cases, but it proves a point.

Edge cases and complexity are AI’s worst enemies. You don’t see the edge cases in small experiments or super simple tasks, there may not be any, but as with many use cases with high impacts for failure, edge cases may be everywhere, lurking in the shadows waiting to strike when you least expect them.

You don’t see the edge cases in small experiments or super simple tasks.

This overconfidence and extension of generative AI into other areas where it’s not well-suited will cause damage. As this tech is put in more and more critical paths, it’s only a matter of time until there’s a catastrophic failure.

Conclusion

There are a lot of people experimenting and a lot of money flowing in the generative AI space, and as with any technological advancement, we should be prepared to be surprised. However, take the reporting on generative AI and any stories hyped up by the AI hustle crowd with a grain of salt. Perverse incentives are everywhere. Generative AI may be a good fit for your use case, but beware, this isn’t without pitfalls. Generative AI is far from some utopian technology, and given critical use cases with a high cost of failure, the only winning move is not to play.