I had a great conversation with Aseem Jakhar for CIO.inc and iSMG. We covered topics surrounding AI Safety and Security as well as deepfakes. I explained why I don’t think the misinformation aspect of deepfakes will affect the outcome of elections and provided my opinion on deepfake detectors. We also discuss how we think we need to throw out the rulebook every time a new technology comes along instead of applying lessons learned.
Humans are social creatures, and friendship and love are relationships that run deep in our history, predating Homo sapiens as a species. We associate these relationships as core features of our humanity, but companies are attempting to change this. Every time a new technology comes along, people try to use it to solve complex social issues that have nothing to do with technology, and with AI, it’s happening again. Would you have a chatbot friend? Would you marry a chatbot? There are companies developing products that hope you will. Welcome to the attempted dehumanization of friendship and love.
Solving Non-Problems
There are few things that I can say for sure, but I will say with certainty that the world won’t be a better place when both friendship and love are simulated, and we treat apps like humans and humans like apps.
The world won’t be a better place when both friendship and love are simulated, and we treat apps like humans and humans like apps.
When we take a step back, one thing that should be obvious in the current generative AI craze is that solving non-problems is far easier than solving real problems. This makes sense. There’s a low cost of failure in addressing non-problems. Hell, you don’t even need to _solve_ non-problems to be successful. Let’s think about it: it’s not like the world has a shortage of writers, artists, and musicians. However, those specific non-problems are a topic for another day.
Speaking of solving non-problems, rather than using generative AI capabilities for well-suited tasks, we’ve witnessed an abundance of what I call shitty AI gadgets. What makes them “shitty” is the fact that they don’t actually solve a problem. The focus for many is how “cool” the technology is without emphasis on whether it solves a problem or does anything at all.
This joke by @plibin on Twitter sums up what every single one of these gadgets looks like to me.
Shove generative AI into every technological crevice possible and hope that money sprouts. These products are only good for setting fire to VC money.
When AI is Your Friend, You’ve Got No Friends
The latest shitty AI gadget is called Friend. No, not a joke. And apparently, they spent most of their raised money on their domain name.The Friend gadget also exhibits higher levels of cringe than other gadgets. Other gadgets at least pretend to do something useful. Friend is happy to do nothing at all.
A glance at their commercial is all that’s needed to address doubts about peak-level cringe.
If you want some faith restored in humanity, read the comments. The people writing the comments are human, and they get it—something that the Friend team doesn’t.
Watching the Friend commercial shows just how disconnected these people are from reality. If they are trying to shed conspiracy theories about how they are secretly unfeeling reptilian aliens, they are failing. I mean, what date is going to put up with this? Oh, what is that around your neck? Yeah… I’m sorry, I just realized I have something else to do.
Of course, all of these miss the larger point that someone invested in the Friend device wouldn’t be on a date in the first place, nor would they be out enjoying time with “real” friends.
In looking to optimize everything, including our personal lives, AI friends make sense. It can be all about us. We’ll never have to listen to them tell us about their problems or need to be a shoulder for them to cry on. We may even enter an era where many people don’t know what true friendship feels like.
However, it’s not just loneliness that would drive someone to AI friends or AI lovers. Part of the problem stems from people wanting sure things. There is no perceived risk, fear of rejection, or potential pain. A chatbot will not reject us or tell us things we don’t want to hear—well, unless we don’t pay the bill. This is a powerful pull that some will find attractive.
Isolating Effects
An AI friend or lover wouldn’t have us out living our best lives in the real world because they have an isolating effect. These gadgets provide users with a false sense of companionship and exacerbate the very issues they purport to solve. Rather than going out, we stay home. We stay home and play it safe rather than going on a date and taking a chance on love. If gadgets like Friend were to take off, this would be a net negative for health and wellbeing.
An AI friend or lover doesn’t care if we live or die. It doesn’t care if we are happy or sad. Subconsciously, even if we fool ourselves, we know this.
I’ve mentioned when AI is your friend, you’ve got no friends. I’m not just referring to the uncaring stochastic companion we haul around, but it’s the fact that it makes people not want to interact with us. This aspect further isolates us from the real world. I mean, which of our real-world friends would put up with this?
If I wore the Friend device to a get-together with my actual friends, they would launch a merciless onslaught of insults and fun at my expense, and that’s why they’re my friends. Real friends keep us honest. They don’t let us get full of ourselves, and they don’t just tell us everything you want to hear. This feedback helps us grow and have greater life satisfaction.
Nothing easy is satisfying or worth having. This applies to friendship and love as well. The modern world promises that we don’t need to delay gratification. There’s no sense of investment. Everything needs to be an instantaneous hit of dopamine. There are very few things in life where instant gratification is nearly as satisfying as a delayed gratification activity.
In a recent interview with Eugenia Kuyda, the CEO of Replika (An AI friend company), she said, “It’s okay if we end up marrying chatbots.”
Here’s her response to a very good question:
Question: “When we started out this conversation, you said Replika should be a complement to real life, and we’ve gotten all the way to, “It’s your wife.” That seems like it’s not a complement to your life if you have an AI spouse. Do you think it’s alright for people to get all the way to, “I’m married to a chatbot run by a private company on my phone?”
Kuyda: “I think it’s alright as long as it’s making you happier in the long run. As long as your emotional well-being is improving, you are less lonely, you are happier, you feel more connected to other people, then yes, it’s okay.”
Feel more connected to other people? Really? This is disconnected, disingenuous, or outright stupid. Sure, it could be simple disingenuousness. After all, her job is to hawk her company’s wares. But it should be obvious that being married to a chatbot won’t make us more connected to other people. This situation reminds me of a documentary I watched years ago about people in love with their RealDolls. They’d take them out for drives, sit down for dinner, and watch TV with them, just like another human. You know what the documentary didn’t show? Their friends!
We highlight this disconnect by examining something simple between real friends, like laughter. Is our LLM-powered friend going to make us laugh? I mean, a real guttural laugh that sticks with us? Or will it try to entertain us with a mindless video it thinks we’ll like, generating a momentary chuckle that gets lost in the din of distraction? This and many more cheap substitutions await us, beaten into submission, until we won’t remember the real thing.
More Cringe
With the Friend device, there’s a supreme disconnection from reality, but this isn’t the exception. This is becoming the rule. The Friend gadget is the most obvious incarnation of this, but this disconnection is everywhere in the AI space. This is on full display when we hear the AI tech crowd talking about creativity and creative arts. You can tell these people have never been creative in their life and understand nothing about art. Not even a little bit.
I can’t remember who said this, but someone commented about this situation, saying that the Silicon Valley crowd is just a bunch of people having fun with their friends. There’s some truth to this. It’s like a Silicon Valley garage band, but instead of music, it’s tech. So, it’s not about art or creativity at all. The point is to make “cool” tech, whether it solves a problem or not. It’s a familiar theme.
However, startups are not the only ones exhibiting this cringe factor and disconnection. Google’s new Gemini video mixes both cringe and dehumanization, all in the name of optimization.
There are so many things wrong with this commercial. All these tech types fail to realize that some things are supposed to have friction. Friction is how we grow and become better. Friction is how we challenge ourselves. Even things like second-guessing and self-reflection are a form of friction. We are optimizing all the wrong things, a topic I’ve covered twice before, in Optimizing Away Human Interactions With AI and Outsourcing Simulated Emotional Connections To Bots.
Now, do you think Sydney would rather get a letter from a little girl who struggled to put her words to paper, leaving every imperfection as evidence of her effort and caring, or from Gemini? Which scenario do you also think would be better for the little girl? The answer is so blatantly obvious, well, obvious to us humans, at least. (I’ll avoid making a second alien joke here.)
Technical Issues
So far, I’ve only discussed the human aspects of technology, but there’s a lot more when considering the technical risks. There’s far too much to cover, but I’ll highlight two. For more information, you can read my post introducing SPAR.
Privacy is one of the obvious issues. This is because all of that data collected and shared with our AI friend is valuable. If there’s one thing we’ve learned from recent history, it’s that data available is data exploited with all of our personal thoughts and interactions monetized and weaponized against us. Even if the startup creating the AI friend application claims to respect your privacy, when they get acquired (possibly specifically for this type of data) all bets are off.
At least the people in the documentary I watched years ago didn’t have to worry about their RealDoll harvesting sensitive data and snitching back to the company.
Perverse Alignment
Can we be sure that our AI friend is aligned with our best interests?
A perverse alignment is the alignment of a system to serve the best interest of the company or organization that created it over the user using the system. There is the potential to nudge and push users to do all sorts of things. This may be to buy products or spend more time on the platform. In the AI friend scenario, spending more time on the platform leads to less time with real friends.
It may be difficult to identify when a system is aligned like this. It’s not like our AI friend will respond, “You’ve been worried about car insurance. Do you know who has great car insurance? GEICO.” I made the same GIECO joke back in February 2023 about AI-powered search engines. I gotta get some new material.
Loneliness
I don’t mean any of this to discount the loneliness epidemic happening with younger people. This epidemic is something Jonathan Haidt covers at length and is infinitely more qualified to address than I am. I’ll give you a hint, though. Do you know what he doesn’t recommend? More technology.
This crisis is, at least in part, fueled by technology. There’s something perverse about layering even more technology to solve a human problem. An old saying about treating the symptoms instead of the cause applies here.
There’s a problem with a device that is basically a super-powered inspirational quotes machine, telling us everything we want to hear. We never get better, we never challenge ourselves, and we never encounter real satisfaction. We get stuck in a loneliness loop, with only momentary relief. It’s like if we had an excruciating headache every day, we wouldn’t put up with it, chewing ibuprofen like it was candy to gain temporary relief every day. We’d try to find the cause and address it. This situation is no different.
The AI Religion
Part of the problem is that AI has turned into a religion. I’ve joked about how these devices often resemble communion wafers, but I don’t believe the Catholic Church has had any influence on them. People have talked about AI in more religious contexts, attacking people without enough faith and elevating people they believe are prophets. AI has died, AI has risen, AI will come again.
Religions seldom involve questions, at least not questions that have answers, which is perfect for our current AI moment and aligns with hype. We have to take it on faith that things will get better, and the sermons from AI prophets aren’t merely an attempt to get more profit.
Read Ray Kurzweil’s new book The Singularity is Nearer for more religion-related disconnections from reality. I swear, I’ve pulled a muscle in my neck, shaking my head at all the misperceptions and misunderstandings contained within the book. But Kurzweil is a prophet in the church of AI, and what I’m saying now is blasphemous. If Kurzweil says something, it requires taking it on faith.
When we dig into it, people like Kurzweil, Chalmers, and Clark push a transhumanist vision for humanity that converts us into the Borg, stripping away our humanity and turning us into machines. Resistance will most likely be futile.
What happens when we evolve not to know or have true love and friendship? Will we be better or worse off? Evolving into a machine doesn’t sound appealing to me, but the transhumanist figureheads push the opposite perspective. Transhumanists push the perspective that merging with machines will make us superior humans, but it will most likely make us average machines. That’s not a good trade. I’ll expand upon this in a different post.
Transhumanists push the perspective that merging with machines will make us superior humans, but it will most likely make us average machines.
Don’t fret over my immortal digital soul. I’ve already prayed my five Hail Turings for the day.
Conclusion
As we navigate the sea of innovation porn, let’s not set our course away from humanity. Core features of our humanity make us unique on this planet, not our processing capabilities. We can have technology that works for us and maintains our humanity. Don’t believe those who tell you it’s a tradeoff. They are selling something.
Also, let’s use LLMs for what they are good for, not for friends or lovers. There are plenty of tasks for which you can apply LLMs to boost efficiency and actually solve problems. Do that. Friendship isn’t a technology problem. Neither is love.
If you are hopeful about the future and of technology but remain skeptical of BS claims and other nonsense, hang in there. More and more people are voicing their opinions, and it’s no longer a lonely hill to stand on.
As predicted last year, there is an increased trumpeting of risks focused on deepfakes and AI-generated misinformation surrounding the 2024 US election. Living up to the mantra of never letting a tragedy or potential tragedy go to waste, media organizations have latched on to the doomsday nature of AI-generated misinformation and its potential effects on global politics. I’ve maintained the risks from deepfakes and AI-generated misinformation on elections are overblown, and I’ve seen nothing so far that makes me believe the contrary. In cases of highly polarized topics, the application of AI to misinformation does nothing to change people’s minds.
Over the past couple of years, this has been a relatively lonely position as it seems (or at least seems to me) that I’ve been on an island about to draw a face on a volleyball. Even experts I agree with on other aspects of AI seem to be sounding the AI-generated misinformation alarm the loudest. But it seems the position isn’t so lonely anymore, as more people add their voices to the discourse. However, I’m still struck as to why so many people believe otherwise, but I think I’ve gotten closer to the core of why this remains a powerful illusion.
Why The Powerful Illusion Remains
There is certainly no shortage of perverse incentives around inflating the risks of AI-generated misinformation. For AI companies, this threat demonstrates the power of their technology; for some, it’s the perfect topic to illustrate the need for increased regulation, but for others, it is a potential business opportunity. Many of these situations can be identified easily, but that still leaves many people with this belief. I think I’ve gotten more to the core of why this is such a powerful illusion, and it comes down to something simple: we aren’t seeing ourselves in other people.
People can’t fathom how having a highly convincing image of something won’t fool people. The problem here is we don’t see ourselves in other people. I mean, are we dolts that believe absolutely everything we see? Of course not. But we assume that everyone else is a bunch of idiots who believe everything they see instantly. We see ourselves as outliers instead of the mean. This perspective also ignores the fact that the presence of fake information has been around us the whole time.
We see ourselves as outliers instead of the mean.
When people point to outlandish beliefs like QAnon as proof that this content would fool people, they are also fooling themselves. A fair number of QAnon adherents don’t believe in anything they share. They share it to troll the other side because it irritates people.
Government Perspective
You may be thinking, didn’t the US Justice Department make a big deal about disrupting a Russian AI-powered propaganda campaign? That alone must disprove your argument. Well…
The statement from the Attorney General is pretty strong, but the details matter. You can read the full press release here.
So, let’s read through. Hmm… okay, okay, okay, Spits out coffee WTF? You’re trying to convince people of the impact of these operations, and that’s the best you can come up with? Some sock puppet account with 23 followers?
Okay, well, maybe this account with this few followers shared a viral video that had a major impact. So, how many thousands of views did his video get?
Five? Wait, do you mean five… hundred thousand? No, five. I hate to point this out, but you have to assume at least one or more of those views came from the person monitoring the account. Maybe I’m being too harsh, and this was the setup for the big reveal, so show me the next one.
Seven. Huh. Well, that won’t hit any type of vitality.
Sorry for the low-quality images. Apparently, the government doesn’t train people to take proper screenshots. It conjures images of some analyst pecking the keyboard with their index fingers. However, these examples fail to prove any point supporting the existential threat of AI-generated misinformation.
There is nothing here a human couldn’t do, so if GenAI didn’t exist, Russia would just put asses in seats. That’s what the Internet Research Agency (IRA) did in the past.
AI-Generated Misinformation’s Rough Time
AI-generated misinformation has actually had a rough time for a while. Last year, it was reported that Russia’s Doppelgänger group was struggling to find an audience. Ouch.
We are already in the era of generative AI and deepfakes, and we’ve had multiple high-visibility elections throughout the world. Still, the misinformation aspects of generative AI haven’t affected these elections. As a matter of fact, it unfolded like I said it would with generative AI used for memes, not misinformation. You’d think, with some evidence now, this narrative would let up, but quite the opposite. Many are doubling down.
This disconnect is most obvious with government types. Recently, the director of CISA warned of the risk of US adversaries causing “unimaginable harm to populations across the globe.” This was in reference to adversaries affecting elections. Really? Unimaginable harm, despite the fact that we have evidence to the contrary?
I believe, to some extent, this comes back to incentives. The misinformation topic seems like the perfect example to push for more regulation, and many refuse to take their foot off that gas.
The Reality
Influencing people through AI-generated misinformation is a much harder problem than people want to acknowledge. In a post I wrote last year called Generative AI, Deepfakes, and Elections: Apocalypse or Dud, I introduced something called the Generative Misinformation Cycle to demonstrate the phases and challenges.
With misinformation, your goal is to influence an outcome. This would be to change people’s minds or get them to take action. All of the other phases, such as generating misinformation and amplifying it on social media, only work for a shot at influencing the outcome. Yet it’s these relatively inconsequential activities that so many people focus on. This is where the confusion sets in. To people’s credit, these are the tangible things people can see and measure. It’s much harder to measure a changed mind. However, pointing to these relatively easy activities and saying that their presence indicates that an extremely difficult thing (influencing the outcome) will happen doesn’t match reality.
What AI brings to the table is assistance with the generation of content and some automation activities. That’s it. So sure, you can create a lot more of it and try to have your bots amplify it, but if a misinformation tree falls in the social media woods and only bots hear it, does it really make a sound?
If a misinformation tree falls in the social media woods and only bots hear it, does it really make a sound?
Sowing Confusion
This is about the time when people will mention using generative AI to sow confusion, but sowing confusion is a far cry from sowing influence. It’s not like when people mentally check out due to confusion that their brains somehow revert to an initialized state where they don’t have an opinion.
It’s not like when people mentally check out due to confusion that their brains somehow revert to an initialized state where they don’t have an opinion.
Even when the technique does work, it’s only effective for unfolding current events and on topics where people aren’t emotionally invested, such as an unfolding global pandemic or geopolitical situation far away from home. Sure, this can cause some negative impacts, but if GenAI wasn’t around, bad actors would do this with humans. And, of course, once people’s strongly held beliefs get involved, the trenches are dug too deep to dislodge them.
Poor Research
Poor research also plays a role here. There is so much junk research on AI-generated misinformation. This research often focuses on the wrong questions combined with model capabilities. In the end, you end up answering positively, the wrong question. Take the example below that I shared in February.
Apparently, in his “extensive research,” he missed the fact that concentration camps aren’t typically associated with US political parties. He basically confirmed that LLMs can say mean things. At this point, it should be well known that LLMs can be made to say mean things, and that is a reality we already have today, not in some future state. But this paper does nothing to answer the real question: does the fact that LLMs can say mean things have an impact on human political polarization? After reading this far, you should already know my perspective.
You should also recognize another common theme that this research misses and it’s that instances and capabilities don’t equal impact. I’ve covered this in my previous posts.
Instances don’t equal impact.
Since the Durably Reducing Conspiracy Beliefs Through Dialogs With AI paper is making the rounds again, I’ll point out that I wrote a whole article addressing the issues with that paper.
The Real Risks
There is no shortage of real risks surrounding generative AI. I’ve talked about this at length. I’m far more concerned with the Internet turning into a junkyard or how tech companies are shoving generative AI into every technological crevice imaginable than I am about a theoretical misinformation apocalypse. These activities have far more impact than any usage of generative AI to try and manipulate an election. However, there is also a risk of overly oppressive regulation.
Pretty much everything on the internet is manipulated. You could also say that it is technically misinformation. For example, applying a photo filter adds information to a photo that wasn’t there, and cropping a photo removes information. You could say the same about grammar checkers rephrasing sentences, document summarization, and a whole host of other tools people use on a daily basis. It gets blurry if you are only focused on the information manipulation aspects.
I’ve said all along that regulating underlying technology is a losing proposition. What should be regulated are use cases. This is because AI is a dual-use technology; therefore, the harm surfaces in the use case, not in the technology. It’s a tough problem, and I don’t envy the people trying to address it.
On the other hand, inadvertent misinformation and junk content cluttering the Internet is a real problem. For example, which of the two photos below is the real photo of the Matterhorn?
Surprise, neither of them. Now, if we take into account everyone’s AI-generated blog posts, news articles that don’t get checked, and a whole host of other content that doesn’t rise to the level of existential threat, we have a world cluttered with garbage.
Garbage like Popes in puffer jackets, fake dresses, AI-written nonsense books, and much more. It’s like taking a stroll through a junkyard instead of a pristine forest. Okay, bad analogy. The Internet has always been a sort of junkyard, but now, instead of strolling through rows of junked cars stacked on top of one another, there are junked cars mixed with heaps of household trash strung about, littering the walkway and stinking up the place. We haven’t reckoned with this yet.
Conclusion
This post has remained a semi-written draft since November 2023 because I always feel like I’ve said what I needed to say on the topic. However, I keep getting pulled back in. As a bonus, there are recent updates and more evidence, so my procrastination seems to have paid off.
Unfortunately, the claims of a coming misinformation apocalypse will be with us long after there is more proof to the contrary. Proponents think they’ve found their ultimate talking point to push regulation. Ultimately, this will be the story of the AI-generated misinformation apocalypse that wasn’t.
The tidal wave of information on AI use smashes the shoreline daily, nearly all of it universally positive. News stories, analyst reports, and anecdotes all lead you to believe that you are already in the dust, no matter how advanced you are. Your competitors are smoking you, and everyone is using AI for everything successfully except YOU. This is the massive headwind many of us pushing back find ourselves in, constantly bombarded with news stories and analyst reports, all in service of telling us we are mistaken. A congregation was sent to consult the Oracle of Gartner and your perspectives have been found wanting.
In the space we refer to as reality, what we think we know about AI usage is wrong. So, how did we get here? How have we become so misinformed? The answer is pretty simple: humans. Okay, well, more specifically, surveys and interviews.
Surveys and Interviews
It’s long been known that survey data is only slightly more valuable than garbage, but when it comes to AI, survey data can be a fully engulfed dumpster fire. There are several reasons for this, but the primary reason this is so bad in the AI space is that nobody wants to look stupid or appear behind the curve. So when the analyst, survey taker, or journalist calls, people start parroting.
The primary reason this is so bad in the AI space is that nobody wants to look stupid or appear behind the curve.
Instead of responding with observations they’ve made or activities they are actually doing, they respond with something they’ve heard, articles they’ve read, experiments they hope work, and a host of other things that aren’t true activities. This equates to people expressing their vibes. This disconnection leads to an opening chasm with reality. Since surveys and interviews are the primary methods to collect this type of usage data, that doesn’t bode well for determining realities on the ground. With the hype turned up to 11, a red flag would be when your survey results confirm a 10.
I’ve pointed out this parroting vs. observation issue in my presentations at various conferences for the past couple of years. Although this parroting makes for some wildly comical analyst reports and news stories, it’s rough if you’re trying to make decisions based on them, or worse, when your boss expects you to produce a magic wand and summon the guardians of innovation because you are being left in the dust.
A few days ago, I read an article from the Ludic blog making the rounds that contained the following image.
This is an obvious red flag, and the author points this out in much more eloquent and spicy language. We’ve long known that most AI/ML/DL projects don’t make it into production, but all of a sudden, LLMs come along, and 92% of companies are finding great success. It’s not real. Speaking of 92%…
GitHub reported last year that 92% of US-based developers are already using AI coding tools. The gut reaction is this feels wrong, but hey, it must be true if the data confirms it, right? So, let’s do a thought experiment. Imagine standing in the frozen dessert section of the grocery store, asking people if they like ice cream. Now imagine asking everyone buying ice cream if they like it. What if you only asked two people, or five people, or ten people?
When it comes to usage data, what does “using” mean? What is the definition put forth in the survey? What is the makeup of the population? Most importantly, what do they define as “AI”? All of this matters, and it doesn’t take much imagination to realize how incredibly biased survey data can be. The flames are further fanned by the illusion that models have more capabilities than they do and companies faking demos.
For a deeper response to some of the common points people make, read the article I mentioned. I have some quibbles with some of the article’s content, but all in all, it’s a solid read, and the spicy language makes it all the better.
In a previous post on GPT-4 Lowering Conspiracy Beliefs, I addressed some of these issues surrounding surveys and survey data. I called attention to dark data categories that often surface when surveys are used. I also recommended David Hand’s excellent book Dark Data: Why What You Don’t Know Matters. The book will change the way you view surveys.
The unfortunate reality is that quite a few people have a vested interest in perpetuating these misconceptions. You’d think this would be the companies building these products since it increases their revenue, and this is certainly happening, but most of them aren’t affiliated with these companies. They want to be seen as the ones with the knowledge. They are influencers trying to drive people to their funnels and people in the tech industry who don’t want to look clueless. It’s hard for people to call you out on something when you are saying the same thing everyone else is saying.
Another red flag was shortly after ChatGPT was released. We were inundated with articles quoting opinions by leaders and executives who had never used the technology and had no idea how it worked or even what it was capable of. But it seemed as though we couldn’t get enough.
Dumpster fire achieved.
Ask Questions
We aren’t helpless in these cases. One of the best defenses is asking follow-up questions and probing beneath the surface. I know, I know. We pay (INSERT ORG HERE) a lot of money, and they say… But bear with me a moment.
One recent technique I’ve used is marking up reports, slides, and other information sent to me to help people focus on obvious issues and force some deeper thought. This gives others an idea of where I’m coming from and helps plant the seeds of these questions in people’s heads. Typically, these reports create more questions than they answer, and responding with, “This is dumb,” is not the best tactic. Here’s a recent example I used for a report discussing GenAI’s security use in 2024.
Along with this markup, I also included data in the email questioning the statistical makeup of the data used in the analysis. Funny enough, for this particular section, there was no information about the sample size, industry verticals, or other important information about the makeup of the sample. This is always a red flag. Maybe it was mentioned somewhere else, and I missed it, but it wasn’t available in this section like in the others.
Often, even asking a simple question, “How” can be super effective.
“Generative AI is completely transforming X business or process.” “Oh yeah? How?”
The questions of how, what, and where can be your ultimate weapons in defense against some of this contradictory data. They inform you if there is something real and help you understand if the use cases proposed to support the strongly worded statements made. There may be good answers to these questions that you may want to consider. There are legitimate use cases, and you do want to stay ahead of the curve, so being better informed helps you take advantage of opportunities.
Misunderstanding the data has negative impacts, putting further strain on your resources to create competing solutions or wasting time trying to recreate something that isn’t even working in the first place. Even if another organization successfully uses generative AI for a task or process, you might be unable to replicate it due to different applications, systems, data, and processes.
Even if another organization successfully uses generative AI for a task or process, you might be unable to replicate it due to different applications, systems, data, and processes.
I’m not bashing analysts or survey takers. Conducting surveys without influencing the outcome is hard. That’s why you can find surveys that confirm just about anything. I’m sure the people writing these reports believe what they write, and it matches the data they have.
Conclusion
The grouping of technologies under the umbrella of AI is certainly useful, yes, even LLMs. Non-generative approaches and more traditional ML and DL have been deployed to solve challenging problems for decades. These approaches are already baked into the systems we use. However, the hype and hysteria throw off any real perception, and you often find that complete transformation aligns more with hopes than realities. Ask the right questions and probe deeper to ensure you are making decisions on the right insights. Find use cases of your own and perform your own experiments. You’ll quickly see what’s working and what’s not.
New, deeply integrated AI-powered productivity tools are on the horizon. A recent example is Microsoft’s Recall, but others are also emerging. For example, there’s Limitless.ai, and if you are feeling particularly nostalgic for Catholicism, there’s the 01 Light from Open Interpreter, which allows you to control your computer remotely through a communion wafer.
All of these tools promise infinite productivity boosts. Just thrust them deep into your systems and watch the magic happen. However, when you watch the demo videos and use cases, it’s easy to understand why most people scratch their heads—just as they did with the Humane Pin and the Rabbit. At this point, they are just setting fire to VC money, hoping that a use case will rise from the ashes.
All joking aside, the tools and their usefulness aren’t the subject of this post. I want to focus on the architectural shift and new exposures we create with these tools. This trend will continue regardless of the use case, tech company, or startup.
Note: I’m on vacation and haven’t followed up on Apple’s AI announcements from WWDC, hence the lack of mention here. I wrote most of this post before leaving on vacation.
New High-Value Targets
One of the things that saves us when we have a breach is that all of our data is rarely collected in a single place. Even in particularly bad breaches, let’s say, of your financial institution, there isn’t also data about your healthcare records, GPS location, browser history, etc. Our world is filled with disparate and disconnected data sources, and this disconnection provides some benefits. This means that breaches may be bad but not as bad as they could have been.
A simple way of looking at it is to say our digital data reality consists of web, cloud, and local data. But even in these different categories, there’s still plenty of segmentation. For example, it’s not like website A knows you have an account on website B. Even locally on your computer or device, application A might not know that application B is installed and much less have access to its data. There are exceptions to this, like purposeful integrations between sites, SSO providers, etc., but the point holds for the most part.
With new personal AI systems, we are about to centralize much of this previously decentralized data, collapsing divisions between web, cloud, and local data, making every breach more impactful. The personal AI paradigm potentially makes all data local and accessible. But it gets worse. This new centralized paradigm of personal AI mixes not only sensitive and non-sensitive data but also trusted and untrusted data together in the same context. We’ve known not to do this since the dawn of information security.
This new centralized paradigm of personal AI mixes not only sensitive and non-sensitive data but also trusted and untrusted data together in the same context.
It’s known with the generative AI systems today that if you have untrusted data in your system, you can’t trust the output. People have used indirect prompt injection attacks to compromise all sorts of implementations. We are now discarding this knowledge, giving these systems more access, privileges, and data. Remember, breaches are as bad as the data and functionality exposed, and we are removing the safety keys from the launch button.
How Centralization Happens
I’ve talked about centralizing data at a high level, but what does that look like in practice? Let’s illustrate this with a simple diagram.
We can envision our three buckets of web, cloud, and local data tied together through a connection layer. This layer is responsible for the connections, credentials, login macros, schedulers, and other methods to maintain connections with applications and data sources. The connection layer allows data from all of these sources to be collected locally for the context necessary for use with the LLM. This can either be done at request time or proactively collected for availability. This connection layer creates a local context that threads down the segmentation between the data sources.
The implementation specifics will depend on the tool, and new tools may implement new architectures. So, it’s helpful to back up and consider what’s happening with these tools. We have a tool on our systems that runs with elevated privileges, needs access to a wide variety of data, and takes actions on our behalf. In theory, these systems could access all the same things we have access to. This is our starting point.
These systems will have access to external data, such as cloud and web data and local system data (data on your machine). Your system could collect data from log files, outputs from applications, or even things such as browser history. Of course, they may also have additional logging, such as recording all activity on your system, like Microsoft’s Recall feature, and storing it neatly in a plain text database which now, due to backlash, has caused changes and now, delays.
Having access to data is only one piece of the puzzle. These systems need to contextualize this information to actually do something with it. Your data will need to be both available and readable. This means it’ll need to be collected for this contextualization.
For example, if you ask your personal AI a question like:
What is the best way to invest the amount of money I have in my savings account, according to the Mega Awesome Investment Strategy?
The LLM needs two specific pieces of context to begin formulating an answer to the question. It needs to know how much money you have in your savings account and what the Mega Awesome Investment Strategy is. The LLM queries your financial institution to pull back the amount of money in your savings account. It then needs data about the strategy. Maybe it invokes a web search to find the result and use that as part of the context (let’s ignore all the potential pitfalls of this for a moment.) It uses these two pieces of data as context, either sending them off to a cloud-hosted LLM or using a local LLM.
The data can be queried at runtime or periodically synced to your computer for speed and resistance to service downtime. All this data, including synced data, credentials, previous prompts, and much more, will be stored locally on your system and possibly synced to the cloud. Since this data needs to be readable for LLMs, it will most likely be stored in plaintext, counting on other controls to provide protection. Your most sensitive data is collected in a single place, conveniently tied together, waiting for an attacker to compromise it.
Even scarier, we will get to a point where we can run this query:
Implement the Mega Awesome Investment Strategy with the money I have in my savings account.
This will leave us with systems that not only use the collected data but also take action on our behalf—operating as and taking action as us. I’ve mentioned before that we are getting to a point where we may never actually know why our computers are doing anything, accessing the files they are, or even taking the actions they take. This condition makes our computers far more opaque than they are today.
This example was just a simple question with one piece of financial data, but these systems are generalized and will have context for whatever data sources are connected. There will be a push to connect them to everything. Healthcare, browsing data, emails, you name it, all stored conveniently in a single place, making any breach far worse. It’s like collecting all the money from the regional vaults and putting it behind the window in front of the main bank.
There’s Gold In That Thar Data
If data is gold, this is an absolute gold mine. As a matter of fact, this data is so valuable it will be hard for companies to keep their hands off of it in a new data gold fever. So, although up to this point, I’ve been talking about malicious attackers having access to this data, it’s also the case that tech companies will want this data as well, and all efforts will be made to access it and use it. This will be through both overt and covert methods. Turning settings on by default, fine print in user agreements, etc.
If you think the startup developing the tool says they respect my privacy and won’t use this data for anything, think again. Even if this statement were true, wait until they get acquired.
Conclusion
First things first, we need to ask what we get from these integrations. Are the benefits worth the risks of security and privacy exposures created by these new high-value targets? The answer to this question will be a personal choice, but for a vast majority, the answer will be no. At this point, there is still more hype than help.
Authentication, authorization, and data protection need to be key in these new architectures. Not only that, but we must put our own guardrails in place to protect our most sensitive data. This is all going to be additional work for the end user. These systems act as us accessing our most sensitive data. Anyone able to interact with them is basically us. There are no secrets between you and your personal AI. Companies also need to ensure that users understand the potential dangers and pitfalls and provide the ability to turn these features off.
There are no secrets between you and your personal AI.
Tech companies must start taking this problem seriously and acknowledging the new high-value targets they create with these new paradigms. If they are going to shove this technology into every system, making it unavoidable, then it needs to have a bare minimum level of safety and security. It’s one of the reasons I’ve been harping on my SPAR categories as a baseline starting point.
Please stick with me for a moment. I haven’t gone off the deep end or witnessed any UFOs in my backyard.
If the likelihood of advanced life in the universe is so high, why don’t we have evidence of its existence? This is the Fermi paradox, and one astrophysicist has a new theory. A new paper by astrophysicist Michael Garrett proposes a theory that advanced AI might be the great filter that makes advanced technical civilizations in the universe rare. So, as these alien civilizations create artificial superintelligence (ASI), this technology eliminates them because their goals aren’t aligned. You know, that old chestnut. We’ve now ported AI doom scenarios to the cosmos.
I’ll acknowledge that there is a non-zero chance that Garrett’s theory is true. It’s a bit more science fiction than a potential reality for me, but let’s take the question the paper poses and use it for a thought experiment. Is AI acting as a great filter that advanced civilizations can’t get past? Possibly, but for a different reason than improper alignment leading to their extinction. What if the AI and its alignment weren’t the problem? What if life was the problem? Let me introduce a new theory that I feel is more plausible than ASI destroying intelligent life through elimination.
A New Theory for the Fermi Paradox: Entertainment
Let’s start by using the assumptions made from the paper itself.
“We start with the assumption that other advanced technical civilisations arise in the Milky Way, and that AI and later ASI emerge as a natural development in their early technical evolution.”
What if AI ruined these civilizations in a different way? Instead of eliminating them, a perfectly functioning and aligned ASI system could remove the capabilities and ambition of the species that created it. Free from not having to toil away (or work as we’d refer to it), life shifts to entertainment over exploration. I use the word entertainment here, but I mean this as whatever form gratification might take for the species.
Life throughout the cosmos may very well prefer simulation to reality.
Life throughout the cosmos may very well prefer simulation to reality. After all, why explore the universe with all of the dangers, hardships, and time commitments as a member of a collective crew when you can explore a simulation of the universe as a swashbuckling hero? Why wait decades or centuries for gratification when you can immediately get it? It could be that more immediate forms of gratification trump all else, not only on Earth but also through the cosmos. This means that gratification through simulation could very well be a civilization’s version of soma.
It’s possible that once truly capable AIs take care of the heavy lifting of keeping civilization going, civilization loses its fundamental sense of identity, which we would call humanity. This condition ushers a retreat inward instead of an expansion outward. A byproduct of this movement could fuel a lack of interest in procreating, and unless the technology keeps them alive indefinitely, they could end up dying out. To me, this is far more likely than rogue AIs exterminating their creators on every planet that has a form of ASI.
Many here on Earth would consider this scenario a utopia and state that this is the goal. But as with all utopias, they mask dystopias. What if some fundamental catastrophe happens, and the civilization needs to flex its cognitive resources to solve it? There wouldn’t be any. This civilization would be lost as it is forced to deal with problems it has never encountered without the systems it relied on and desperately needs skills it has never developed. A perfectly aligned AI system isn’t the end of the story and brings challenges of its own.
Imagine us discovering these new worlds and finding them abandoned, devoid of biological, intelligent life but filled with their remnant technology, humming along as if nothing happened. It would appear as if the non-intelligent life were intelligent. Maybe even as if plant life constructed their supercomputers. In a sort of way, if aliens visited our planet devoid of humans, it may appear as though cows constructed skyscrapers.
If you think this couldn’t happen, that some part of the population of alien species would maintain an independent spirit, I’m not so sure. I said in a previous post that we can’t seem to view the past without the lens of the present, and we can’t envision the future without using the same lens. This framing affects our thinking. Like on Earth, AI doesn’t exist in a vacuum; it exists alongside many other technologies. These technologies will shape and impact the population before the arrival of ASI, so think about things like social media and video games here on Earth.
Earth
I’m not trying to humanize aliens, but you’d have to assume that an intelligent species with the motivation or need to explore the universe and try and find life on other planets would share some attributes with us. Even attributes like curiosity, ambition, and necessity would be at least three traits we’d have in common. So, let’s use our humble corner of the galaxy for a moment.
Let’s assume that AI delivers on all its promises and people don’t have to work. What next? How would people spend their time? There is a romantic view that we’d all become philosophers and artists living a life of leisure and peace, but this isn’t humanity’s destiny. This narrative is a concoction being marketed to us under the guise of progress to temper unease. The fact of the matter is that generative AI today is devaluing these creative enterprises to a point where nobody will benefit from them in the future.
There’s no doubt that if AGI/ASI arrived tomorrow and nobody had to work, many people, possibly most people, would choose entertainment along with other mindless, time-wasting activities like swiping and scrolling. People would still create content, but more for attention than money. However, there would still be some who would maintain their curiosity and want to explore the cosmos.
By the time ASI arrives, we won’t be the same people we are today.
Tomorrow is easy to imagine, but what about 20 years or more? The technology we have today and the technology in development in the short term will change us to a point where we may not recognize ourselves at this time. This has already happened in my lifetime.
By the time ASI arrives, we won’t be the same people we are today. The technology that fills the gap will bring more entertainment, change our values and habits, and bring far more cognitive offloading. This is a massive recipe for transformation that may alter humanity’s trajectory as our focus and goals shift before the arrival of AGI and ASI. While we worry about AI alignment, we miss the possibility that maybe it’s us humans who need to be aligned.
In a previous post, I mentioned that this is how we get to the world of Idiocracy. So it could be that all of these alien civilizations took the Idiocracy route, prioritizing entertainment and losing ambition for answering the big questions about the cosmos. Ultimately, they may not care if they are alone as long as they are entertained.
Besides, if ASI wanted humanity gone, it could just play the long game. We’ve been inventing ways to make ourselves extinct for quite some time. Maybe entertainment is one of those ways.
Conclusion
Is my theory an explanation for the Femi Paradox? I have no idea. I know it’s more probable than the extinction scenario, even here on Earth. A perfectly aligned system brings challenges of its own, and I’m writing this as a cautionary tale for our species. While we search for technological progress and a cure for hardships, we must ensure we put some guardrails around our humanity. Otherwise, we have a roadmap to where this leads: cows constructing skyscrapers.
You might wonder why AI companies are working on seemingly simple and unimportant advancements in AI when there are much more significant problems to solve. Why would companies trying to create AGI get sidetracked by focusing on potentially already-solved problems? A couple of examples are OpenAI’s voice cloning, Google’s VLOGGER, and Microsoft’s VASA-1.This research, for many, only seems to have use cases for fakes and frauds, but I believe this work signals something much deeper: that we could be near the peak of LLM capabilities. With AGI off the table, it is time to go deep and get very personal.
Peak LLM
Although you can do some cool things with LLMs, and we’ll no doubt see further applicability in other use cases, it’s a far cry from their touted value. You know what I’m talking about, the more impactful than the printing press crowd that still seems to swarm every conversation on the topic. These people talk about 10x, 100x, and even 1000x productivity boosts with LLMs. Compared to bold AGI claims and nonsense productivity levels, a 10% efficiency gain seems inconsequential.
The Wall Street Journal reported that the AI industry spent $50 billion on the Nvidia chips used to train advanced AI models last year but brought in only $3 billion in revenue. Ouch! There is reporting on the dismal outlook for generative AI, and some foresee a new Dotcom crash.
People have become more skeptical of claims (as they should), and it seems that many more people are noticing. You can’t believe the demos you see. Many are highly controlled or manufactured altogether. Even the SORA demo that everyone lost their minds over wasn’t what it purported to be.
LLMs are under-delivering on their overhyped promises.
I don’t know what to think about the economic angle. It’s not my area of expertise. I just now know that LLMs are under-delivering on their overhyped promises. Where leads economically, I don’t know.
Many LLMs, including open-source models like Llama 3, are catching up to GPT-4. Even if they don’t have the exact level of performance, they are close, which should tell us something. We may be hitting peak LLM capabilities. This means GPT-5 won’t be AGI or exponentially better than GPT-4. GPT-5 may be better than GPT-4 in some ways, but it is far from a groundbreaking explosion of capabilities.
This lack of performance isn’t going unnoticed at the companies building the technology either. This is why a new approach is needed by companies looking to monetize AI investments further. There’s about to be a shift away from a focus on AGI (although they’ll still talk about it) and ever more capable models to you. That’s right, you.
You’re Next
Just because we may be hitting peak LLM capabilities doesn’t mean things will stop. When you’ve reached the limit of going wide (general), you go deep (personal). This will be a sleight of hand shifting from purely training larger models on more data, creating more capabilities in a broad sense, to deeper, more personal integration.
These companies will make it all about you, not because you are the most important aspect, but because you are where the data is. With systems that are closer to you and more integrated with your data and activities, these companies are hoping to make the products more sticky, with the beneficial exhaust of having access to all your data.
The hope is that an epiphany will sprout from your screen as you find the same tools you previously could take or leave now indispensable. Or maybe even fool yourself with the tech, as the public launch of ChatGPT showed. ChatGPT became a social contagion not because people found it so indispensable but because we are bad at constructing tests and good at filling in the blanks.
But don’t take my word for it. Sam Altman has already started pivoting in this direction. Here’s what he says about the goal of AI: “A super-competent colleague that knows absolutely everything about my whole life, every email, every conversation I’ve ever had, but doesn’t feel like an extension.” That’s pretty creepy. But there’s more.
You can make the tech more sticky by allowing people to personalize and customize in more advanced ways. Technology like voice cloning and animating faces supports this customization aspect. When you can choose whoever you want to be your assistant’s AI avatar, you can anthropomorphize it more. How would you feel if a random stranger used your face and voice as their personal assistant? What about a family member? Is this creepier still? Oddly enough, it serves no purpose for the individual user. It doesn’t make the tool any smarter or more capable. It only exists to manipulate us or allow us to manipulate ourselves.
In the end, you’ll be blamed for LLMs’ lack of success by not allowing them to plunge deeply enough into your life. There’s a saying that if you don’t pay for something, then you are the product. Well, in the age of generative AI, you can pay for something and still be the product. The future’s so bright 😎
Even Deeper
AI companies are doing their best to make this technology unavoidable. We are getting AI whether we want it or not. It’s being baked into the very foundations of our computing systems, and even your humble mouse hasn’t escaped this integration.
How you deactivate these integrations will be anyone’s guess, as the flood of new integrations infects every application imaginable. A security check will be due soon, but security issues aren’t the only problem. As I’ve said, we are creating a brave new world of degraded performance. In an attempt to make hard things easier, we may make easy things hard.
Applications of narrow AI are cool and can be incredibly useful for certain tasks, but does it warrant hooking everything up to LLMs and hoping for the best? I don’t think so, and this approach is fairly misguided, opening us up to unnecessary risks.
Conclusion
We must be much more selective before blindly accepting deep data access and personal integration for these tools. This can start with a few relatively simple questions. What do we hope to gain from this access? How will this provide a measurable benefit? And, most importantly, are the trade-offs worth it? The answers to these questions will be different for everyone.
In many cases, it appears that for the small price of your soul, you can appear and sometimes feel marginally better in some aspects but be measurably worse in others. Does that sound like a good trade?
So, let’s talk about posthumanism for a moment. Yes, posthumanism is actually a thing, and it can sound like a rather odd movement to cheerlead. After all, we as humans aren’t done being human yet. Posthumanism’s adherents are anxiously awaiting the next stage of human evolution, homo technologicus. Yes, it’s also a real thing. I’ve also heard terms like techno-progressivism thrown around. As serious as some of these people may be, their concepts are surrounded by techno-utopian bullshit.
As amazingly silly as this sounds, their views aren’t far off from those of many people these days. Everyone from pure techno-utopians to level-headed “normal” people is kinda thinking the same thing. Let’s slap a bunch of tech inside our bodies and see what happens.
My goal with this post isn’t to address all the narratives or poke even more holes in the logic. I’m writing a book covering this and other topics. For this post, I want to point out a few glaringly obvious issues that should get more attention. The point of this post is that there is no free lunch regarding human augmentation.
Human Augmentation Must Be Universally Good, Right?
I never cease to be shocked at the casual nonchalance of people discussing slapping a bunch of tech inside their bodies, melding our brains with machines. I realize there’s a cool sci-fi aspect to it, but in real life, we have things called consequences. It’s different if there is a cognitive or motor impairment that the technology corrects for, another thing entirely when no impairment exists.
As a security researcher, I can’t bring myself to imagine these systems not being vulnerable to attack and, almost as bad, being used to manipulate us. We like to think of ourselves as the pillars of agency, but in reality, we can be nudged to do all sorts of things, resembling more automatons than humans.
This means that any of these systems would need to have a safe technical baseline. For a basic framework of a safe baseline, see the SPAR categories I’ve outlined previously.
I could address many other technical issues, but for the sake of this conversation, let’s call it a perfect technical implementation. A cognitive symbiosis of mind and machine without any technical issues or glitches. It is a completion of the techno-utopian dream.
Let’s look at why, even in a perfect implementation, there is still no free lunch.
Socrates
To look forward, let’s look back. This is Socrates. Totally not a fake photo, by the way.
Socrates has become a popular punching bag for the AI crowd. Apparently, dunking on a 5th-century BCE philosopher has become some sort of modern-day sick AI burn. So, what sin did Socrates commit that is so egregious to AI leaders today? He was against writing things down.
Socrates worried that writing things down would affect his memory, so he became a punching bag. However, what many don’t realize is that he wasn’t wrong. Writing things down can negatively affect your memory.
We can’t seem to imagine the past without viewing it through the lens of the present. People’s memories were far better in the past than they are today, even pre-social media and the attention apocalypse. It doesn’t take much thought to recognize this. In ancient times, when most people couldn’t read or write, the only place to store knowledge was in their heads. Even asking someone else, you were querying tribal knowledge stored in someone’s head. To his credit, Socrates stumbled onto cognitive offloading and recognized one of the effects.
Ultimately, we are better off for writing, and the benefit of writing things down far outweighs the benefits of a localized, tribal memory, even if individual personal memory is decreased. There are also other interesting effects of writing that Socrates missed, such as exploring thoughts and ideas and some of the memory-reinforcing effects. So, let’s forgive a 5th-century BCE philosopher their faults and focus on what he recognized for a moment: cognitive offloading.
Cognitive Offloading
Cognitive offloading is using physical action to alter the information processing requirements of a task to reduce cognitive demand. We all do this every day. If you’ve ever left yourself a note or set up a meeting in your calendar application, you’ve performed cognitive offloading.
This activity is beneficial since we only have so much cognitive capacity. It’s not just memory but decision-making skills as well. There’s a famous story about President Obama and why he only wore gray or blue suits. He was paring down his decisions.
I know it seems I’m making the posthumanist argument for them, but bear with me. Not all cognitive offloading is the same. In 2016, I heard the evolutionary biologist David Krakauer discussing cognitive artifacts on the Making Sense podcast. This was in the context of discussing complexity and stupidity. He referred to complimentary and competitive cognitive artifacts.
Without being too wordy, complementary cognitive artifacts help you create a model of the problem and are tools that rewire our brains to make problem-solving more efficient. These are things like maps, language, and even the abacus.
Competitive cognitive artifacts don’t augment our ability to reason but instead replace our ability to reason by competing with our own cognitive processes. Classic examples are the calculator or GPS navigation.
The interesting thing here is that complementary cognitive artifacts have imprinting and additional positive effects. For example, being proficient with maps increases spatial awareness. On the other hand, with competitive cognitive artifacts, you are probably worse off when the artifact is removed. For example, using GPS navigation systems degrades spatial awareness, so when it is removed, you are less capable than before.
I’m not arguing that we should destroy all calculators (or GPS navigation systems); I’m only pointing out the impacts of reduced cognitive function. It’s also interesting to consider that AI tools are almost universally competitive cognitive artifacts. We assume, wrongly, that there isn’t a cost to this augmentation. I mean, everything has tradeoffs in life. Technology is no different.
To avoid making this blog post a book a whole book, let’s look at memory.
Memory Storage
Most humans realize that memory is a limitation. Unless we are savants, there are only so many things we store in our heads. But we may be taking the offloading of memory too far. Let’s think about what we are actually doing. As humans, we are transitioning from knowing things to knowing where things are stored. We’ve treated this as universally beneficial without considering side effects.
We are transitioning from knowing things to knowing where things are stored
AI didn’t initiate this trend, but it has accelerated it, especially with systems like ChatGPT, which people use as oracles. This means the information we are retrieving may never have existed in biological memory in the first place and, more interestingly, may not be stored even after we retrieve it. Anyone who’s ever followed a YouTube tutorial on how to do something and, despite performing the task, had to review it again the next time can attest to this.
This brings up some interesting thought experiments. Is someone who doesn’t have any deep knowledge contained in their biological memory smart? After all, information on astrophysics is a search away. Would we say someone proficient at searching Google or prompting a language model is smart? Okay, let’s phrase the question a different way.
Is an average human + Google (or insert favorite AI tool here) smarter than Einstein or Von Neumann? After all, they have access to far more information far more quickly than either of those scientists ever did. Of course, the answer is no. We instinctively know there’s something more to knowledge and intelligence than merely knowing where data is stored or getting a summary from a document.
There’s no doubt that people may feel like Einstein, but that’s a topic for another day.
Human memory is getting worse, no doubt, due to technology. At the veterinary office I visit, I’ve seen people walk out of the exam room to use the restroom, go to the front desk, or go out to their car, and not remember which exam room they came out of. A clear degradation of spatial memory. These weren’t kids on TikTok or people staring down at their phones. People of all ages are represented.
But, not all memory tasks are straight lookup tasks, and memories spontaneously emerge. Sometimes, I bust out laughing when a memory pops into my head. This spontaneous surfacing has benefits, such as the creation of epiphanies and novel concepts creating a satisfaction that can’t be replicated with technology. What happens when this spontaneity disappears? Not only are we worse off, but it leads to more questions.
How do we develop novel ideas and concepts if we don’t have the right knowledge in our biological memory? It’s one thing to have knowledge and some novel concepts in memory and then explore external storage locations for further data. It’s another thing entirely to have no deep knowledge contained in biological memory and expect novelty to emerge because of access to external storage. I know the techno-utopians would say that we’ll build algorithms for this, but it’s a challenging problem and not the same thing and wouldn’t lead to the same results.
Humans + AI = Superhumans?
Human augmentation with AI is being sold as an intellectual get-rich-quick scheme, but the reality is gaining knowledge is hard. Sometimes, it is very hard, and there aren’t any shortcuts today, no matter how many prompts we create or documents we summarize. However, cognitive illusions are easy to come by. We end up fooling ourselves into thinking we know more than we do. Once again, AI didn’t start this trend. It’s merely the accelerant.
There’s a fundamental illusion clouding many people’s perceptions. Just as we can’t seem to view the past without the lens of the present, we can’t envision the future without using the same lens. We tend to assume we’ll keep our same faculties and gain more capabilities, resulting in some sort of win-win situation.
We mistakenly think human augmentation makes us superhuman, but in reality, it probably doesn’t. Despite knowing where information is stored and being able to perform some additional computational tasks, which may give us some superhuman capabilities in a few narrow areas, the reality is it may not make us superhuman overall and probably makes us worse. These additional capabilities will create very real and expanded blind spots and deficiencies. Of course, these won’t be identified until far too late, and everyone will claim not to have seen them coming.
These additional capabilities will create very real and expanded blind spots and deficiencies.
We haven’t even asked ourselves what we hope to get from this symbiosis or augmentation. There is just this generic sense of “enhancement,” but nothing overly specific. It’s one thing if the augmentation addresses some deficiency, such as reduced cognitive or motor function, but what are we addressing when a perfectly functioning human decides to augment themselves?
The reality is that when this symbiosis happens, we will become completely dependent on technology for far more than complex tasks; we will also be dependent upon it to function in our daily lives, even for simple tasks. This is because we will use the resources to offload even more cognitively, regardless of task complexity. Who wins in this scenario? Tech companies? Society? Us? At this point, will the technology still be working for us, or will we be working for the technology? More importantly, at what point do we stop being recognizable as humans?
Parting Thought
I’m not opposed to human augmentation or even being augmented in some way myself. But as an adult who has lived on planet Earth for a bit, I want to understand the tradeoffs. Understanding the costs is essential to determining whether the augmentation is worth it. It seems that in some cases, we may be stiffed with a hefty bill that we never would have agreed to ahead of time.
When it comes to being human, there are certain things we’d like to protect and certain things we are fine giving up. This will be different for each individual, but we all have this. These considerations will have to be part of our future decisions.
Our brains seek to free up resources and limit the amount of work they perform to create brain capacity for other tasks. In short, our brains seek to offload as much as possible. This is something we don’t consciously realize. It’s one of the reasons we prefer getting an answer to solving a problem. Our brains seek the offloading path, whether it’s helpful or not. This evolutionary quirk may have served us well in the past, but with technological advances, it may not serve us well in the future.
The movie Idiocracy is a cult classic that has been quoted more and more over the past few years. Here’s something to think about. It could be that Mike Judge got the future outcome of the movie’s setting right but just got the premise wrong. The only way the world of idiocracy could have come about is if highly capable AI had been in the background, making everything work and, of course, manufacturing Brawndo. Brawndo has electrolytes!
There has been some buzz over a new paper called Durably Reducing Conspiracy Beliefs Through Dialogs With AI. You can read the paper here. In their paper, they found a roughly 20% reduction in conspiracy beliefs, and this reduction was still in effect two months later. So far, so good.
To be fair to the paper’s authors, it has not been peer-reviewed. I’m also not directly refuting their results, but I am questioning their data, which affects the results. In this post, I’d like to highlight a few items that should raise eyebrows as well as give people some ideas for a more critical eye when it comes to data. Far too often, people read the abstract of papers like this and share the post, but we must give a more critical eye to research rather than just taking results for granted.
The Obvious
There are a couple of obvious observations here that I’ll quickly note and move on since it’s not the subject of this post.
If the outcome is true, then the inverse is probably true. So, if AI can break people’s conspiracy beliefs, it also has the power to implant them.
This wasn’t the default version of GPT-4. They used GPT-4 Turbo and specifically tailored it to have these dialogs.
This study was done outside the normal area where people with conspiracy beliefs would encounter such tools.
In an age of personal AI, people with conspiracy beliefs may have their own AI that they’d use to refute these tools. So, the AI vs AI fight club is probably closer than we think.
Now, let’s look at the bigger problem.
The Bigger Problem, Data
As soon as I saw this study, a huge question surfaced. Where did they find these conspiracy theorists?
Conspiracy theorists are typically suspect of government and academia. On a recent road trip, a truck passed me with a bumper sticker stating, “Not a Ph.D.” This was followed by several other bumper stickers claiming the election was stolen, Q something or other, and a few more that could be easily guessed. However, this person didn’t appear to claim the earth was flat, so I guess that’s a silver lining, but there’s still a catch.
Conspiracy beliefs are like Pringles. It’s hard to have just one. These beliefs make them more paranoid and suspect, especially in studies and academia. I find it hard to believe that these true believers would willingly participate in such a study and that a few rounds with a chatbot would reduce their beliefs.
In the real world, family members and loved ones have implemented similar strategies to no avail. These people have emotional ties and strong bonds, and even that wasn’t enough to change their minds. The only thing that helped was a completely different tactic, not refuting the beliefs but telling people that they were worried about them and that they cared for and loved them.
To change beliefs, one’s mind needs to be open to new possibilities. Conspiracy beliefs are powerful because they allow one to play both the victim and the hero at the same time. I wrote about this aspect back in 2020 with an article called The Cult of Conspiracy Theory. All of this is evidence of pretty strong cognitive distortions that conspiracy theorists hold. So, back to the question. Where did they find these conspiracists?
They used a product called CloudResearch Connect. This is a site that pays people to take surveys.
I read the website but didn’t do a deep dive into this company or examine its methods more closely since this wasn’t important for the points I’m making in this post.
Now that we know the source, a new question arises. Are over a thousand conspiracy theorists waiting to complete surveys on Connect? It’s possible but doubtful. Here is another question to ponder. Is it more likely that people with durable conspiracy beliefs are a part of CloudResearch Connect, or is it more likely that people looking for extra cash would be willing to answer questions with the persona of a conspiracy theorist?
When reduced to this question, the answer is fairly obvious. I also get it. It’s hard to find people to participate in studies when you are researching topics like this. Far too often, this research is done with participants made up of the student body—hardly a representative sample of the real world.
When Data is Dark
The participant data in this paper covers multiple categories of David Hand’s Dark Data. In his book, Dark Data: Why What You Don’t Know Matters, he covers 15 categories of what he calls dark data. It’s an illuminating read, and once you see data in these categories, you won’t be able to unsee them. Hence, despite reading the book four years ago, the categories stuck with me.
Immediately, I spotted at least three main categories of dark data.
Choosing just some cases
Self-Selection
Feedback and Gaming
Choosing Just Some Cases The researchers choose to count on Connect for their entire sample. The sample provided by Connect may not be representative of the real world. This may cause them to miss the effects of real people with conspiracy beliefs and instead have people posing as conspiracy theorists.
Self-Selection All the people filling out the surveys volunteered to be part of the study. There may be a stark difference in how people who decide to be part of the study answer the questions from those who do not. In this case, actual conspiracy theorists.
Feedback and Gaming This happens when the feedback itself influences the values of the data. In this case, getting paid to be part of the study. Participants may be more likely to answer favorably to certain questions, playing the role of a conspiracy theorist to get paid.
In fairness, the paper’s authors were aware of some of these issues and tried to account for automated responses and confidence in their conspiracy beliefs. This can be seen in the Materials and Methods section of the paper. I think, in the end, there just weren’t any true conspiracy theorists on this site. Well, that’s my theory anyway.
Most Surveys Results Are Garbage
Let me give an example of how to think about these scenarios more simply. If I stood in the frozen dessert section of the grocery store and asked every person buying ice cream if they liked ice cream, I’d be in the high 90% range for respondents. This would be different if I asked people at the entryway to the grocery store or on the busy streets of Manhattan. I’d also have to ask if my sample size was representative of the population I was measuring. Did I ask everyone walking into the grocery store or only ten people?
I saw a survey a few years ago that made broad statements about the cybersecurity industry worldwide by surveying less than 100 people. Is it possible for less than 100 to represent an industry with many different focus areas across many different industry verticals, in many different countries, and at various levels of seniority? Of course not, but this type of thing happens all of the time. Survey data quality has reduced dramatically in value over the years. This, combined with poor data collection, equals garbage.
For a quick gut check on surveys.
Does the sample size seem representative of the population?
Does it seem likely people would answer questions differently based on the collection?
Did respondents self-select?
Here’s an example to clarify the self-select question since it may be blurry. Say the International Pizza Lovers Association sent out a survey. As a pizza lover, I may be more likely to respond than someone who is lukewarm or doesn’t like pizza. So, I self-select based on my enthusiasm for the topic, thereby biasing the data collection. This can also be negative. Anyone who’s spoken at conferences and gotten feedback knows that the people most likely to fill out feedback forms are the ones who didn’t like that talk.
Conclusion
I think the manipulation of humans by bots is a fascinating topic that deserves more research. However, for any research results to hold, reliable sampling of the population being studied must be conducted. Otherwise, we may be fooling ourselves. There’s an ongoing replication crisis in academia across various disciplines. In this post, we examined one of the contributing factors to this crisis.
Everyone from tech companies to AI influencers is foaming at the mouth, attempting to get you to mainline AI into every aspect of your personal life. You are told you should outsource important decisions and allow these systems to rummage through all of your highly personal data so you can improve your life. Whatever that means. With the continued push of today’s AI technology even deeper into the systems we use daily, there will inevitably be a data-hungry push to personalize this experience. In other words, to use your highly personal, sensitive data to whatever ends a 3rd party company would like.
Although we may have a gut reaction that all of this doesn’t feel right and may be dangerous, we don’t have a good way of framing a conversation about the safety of these tools. The ultimate question many may have is, are these tools safe to use?
The answer to this question comes from analyzing both the technical and the human aspects. In this post, I’ll address the technical aspects of this question by introducing SPAR, a way of evaluating the technical safety attributes, and discuss what it takes to achieve a safe baseline.
Personal AI Assistants
Personal AI assistants are the next generation of AI-powered digital assistants, highly customized to individual users. Think of a more connected, omnipresent, and capable version of Siri or Alexa. These tools will be powered by multimodal large language models (LLMs).
People will most likely use the term Personal AI (yuck) for this in the future. I think this is for two reasons. First, AI influencers will think it sounds cooler. Second, people don’t like to think they need assistance.
Personalization
Personalization makes technology more sticky and relevant to users, but the downside is that it also makes individual users more vulnerable. For personal AI assistants this means granting greater access to data and activities about our daily lives. This includes various areas such as health, preferences, and social activities. Troves of data specific to you will be mined, monetized, and potentially weaponized (overtly or inadvertently) against you. Since this system knows so much about you, it can nudge you in various directions. Is the decision you are about to make truly your decision? This will be an interesting question to ponder in the coming years.
Is the decision you are about to make truly your decision?
Safe To Use?
Answering whether a personal AI assistant is safe to use involves looking at two sets of risks: technical and human. You can’t evaluate the human risks until you’ve addressed the technical ones. This should be obvious because technical failings can cause human failings.
On the other hand, this isn’t about striving for perfection either. Just like drugs have acceptable side effects, these systems have side effects as well. Ultimately, evaluating the side effects vs the benefits will be an ongoing topic. If a technical problem with a drug formula causes an excess mortality rate, you can’t begin to address its effectiveness in treating headaches.
SPAR – Technical Safety Attributes
Let’s take a look at whether, from a technical perspective, an assistant is safe to use. Before introducing the categories, it needs to be said that the system as a whole needs to exhibit these attributes. Assistants won’t be a single thing but an interwoven connection of data sources, agents, and API calls, working together to give the appearance of being a single thing.
For simplicity’s sake, we can define the technical safety attributes in an acronym, SPAR. This acronym stands for Secure, Private, Aligned, and Reliable. I like the term SPAR because humans will spar not only with the assistant but also with the company creating it.
There is no such thing as complete attainment in any of these attributes. For example, there is no such thing as a completely secure system, especially as complexity grows. Still, we do have a sense of when something is secure enough for the use case, and the product maker has processes in place to address security in an ongoing manner. Each of these categories needs to be treated the same way.
Secure
Although this category should be relatively self-explanatory, in simple terms, the system is resistant to purposeful attack and manipulation. These assistants will have far more access to sensitive information about us and connections to accounts we own. The assistant may act on our behalf since we delegate this control to the assistant. Having this level of access means there needs to be a purposeful effort built into the assistant to protect the users from attacks.
Typically, when users have an account compromised, it is seen as more of an annoyance to the user. They may have to change their password or take other steps, but ultimately, the impact is low for many. With the elevated capability of these assistants, there is an immediate and high impact on the user.
Private
Simply put, a system that doesn’t respect the privacy of its users cannot be trusted. It is almost certain that your hyper-personalized AI assistant won’t be a hyper-personalized private AI assistant. Perverse incentives are at the core of much of the tech people use daily, and data is gold. In fact, it seems the only people who don’t value our data are us.
Your hyper-personalized AI assistant won’t be a hyper-personalized private AI assistant.
Imagine if you had a parrot on your shoulder that knew everything about you, and whenever anyone asked, they just blurted out what they had learned. Now, imagine if that parrot had the same access as you have to all your accounts, data, and activities. This isn’t far off from where we are headed.
Your right not to incriminate yourself won’t extend to your assistant, so it could be that law enforcement interrogates your assistant instead of you. Since your assistant knows so much about you and your activities, it happily coughs up not only what it knows but also what it thinks it knows. Logs, interactions, and conversations could be collected and used against you. Even things that may not be true but are inferred by the system can also be used against you.
Aligned
AI alignment is a massive topic, but we don’t need a deep dive here. What we mean by alignment in hyper-personalized assistants is that they take actions that align with your goals and interests. The your here refers to you, the user, not the company developing the assistant. So many of the applications and tools we use daily aren’t serving our best interests but the interests of the company making them. However, this will have to be the case in the context of personal AI assistants. Too much is at stake.
These tools will take action and make recommendations on your behalf. In a way, they are acting as you. You need to know that actions taken or even nudges imposed upon you are in your best interest and align with your wishes, not any outside entity’s wishes. Given the complete lack of visibility in these systems, this will be hard to determine, even in the best of cases.
Reliable
A system that isn’t reliable isn’t safe to use. It’s almost as simple as that. If the brakes in your car only worked 90% of the time, we would assume they were faulty, even though 90% seems to be a relatively high percentage.
The problem here is that other factors can often mask issues with reliability. For example, if we get bad data and never verify the accuracy, we won’t know that the system is unreliable. Quite often, in our fast-moving, attention-poor environments, we don’t know when our information is unreliable.
Additional Notes on SPAR Attributes
SPAR attributes aren’t simply features that can be attained and assumed to maintain their status in perpetuity. These features must be consistently re-evaluated as the system matures, updates, and adds new functionality. You can see this in Social Media. Back in 2007 and 2008, when I was researching social media platforms, these were mostly issues with the technology. However, if you look at the dangers of social media today, the technology is fairly robust, and we encounter human dangers.
Of course, startups can also be acquired, opening new dangers to people’s information and actions taken. The startup with a strong data privacy or alignment stance can become a big tech company that doesn’t respect your privacy and emphasizes its own goals.
It’s important to realize that none of these categories have been attained to an acceptable level today despite the constant hype surrounding the technology. There is no doubt that today’s technology, with all of its flaws, will be repackaged and marketed as Tomorrow’s Tools.
SPAR Attainment
Once a system has SPAR attainment, which means it properly addresses SPAR attributes, then we can consider the technology to have an acceptably safe baseline. That certainly doesn’t answer our question about whether the technology is safe to use, but what it does do is give us a safe baseline to further evaluate the potential human dangers and impacts.
Conclusion
I hope this post provides a useful starting point for discussing personal AI safety, which is about to become a massively important topic. As AI gets more personal, we must evaluate potential tradeoffs and set boundaries. We can’t do this until the technical safety attributes are accounted for.
To add to the complication, the speed at which these tools are created and the lack of configuration options makes that nearly impossible. Unfortunately, it will remain in this state for quite some time. Still, if organizations address SPAR attributes, it makes it much easier to consider having a safe baseline from which to provide further explorations of safety.
Historically, attackers have targeted large, centralized systems that only represent a small amount of an individual user’s data. This is high value for attackers, but it has a low impact on individual users. This will morph in the coming years. Hyppönen’s Law needs an update in the AI era because in a world of highly personalized AI, if it’s smart, you’re vulnerable.
Hyppönen’s Law needs an update in the AI era because in a world of highly personalized AI, if it’s smart, you’re vulnerable.
