Seems the Hawk Tuah girl isn’t the only one hawking things lately. If I had told you a couple of years ago that a large tech company like Microsoft would be peddling personal companions, you’d think I’d lost it. However, here we are in 2024, and the game is changing. The ultimate question is, are we all getting rugged, just like the Hawk Tuah girl with her cryptocurrency? After all, we were promised that the monumental investment and extensive environmental impacts were worth it because we’d have cured cancer, reduced the cost of goods to zero, and eliminated the need for work, but instead, we got videos of dogs surfboarding and AI lovers that convince us to self-harm. Let the great reframing begin.
Companions, Not Assistants
You’d think tech companies would rather sell earth-shattering innovations that change the world or create massive B2B deals than create AI pals. But that doesn’t seem to be the case. But don’t take my word for it. Here is Microsoft AI CEO Mustafa Suleyman calling them companions and not assistants.
Microsoft AI CEO Mustafa Suleyman says your AI companion will soon be playing video games like Minecraft and Call of Duty with you pic.twitter.com/nwkuzcGZHt
In this video, Suleyman says you’ll play games like Call of Duty with your new, ever-present companion. He says, “You’re gonna be like, It’s my Copilot, of course, I want it to be there.” Like, yeah, duh! The awkward delivery of shoehorning the word “copilot” in this context is the icing on the cake.
Copilot doesn’t doesn’t scream adventure or excitement. The old term wingman at least conjures images of excitement, adventure, and possibly getting into trouble. Copilot sounds mechanical, rigid, and unfeeling. It’s about as exciting as having your eyelids forced open to watch condensation run down the side of a glass.
This focus on gaming isn’t confined to Microsoft. Google’s recent Gemini 2.0 announcement also highlighted gaming.
We’re building AI agents to help you navigate the virtual world of video games. 🎮
With Gemini 2.0’s capabilities, they can watch what’s happening on screen in real time, and offer up suggestions for your next move.
Microsoft isn’t alone on the companion front. Google invested 2.7 billion in Character.AI to rehire key members of the team. Character.AI’s tagline is “Personalized AI for every moment of your day.” Yeah. Cool, but no thanks.
In the near future, tech companies will go hard in the paint on companions over assistants. You may also wonder why these companies apply their massive AI investment to build the ultimate cheat code for video games or an AI buddy. It aligns with their goals.
Exploitation
Tech companies will reframe the pitch from assistants to companions to exploit users. This exploitation will be for two major reasons: stickiness and data. Where products like Friend.com are laughably pathetic, it would be a mistake to assume products from Microsoft or Google would be similarly so. They won’t be some whiny chatbot that needs attention for the sole purpose of companionship. They’ll also have some utility, which will make them appear more rounded.
I won’t get into the human impacts in this article. I have another article where I discuss the human harm from this shift.
Sticky
Every company wants their products to be sticky. The stickiness factor means you work the product into your life and are less likely to switch to a competitor’s product. The hook for AI companions is anthropomorphism, which is our tendency to ascribe human traits to non-human entities. This is because there’s a higher likelihood of anthropomorphizing with an AI companion over an AI assistant.
The goal is to get you to feel a connection or spark with your AI companion that blossoms into something deeper. This doesn’t have to be as deep as falling in love, although some certainly are today. Think of this more as a feeling of warmth. For example, if your AI companion sent you a message telling you to have a great day at work and that made you feel good, that’s where it starts, but the goal is to make it more addictive.
This is why customization and personalization are key. Products will be changed, such as the ability to change their name, voice, and a host of other characteristics. Nobody is going to warm up to something they have to call Copilot. Attach a name, a face, and a voice, and people will imagine a soul.
Attach a name, a face, and a voice, and people will imagine a soul.
Gameplay plays a role in deeper feelings and integration. Playing games with your friends is not only enjoyable, it’s a bonding activity. Bonding with a piece of anthropomorphized technology creates a deeper hook.
Here’s Sam Altman saying he had forgotten how to work when ChatGPT was down.
I don’t believe this for a second, but he REALLY wants this to be true. He’s wrong in reality but right in theory. If ChatGPT were as good as Sam wants it to be and as close to us as he wants it to be, there would be truth to this.
Data
I’m going to let you in on a secret: no matter how much data you give tech companies, they still want more. Okay, so it’s not a secret, but the insatiable desire for more data leads companies to push even harder to place the tech closer to us. Let’s imagine we invented a device called F**k It, Monitor Everything I Do, now known as FIMEID. The device’s sole purpose is to collect data for analysis and exploitation.
Companies would love this device because it is such a rich, concentrated data source. Most of the time, a single tech company doesn’t have all of our data but bits and pieces from various apps and activities, but a FIMEID would create a rich, concentrated data stream.
Despite not understanding the harm from the device, people wouldn’t use a FIMEID because they don’t appear to get anything in return. What’s the difference between a FIMEID and an ever-present AI companion? Functionality, that’s it. However, the AI companion can go far beyond a FIMEID and monitor what you are also thinking. This is because people have a tendency to share thoughts and highly personal information with an AI companion despite knowing it’s AI. An AI companion can also nudge us to take an action or probe us for more information. The proximity of the AI companion means it can also plant data as well. For example, if we are considering buying a new car, the AI companion can manipulate us by increasing the temperature, dropping more hints, and even steering toward a specific purchase. Did we make any purchase because it was something we actually wanted? This question will need to be asked much more in the future.
An AI companion can also nudge us to take an action or probe us for more information.
If something is an ever-present companion, then it’s always capturing data. That means even the fumes of everything you do will be monitored, captured, and exploited. Let’s think about the gaming example for a moment. Gaming creates a lot of data. For gaming, the companion would need wide access to your computer and devices and will collect data about your moves and strategy, and even how many times it had to help you. This data results in a psychological profile. Ultimately, the goal is to lump you into a category where this information can be exploited further, much like your car narcing on you to the insurance company.
This monitoring conjures a vision similar to the human batteries in The Matrix, but instead of generating electricity, it’s a constant flow of data.
We all know what it’s like to have someone nice to your face but always talking shit behind your back. That’s an ever-present AI companion, helpful to your face but talking data shit behind your back. It’s always working in service of a goal that isn’t yours. This is the Alignment aspect I discuss in SPAR.
Now, is it possible to build an AI companion that doesn’t pilfer all of your data and spy on you? Yes, of course. Is it probable? Absolutely not! Despite the impression, tech companies aren’t in the business of just giving you stuff for free. Even when you pay for it, quite often, you are still the product.
Safe To Use
As long as we continue to cling to more academic definitions of AI safety, product safety will lag, particularly among the wider public. Even the most aligned model could still be slapped into a product that isn’t safe to use. Since personal AI tools are products, we must shift our thinking toward safe-to-use criteria encompassing the entire product. My goal in creating SPAR was to define four basic buckets to consider whether a personal AI tool is safe to use. These buckets are Secure, Private, Aligned, and Reliable.
SPAR doesn’t have any formal benchmarking criteria. It’s meant to frame the conversation around the technical categories that make up safe-to-use criteria. As a result, it’s not measurable and only gut-checkable. I’ll revisit this in the future.
Using SPAR as a gut check reveals that today’s AI companion/assistant tools fail in pretty much every bucket of SPAR, making them unsafe to use. As we’ve seen from the previous sections, these tools are not aligned with your best interests. Even if they were made more secure and reliable, there will continue to be privacy and alignment issues by design. Remember, you may still be the product even when you pay for something.
The past few months have witnessed a rash of completely absurd AI predictions. These claims come not from the usual suspects but from the tech leaders’ mouths themselves, lending further legitimacy. However, what people fail to realize is that these are pieces of performance art. Performances enacted not for you but for a singular audience: investors.
AI Performance Art
When tech leaders and personalities make podcast appearances or speak at events, they aren’t talking to you or the audience they are in front of. They are creating performance art for investors. This has always been the case, but not to the extent we’ve seen lately. This effort has been stepped up quite a bit in the past month with some mind-numbing statements.
You can see a small sample of these performances below. Trust me, there are a lot more.
I respect Anthropic and their work, but Amodi’s statements here are nonsense. You read that right, not AGI, but ASI by 2026 or 2027. As a reminder, 2026 is basically a year away. If he believes this (which I doubt), it’s based on vibes, not actual evidence or observations.
He’s just talking Shmidt. This is certainly the dream. However, just because LLMs are “good at code” doesn’t automatically lead to recursive self-improvement. Even if we have promising experiments, they will likely be too unreliable or vulnerable to put into production.
Ah, there he is. That’s right, we’ve been getting 10x improvement every year. You might ask where this has been happening, which would be the correct question.😆
Not to be outdone by Elon, how about 10,000x smarter than a human? I mean, what does that even mean? These numbers are just made up and absurd. These ridiculous exponential increases are something I’ve already made fun of in the past.
Speaking of silly exponential numbers, there was a rumor that someone at OpenAI said Orion, OpenAI’s next model, would be 100x more powerful than GPT-4. If it were, it wouldn’t be called Orion. It would at least be called GPT-5, and people wouldn’t shut up about it. Here’s a prediction. Orion’s performance will disappoint because people’s expectations are far higher than what will be delivered. The expectation is GPT-5, not GPT-4.1.
Genuflect in front of thine server farm, lest thy models collapse!
Someone may have uttered deep learning is divine because it starts with a “D,” but they didn’t mean that literally. Oddly enough, the lack of shame in which he delivers these lines is really something to behold. Although it seems like there’s a mini Altman hype man inside of his head controlling the words coming out of his mouth, in reality, it’s probably because OpenAI is projecting losses of 14 billion dollars in 2026. Ouch! He needs people to believe, to have faith. Preach!
Even when Altman and others talk about the potential of their technology to destroy humanity, it’s a sales pitch. They claim their technology is so good and so powerful it could wipe us all out, so please give us money. This is something I referred to before as the human extinction humble brag.
This is the same behavior we made fun of when the crypto bros did it, but we now take it seriously because it’s AI. Say what you want about the crypto bros. At least putting Dogecoin on the moon is possible. Finding god lurking in gradients is something else entirely.
Oh yeah, there they are. No comment necessary.
None of the previous statements are grounded in any reality. They are all bullshit. And whenever someone is bullshitting, it’s hard to determine if they actually believe their statements or not. The world is far more complex than we give it credit for, and it’s also true that sometimes, an unexpected innovation comes along and changes everything. This is what they all hope for. That some innovation clicks before the clock runs out on investment. Or divine intervention in Altman’s case.
The sad part is that almost everyone will forget these silly predictions. No doubt many have forgotten about them already. There is never any accountability and yet people continue to hang on their every word. The problem is there is no one place where these predictions are collected and presented like the bullshit Picaso it is. If there is, please let me know.
Why Now?
The increase in hype-laden statements is because, until recently, AI hype had been mostly self-fueling. But 2024 has brought unwanted criticism to the generative AI space. I noticed this starting to take a turn in July when Goldman Sachs released their report: GenAI Too Much Spend Too Little Benefit.
After this report was released, the media began to report more critical assessments of generative AI. These critical assessments spelled out that the generative AI craze might be a bubble. But that’s not the worst of it.
If you’ve watched any of my conference presentations this year, you’ve probably heard me talk about the performance plateau in large language models. Saying that, if you are hoping for much more capable models to solve your problems, they aren’t coming any time soon. This plateau was obvious when looking at the data but was never acknowledged, but people are noticing it now. This doesn’t mean LLMs are useless, people are using them for a variety of tasks today. What it means is that if you require greater capability and reliability, you may be waiting a while.
Now, news reports like this from Bloomberg cover diminishing returns, and other articles talk about a shift in strategy toward other mechanisms to address the slowdown. Of course, none of this is represented by the leaders in their wild predictions.
Combine this plateauing with the fact that model training appears to be the fastest-depreciating asset in history, and the picture doesn’t look good.
When you look at the financials, why train new foundation models yearly when the benefit is so low? Maybe as a marketing exercise or other activity unrelated to model improvement, but the costs don’t seem to align. As I mentioned earlier, OpenAI is projecting losses of 14 billion dollars in 2026. This hemorrhaging of money is non-sustainable.
But all of this is rather Orwellian. We are told to reject the evidence of our eyes.
No, AGI Isn’t Imminent
Here’s a graphic from Reddit charting the prediction of when we’ll achieve AGI. Demis Hassabis is the one on the list I’d take most seriously. Deep Mind is a serious AI lab doing serious work and not putting all their eggs in one big LLM basket. I still think these are mostly guesses with some hopes mixed in. The reason Kurzweil is close to Hinton and Hassabis is because he went The Price Is Right route and chose his number based on the fact that it was one less than 2030.
However, tech leaders know that predictions like these trigger influencers. Influencers are the hype agents trying to get people stoked. When people are stoked, investors take notice. So many social media feeds of so many supposedly serious people are turning out to be pretty embarrassing and will be even more so in a year or two. If anyone had any attention span left, that would be worrisome.
Quite a lot of truth is found in this simple statement from Pedro Domingos. Many assume that because things like LLMs have so much information, they must be close to AGI. But instinctively, we know that access to information isn’t knowledge. Otherwise, everyone with a web search would be a genius. Then again, Pedro’s comment aligns with my biases, so I guess I have to be careful.
Hype Has Consequences
You might ask, why do I care about any of this? Well, it’s because hype has consequences. The inevitable outcome of all this hype is that technology gets shoved down our throats. Generative AI is easy to manipulate and potentially unreliable, a cocktail for disaster in high-risk applications. The danger is that we rush something that appears to be working into production and hope for the best. Over the next couple of years, we’ll see the push to cram generative AI further into the systems and processes we use on a daily basis, including high-risk and safety-critical systems.
This push won’t be based on generative AI being the best tool for the job but on a push for monetization. Tech companies need to show some return on the monumentally massive investment they’ve made, so this push becomes another form of performance art for investors. Tech companies are throwing a plate of spaghetti at the wall and hoping that a noodle sticks.
Why do you think there is an increased coziness with the US government? They don’t see an ability to make a difference. They see dollar signs. Things like DOGE and Sam Altman co-chairing the new mayor of San Francisco’s transition team are like asking drug dealers for guidance on prescribing drugs. Despite this, I truly hope DOGE succeeds because if it fails, it will be bad for a lot of people, so my fingers are crossed.
Government streamlining and modernization are noble goals, and I think AI and automation certainly play a role, but it’s about choosing what’s best for the people these systems serve. In this scenario, you are optimizing for different things that may not be intuitive in a traditional business sense. These are real systems affecting real people, not toy examples in the lab.
I joked that this could lead to some strange Kafkaesque nightmare in which people are stuck in a loop, unable to get a resolution. Or, you have an algorithm that works so well at saving money by denying people benefits. This is easy to shrug off if you don’t require government assistance, but it’s an entirely different story for people who rely on it or when a disaster strikes. These updated systems and reduced staff scenarios may appear to work and deliver promises in the immediate implementation but fail spectacularly when they are needed most. We caught a glimpse of this with the Healthcare.gov launch, and that was just a website.
But, China Tho
Typically, you get the But China Tho argument when there’s any pushback. This argument states we must remove all the brakes and accelerate into oblivion because of the risk of China getting to AGI first. Damn the harm, full speed ahead.
However, if we could squeeze some extra performance out of a car by removing the steering wheel, we still wouldn’t do it because we understand something simple. A car’s performance isn’t solely based on acceleration, and neither is AI. Acceleration is bad if the vehicle is speeding in the wrong direction.
Recently, the U.S.-China Economic and Security Review Commission put out a report that recommended creating a Manhattan Project-like program dedicated to racing to and acquiring an AGI capability. In this section of the report is this:
Provide broad multiyear contracting authority to the executive branch and associated funding for leading artificial intelligence, cloud, and data center companies and others to advance the stated policy at a pace and scale consistent with the goal of U.S. AGI leadership.
There’s a predictable outcome here if something like this moves forward. Agendas and ulterior motives will co-opt this project, not setting the United States up for success. There’s a current tunnel vision with LLMs that has people deep in the sunken cost fallacy.
The United States’ strongest assets are its tech companies. Despite my criticism of their hype and lack of respect for privacy, they are vital to the success of the US economy. I’m also highly critical of the sentiment some have adopted to “break up the tech companies.” I’m not a tech critic, I’m a hype critic. However, setting up a massive pot of money that they can draw from, like an ATM, is not something I’m in favor of either.
Here’s something else to think about. What if, by maintaining a relentless hyper-focus on LLMs, China (or another country) gets to AGI first by focusing on other approaches? This is a real risk.
What if, by maintaining a relentless hyper-focus on LLMs, China (or another country) gets to AGI first by focusing on other approaches?
I may have to eat my words at some point if AGI does sprout from LLMs. It’s certainly not impossible. However, if we cobble together something that resembles AGI from generative AI, it will most likely be AGI based on toothpicks and bubblegum. What I mean is a whole lot of patches, layers, plugging, and human intervention.
My AGI Prediction
Okay, so now it comes to me. What’s my AGI timeline prediction? Well, I predict we’ll have AGI by—
Of course, I’m not going to answer that. I’d guess based on no evidence, just like many others I’ve highlighted. I have no particular insight, and I’m not working at a research lab trying to build AGI. Despite this, I have some thoughts related to my area of expertise.
The last slide of my keynote at Agile DevOps USA in October mentioned AGI. Discussing this slide, I made a few statements about how I didn’t think that AGI would be built from LLMs and that it probably wouldn’t come by 2026 or possibly even 2029. So, I guess that’s as close to a timeline prediction as you’ll get from me on AGI—not when I think it will happen, but when I think it won’t happen. I’m certainly not an AGI skeptic, it’s possible and will happen.
More importantly, I predicted that no matter what form AGI takes, it will be vulnerable to attack and manipulation. I mentioned that this would especially be true if it were built on top of LLMs (remember, toothpicks and bubblegum.) Maybe something about generalizing across many tasks in the real world makes things vulnerable. This is something I mentioned back in February of 2023.
To make matters worse, we may be stuck with the vulnerabilities that get identified because there is no fix. Think of examples like adversarial policy attacks. We’ve all heard of AlphaGo beating Lee Sedol at Go. However, most don’t know that even average Go players can beat superhuman Go AIs using adversarial policy attacks. Yes, the stakes are low in the game of Go. However, this is a cautionary tale.
We may be stuck with the vulnerabilities that get identified because there is no fix.
Combine these potential issues with the fact that humans don’t do a good job of finding vulnerabilities in a system before it is launched into production, and we have a recipe for lingering problems. When these lingering problems are in high-risk systems, disasters are only a couple of steps away, and there’s not much we can do about it.
One constant throughout the generative AI craze is summarization. Why read a book, listen to a podcast, or YouTube video… Just summarize it! Large swaths of content, distilled into several bullet points with countless hours saved. However, this isn’t the utopia many claim.
We all love a good shortcut. Humans are wired for them. This is why we are so good at cognitive offloading, but the tradeoffs from shortcuts are never recognized or shoved deep into our subconscious. Every shortcut has tradeoffs. With generative AI, tradeoffs are never acknowledged or discussed. However, here’s an inconvenient truth: knowledge and understanding aren’t generated from bullet points.
Fake Optimization
Many of the claims made by influencers, transhumanists, and the e/acc community revolve around fake optimization. Fake optimization claims that something lowers friction for a task or activity while not providing the same value.
So many things in this world require friction for success, especially knowledge and understanding.
These people see everything as a game of lowering friction, but there’s just one problem. So many things require friction for success, especially knowledge and understanding. To go further, there are many activities where the friction of the activity is the point, such as art or meditation. However, telling people that won’t get clicks and someone’s “thought leader” badge may be revoked. So we end up with the environment we have today, with everyone from tech leaders to influencers telling people friction is about to be a thing of the past.
Take this example of promising people they don’t have to put in the work and still gain the benefit. Anyone claiming you can gain the same value from cramming three hours into three minutes demonstrates a fundamental lack of understanding of how knowledge transfer works and a near-religious level of faith in AI.
If we step back, people listen to content like podcasts for two different reasons: entertainment and information. Quite often, it’s a combination of both. So, by summarizing, we’ve removed all of the entertainment factor, immediately reducing the value of an activity. However, before we get too far, let’s examine a scenario that should be obvious to people.
Imagine summarizing a one-hour stand-up comedy performance. “Just tell me the best jokes.” Is that really an hour saved? Of course not. It won’t be funny, and anyone who thinks differently has been sitting behind a computer screen for too long. We instinctively know that comedy is situational and relies on context and delivery. Comedians like Mitch Hedberg prove this point.
The comedy scenario is obvious for most to understand. However, what’s difficult to understand is that a similar value loss also exists for non-entertainment activities. Summarization isn’t the shortcut people think it is. Without the surrounding context, we may not be committing these summaries to memory, where we can take action on them or put them to use.
Thinking Deeply
There’s no thinking deeply about bullet points or summaries. You can’t. This is because the action of summarizing strips away all of the context. For thinking deeply, the context is key. Summaries are just a set of condensed words shoved into a predetermined space. Important bits of information (sometimes the most important bits) are left out. There’s no way they can’t be.
There’s no connection to bullet points and summaries, no deeper meaning, emotion, or content to chew on mentally. Nobody contemplates something deeper or dreams about something bigger with summaries. The same can’t be said about reading a book or other longer-form content. The inherent dehumanization of summaries drives some of this lack of connection.
In summarization tasks like these, we take someone’s uniqueness, including their perspective, delivery, language, and flair, and crush the life out of it to get the resulting bullet points. This act results in a shift. Instead of viewing someone as a person, we view them as data or a product to be manipulated, and summaries strip humanity away, leaving us with several cold sentences generated from the compactor of a black box.
Make no mistake, the dehumanization aspect is a selling point for many. The human aspect is often seen as flawed, whereas the AI aspect appears superior. But this perspective doesn’t serve us well—you know… we humans—especially when it affects our ability to think deeply.
There can be rare exceptions where a quote or simple statement does cause some deep thought. For example, this quote is often attributed to Einstein, even though he never precisely said these words.
“If you can't explain it simply, you don't understand it well enough.”
A statement like this can trigger deeper thoughts about ourselves and our view of knowledge. As a theoretical, let’s pretend Einstein was on a podcast and uttered this statement, making a larger point about knowledge and understanding. Mediated through an AI system in a summarization task, this statement could be transformed into:
“You need to explain things simply.”
The difference between these two examples is stark, and they do not even remotely mean the same thing. There’s certainly nothing to think more deeply about in the second example.
The ability to think deeply about any topic is a skill we are losing fast and for younger generations, possibly never cultivating in the first place. Our modern world, filled with its distractions, is not only pulverizing our ability to ponder, to wonder, and to dream, but also to question.
The act of questioning requires effort and friction. It isn’t purely asking a question to an AI system and getting a response because the act of questioning isn’t easily satisfied. Don’t let people reframe them as equal. We will not be better off for it.
Context, Value, and Illusion
In reality, longer-form content can be bloated. I’ve read books that should have been four chapters and podcasts that could have been reduced to thirty minutes. However, it’s a mistake to consider context as bloat and an even bigger mistake to assume an LLM knows the difference. This is because you often can’t tell the difference until after the fact. Something that seems like bloat at the beginning is context in the end. That pointless story turns into a connection reinforcing a particular point.
It’s a mistake to consider context as bloat and an even bigger mistake to assume an AI knows the difference.
Let’s consider the importance of context for a moment. Consider something larger, such as a slide deck from a presentation. There are not just several but many bullet points along with images and diagrams. If you are already an expert on the topic, it may be possible (but not always) to glean something from the slide deck. However, the real value is the context in which the content was delivered and the commentary around it. Conversely, if you watched the presentation and had the context, the slides are helpful because they can reinforce the content and even jog your memory. This is true for all sorts of content.
You may be convinced (or not) by a set of built points or summaries, whereas hearing the whole argument would have proved otherwise. In life, we say it’s all about context, but context is what we discard when we summarize.
Also, even for general accuracy, the act of summarization strips away all of the supporting or disproving elements, leaving us with a couple of sentences that may or may not be important. Without the context, how do you know if a point is accurate? You have to blindly trust the system.
One of the most commonly encountered bits of summarization is survey results. Most people never dig into the details of surveys or studies, but this is where you find issues. These are problems with the approach, sample size, sample diversity, and many more pieces of context that may cast a shadow over the results, transforming those groundbreaking results into more questions than answers. Summarizing everything leads to many misunderstandings.
We spend little time evaluating the proposed value from summarization. We are told we can spend far less time yet gain a commensurate level of insight from summaries. This value proposition speaks to our modern low-attention-span world, but if we take a step back and consider the realities, it just doesn’t jibe for the reasons outlined in this article.
Much of this disconnection stems from a lack of presence. We need to exercise a certain amount of presence to read a book or join a meeting. However, this is becoming a lost skill. New technology promises we no longer need to be fully present again, but there are consequences in nearly all contexts. This is why the Illusion of Presence is one of my cognitive illusions created by personal AI personas.
Unfortunately, we do end up fooling ourselves. Using an AI to summarize content for knowledge gives us the illusion that we are working smarter and creating more knowledge with less effort, but as we’ve seen, that’s not the case. The reality is a world of summaries creates a world of fools.
A world of summaries creates a world of fools.
Although harsh, if we consider what we’ve already discussed, it makes sense. Not only are we not gaining the promised value from activities, but we also fool ourselves into believing we do.
AI Mediation
AI mediation is both a bug and a feature. What we want out of content may very well be in the dense center of some data blob. However, something must be said about getting all of our information mediated through an AI system. So much of our world is already mediated by algorithms, and we aren’t exactly better off for it. We are pushed and nudged in various directions, making us more predictable, with all of us shoved toward the dense center of a distribution. But what you don’t find there are uniqueness, creativity, or innovation. Sparks, inspiration, and innovation don’t come from bullet points, although you are certainly being sold on the opinion that it can.
Ultimately, we leave it up to an algorithm to determine the main points, the most important things we should pay attention to. A black box plucking data points with some higher purpose that nobody understands. Many times, the points being distilled may very well be the most important, but certainly not always, and without context, it’s impossible to tell.
Ultimately, we need to ask ourselves a question. How many filters do we want between us and reality? Using AI for mediation is yet another filter on top of reality. We should work to remove filters in places where the activities are important to us.
I’m not trying to overplay the dangers here. You certainly won’t be hurt by occasional summarization tasks with an AI system. However, when used often, there is not only a value mismatch, but it can also warp our understanding of reality. So, there are consequences.
Wasted Time, Not Optimization
The funny thing is we don’t even ask ourselves if the time spent is worth it. Let’s say we cut down on reading time to generate summaries instead. This way, we can cover more ground on more topics. Many may consider this a solid strategy. Subconsciously, this also feels right, which makes it a powerful argument. This is why influencers are so fooled by it. However, when we dig deeper, it’s not the benefit it seems.
So, in the three hours to three minutes optimization sale, you lose time. The three minutes are wasted because you never had the content reinforced with the surrounding context. It becomes bullet points scrawled across a mental billboard as you drive past at 120 mph. Of course, this assumes that the content distilled wasn’t so generic to be a waste in the first place.
Say, for instance, that we use AI to summarize Peter Attia’s book Outlive or possibly one of his podcast appearances. One of the summary bullets may be:
Put a larger emphasis on Zone 2 training.
Okay, but why? What is Zone 2 training? How do I do that? Answers to these questions were covered in the surrounding context, but now you spend extra time tracking down the answers.
Multiple people have already joked that we are on the cusp of someone writing something based on bullet points only to have the other person convert it back to bullet points. There’s something rather dystopian about this.
If something is worth learning, then it’s worth spending time on. This was true in the past and will be true in the future.
Conclusion
There are no shortcuts to creating knowledge. Knowledge generation always takes friction, but through this friction comes reward. When we take shortcuts, we deprive ourselves of the reward, leaving us with a hollow task that doesn’t provide the same value. Ultimately, nobody gets smart from bullet points.
I’m not claiming all summarization tasks are bad. They may be helpful and fine for task-based systems and under certain conditions. But they are not for generating knowledge and understanding. It’s becoming increasingly obvious that we must defend our cognitive functions because nobody else will.
As a kid, I had some rather eclectic reading habits. One of the books I read was Ki in Daily Life by Koichi Tohei. I read it in a quest to unify my mind and body. I was a kid. I had no idea what that meant. At the time, I was fascinated by how the human mind could be unlocked and the potential of connecting with the universe through focus and daily practice. Something I still struggle to conquer as an adult. I’m not attempting to embellish my level of childhood insight. I was watching a lot of Jean-Claude Van Damme movies, practicing my splits and high kicks as well.
What does any of this have to do with technology? The world of the present seems poised to shift from a focus on the mind to a focus outside the mind. As the technology powering tools like ChatGPT and Claude morph into more connected personal AI tools, these tools will take on multiple personas. So, what does AI look like in daily life? What personas will AI play? Before we get to that, let’s first examine overreliance.
AI Overreliance
Whenever the risk of overreliance is discussed, it’s typically framed in the context of automation bias, the human tendency to prefer the output of automated systems. Humans using these systems may not question their output, leading to poor decisions, cascading failures, and the amplification of biases. These issues are often discussed in purely technical terms, describing how technical issues can manifest or how the output of a system can harm other people. These are all serious problems, but what often isn’t discussed is what happens to our cognitive abilities when we over-rely and overuse AI.
This sort of daily overreliance leaves a gaping hole you could drive a truck through because as our capabilities diminish, we are less likely to spot errors and keep the system in check.
Daily Overreliance
Here is a recent article from Microsoft that covers the topic of overreliance. I have some quibbles with this article, but it makes for a good demonstration since it explicitly calls out four basic shapes that overreliance takes:
Naive overreliance
Rushed overreliance
Forced overreliance
Motivated overreliance
This breakdown is instructive, and thinking about the topic in this way is beneficial. However, I’d argue that this still primarily focuses on technical aspects and is missing a key category: Daily Overreliance.
Daily overreliance occurs when we use an AI tool in our daily lives or even repeatedly for the same task. Usage can extend to both work and personal tasks and will soon encompass both, with the uptick in assistants becoming personal AI tools.
The more integrated AI is in our daily lives, the more we will use these tools for activities that we may not consider using them for today. These include who to be friends with, maximizing happiness (whatever that means), planning, communication, and a whole host of other activities.
Daily overreliance not only leads to the same technical issues covered in other articles but also to cognitive atrophy and a lack of skill development. This overreliance also fuels cognitive illusions, which we’ll cover in the future.
Overreliance Is The Goal
Make no mistake, the risk of overreliance is also the goal of many tech companies developing the technology. Nobody is investing massive amounts of money in AI companies for simple productivity tools or a 20% boost in human efficiency. So, it’s fascinating to observe overreliance being called out as a risk while simultaneously being the goal. What a time to be alive.
AI Is Competing With Us
We compete with AI, even as we use the tools for ourselves. I’ve covered cognitive offloading before and described how we transition from knowing things to knowing where things are stored. In that article, I also mentioned complimentary and competitive cognitive artifacts. AI is a universally competitive cognitive artifact.
When we use AI, we feel like we are bending a powerful tool to our will, much like a wizard conjures spells with a magic wand to make things happen. We imagine the prompt uttered is the spell, and the AI tool is the wand. There are parallels in this hypothetical example.
The wizard doesn’t know how the wand works, and if the wand is unavailable, they cannot complete their tasks. Imagine a scenario in which the wand does everything for the wizard. How does the wizard keep the wand in check if they’ve lost their skills or never developed them in the first place?
When children use AI for daily tasks, they may never develop the cognitive skills necessary to think deeply, focus, or reflect, compounding the damage from mobile devices and social media. This is why the rush to shove generative AI into the classroom can have devastating consequences if not thought out or implemented with an actual plan and measurable goals.
Thinking of AI as a competitor instead of a collaborator spawns a different mindset.
AI competes with us, even as we use it for our own tasks. Thinking of AI as a competitor instead of a collaborator spawns a different mindset. A competitor may give you bad information. A competitor may want to take something from you. Thinking adversarially brings a bit of skepticism and allows us to erect guardrails around activities we’d like to protect and outputs we may need to check. This is the best of both worlds, allowing us to consider using AI selectively instead of indiscriminately. So, ponder this the next time one of the just use AI for everything people starts running their mouths.
Personal AI Personas
When considering using personal AI tools in daily life, we can envision the manifestation of several personas. These personas will play various roles in daily life, crossing personal and professional boundaries. These personas supercharge overreliance risks by outsourcing cognitive functions to these tools, making us even more dependent on the technology and fueling even more use.
I’ve broken this outsourcing into the following six personas representing roles that personal AI tools will assume in daily life.
The Oracle
The Recorder
The Planner
The Creator
The Communicator
The Companion
Each role represents an outlet for cognitive offloading and contributes to potential cognitive illusions and cognitive atrophy. The most obvious is the illusion of knowledge, but the list of cognitive illusions is a conversation for another day.
In a way, we are outsourcing authority as well, allowing these systems control over our daily lives, perceptions of the world, and even our actions. The more we outsource to personal AI systems, the less we will be able to keep them in check. There are no firewalls around these personas or around the tasks and task types we feed to personal AI systems. This blending of tasks and personas leads to quite a few downsides.
Note: Although I won’t be diving into the harms when I imply there are negative impacts from allowing AI to play these personas, these impacts manifest from repeated and even overuse of the technology for the role or activity. Being selective about use and application minimizes impacts and should be the goal, allowing us maximum benefit while minimizing negative impacts. Also, I’m merely introducing the persona with a brief description, not diving deeply into each in an attempt to keep the word count of this post in check. I may expand on these later.
The Oracle
The Oracle persona manifests when people use AI tools as an all-knowing question-and-answer system. Since, deceptively, the system appears to have representative knowledge of humanity, users are happy to type questions and receive answers, closing the loop on curiosity. However, it’s important to note that the questions asked to an AI-based oracle run far deeper than retrieving facts you’ve forgotten, such as retrieving the year the song Under Pressure was released.
Take, for example, questions about who you should marry or even who you should be friends with. Answers to such deep questions should come from exploration, not a Q&A system. Of course, these questions won’t be asked in such a straightforward way. They may be combined with The Planner persona to achieve a goal, such as maximizing life happiness or trying to optimize your career. Through these activities, we dehumanize people, turning them into objects to be manipulated rather than other human beings living their own lives with their own thoughts and emotions.
These systems appear to know more and know better than us, so we will inevitably overuse these systems for all sorts of decisions in our daily lives, receiving more answers and questioning even less.
The Recorder
One of our most obvious cognitive limitations is our brain’s capability for recall. There is only so much we can remember and surface when needed. This limitation is why we set calendar appointments or scribble reminders on sticky notes. Even when we make a purposeful effort to remember things, we can still forget if too much information is given or there is too much time between needing to recall the information.
With personal AI systems, even less cognitive effort will be expended for remembering things. We will count on these systems to remember things on our behalf. Agents running on systems will record and transcribe whatever we choose. Meetings, emails, YouTube videos, podcasts, personal conversations, and everything in between are all recorded and available whenever we want to review them. Even if we don’t want to review them, insights will be distilled for us automatically. There will be no reason to be fully present ever again.
The recorder role not only records but, combined with The Oracle persona, also makes sense of the content for us. It may seem like optimization when our personal AI tool spits out a single action item from a one-hour meeting we missed or weren’t paying attention to, but our lack of presence has negative impacts.
We didn’t have a seat at the table and couldn’t influence the direction or demonstrate our value to the project, conversation, or leadership. We weren’t able to build bridges or foster connections with others. We may also get the wrong idea and context from the meeting. Sure, maybe the full transcript is available, but if we feel these tools are created to optimize our time, why would you go back and read the transcript or play the full meeting recording? This is a surefire recipe for miscommunications and other issues.
The negative impacts run deep. The less we use our memory, the worse it gets. Socrates was right.
The Planner
We’ll use The Planner persona when we want to set a goal for the system to accomplish on our behalf. The system will use its capabilities and connections to perform all of the planning and tasks necessary to accomplish the goal, setting all of the activities in motion, with our brains doing none of the work.
Humans plan and execute every day without even realizing it. Much of this planning and execution is done subconsciously. For example, if we wanted a bowl of cereal but realized we had no milk, we may formulate a plan to rectify the situation. This plan may include putting on our shoes, grabbing our keys, driving to the store, purchasing the milk, and returning home. We don’t document this plan or map out a strategy, but it is formulated subconsciously in our prefrontal cortex and executed without much thought. But planning isn’t just for simple things like getting milk or considering what to wear for the day. So much of our daily lives contain planning and strategy.
Regarding personal AI, we assign authority to these systems due to our perception of their capabilities, but these can also be illusions. AI contributes to the illusion of knowing more and better than humans. This assumption isn’t new and even has its own bias, automation bias, which was mentioned earlier. Automation bias is the tendency of people to prefer the output of automated systems, even when contradictory information is present. We tend to know that humans are flawed, biased, and prone to mistakes, so we trust the output of these automated systems more than our judgments or the judgments of others.
Extending the Oracle persona, we will use these systems for feedback and direction on all sorts of work-related and personal tasks. We will treat these systems as the authority, assuming they know best, and allow them to make critical and benign decisions on our behalf. This will extend far beyond the typical scenarios people associate with automation bias.
With the advent of personal AI, we will count upon these tools to plan just about everything, plotting a course to goals and mindlessly nudging us in various directions. Although this may seem like a sound thing to do, many will use these tools to plan all sorts of things that we don’t use tools to plan today. For example, we may want a personal AI tool to plan a night out with a significant other or maybe to optimize finding a significant other in the first place.
The Creator
Using AI tools to create things is a common task today. Many use these tools to generate images and write creative content. The creator persona is about much more than just creating images. It’s for when the tool does the work of creation across various use cases, including writing, coding, games, and many others.
To focus on creativity for a moment, anyone who’s ever truly been creative knows that surprise is an important part of creativity. In the book I, Human, Tomas Chamorro-Premuzic says:
Surprise is a fundamental feature of creativity. If you are not acting in unexpected or unpredictable ways, then you are probably not creative.
I think this is true, but I’d also take that a bit further. Many may claim they are surprised at the output of a generative AI tool and that this is the same thing, but it’s not. Being surprised isn’t the same as surprising yourself. Surprising yourself is the primary satisfaction that results from creative endeavors. It can be hard to understand the difference if you’ve never surprised yourself or noticed surprising yourself, but that doesn’t mean there isn’t one.
Being surprised isn’t the same as surprising yourself.
Ultimately, the creator persona deprives us of creative satisfaction and creates the illusion of creativity. I’ll expand my thoughts on this in the future.
The Communicator
The communicator persona is when we outsource communication between humans to AI tools. We can think of this as something as simple as using AI to construct an email or something more complex like creating a bot with our voice to talk with our parents so we don’t have to. It may seem like there aren’t any downsides to the communicator persona, but there are impacts when we outsource these interactions to AI. I’ve written about this previously in how we are optimizing away human interactions with AI.
As communication has moved online and become more asynchronous, we’ve lost touch with some of the subtler aspects of human communication. This has led to us feeling that communication is more of a burden. With today’s online business and distributed workforce, communication with other humans has become viewed as a task or a checklist.
This is why one of the touted examples of these AI systems is handling email in our inbox, automatically prioritizing messages, and responding on our behalf. Therefore, the human aspect of this communication is removed, and the task portion is checked off. But even in the boring world of business communication, the human aspect is still important.
When we outsource communications to automation, we miss opportunities to build relationships and make our voices and opinions heard in critical contexts. This leads to a lack of trust and importance. Suppose it came time for a workforce reduction. Would we let go of a resource that provided valuable feedback and engaged in communication or the one that outsourced responses to a bot and couldn’t be bothered with responding back to us?
More importantly, we miss opportunities to connect with our fellow humans and build relationships with them, opting to treat others as tasks or objects that need to be manipulated. When we let our communication skills atrophy, a whole host of uniquely human qualities disappear, transforming us into machines.
The Companion
The Companion persona is when the AI tool acts as a friend or romantic partner. The Companion persona isn’t part of the future state of technology. It’s happening today. Startups like Friend, Character.ai, Replika, and many others are pushing this use, sometimes with devastating consequences. These companies are even marketed with straight-up bullshit.
That’s right, a soul. Our chatbot has a soul and a deep connection to us, yet it doesn’t care whether we live or die. I’ve written about this nonsense previously, so I won’t go deeper into it here.
As personal AI tools become more part of our daily lives, more people will begin to feel a connection with them, mistaking the interactions for meaning. This will fuel the illusion of companionship and lead to more devastating consequences for our mental health and humanity.
Cognitive Illusions
Cognitive illusions manifest from the overuse of these tools in the mentioned personas. These illusions cause a wide range of negative impacts on our health and wellness, as well as our cognitive abilities.
I won’t cover the illusions created by these personas in-depth, but here are some highlights.
Illusion of Knowledge
Illusion of Capability
Illusion of Memory
Illusion of Agency/Control
Illusion of Presence
Illusion of Creativity
Illusion of Certainty
Illusion of Companionship
Conclusion
In the next few years, these tools will be pushed closer and closer to us in a quest for profitability. All of the known flaws with this technology will not be fixed, but even if they were, that wouldn’t be the extent of the harm. This is why I created SPAR to frame the conversation around personal AI safety.
However, this article covers harms that extend beyond the technical issues and make the harms personal. We must be selective in using these systems and draw firewalls around tasks and activities we want to protect, an increasingly difficult task in a world where we prefer the easy button.
Even though the 2024 US election is less than two weeks away, we are still being bombarded with news reports and experts blaring on about the dangers of deepfakes for this election. When I was recently traveling abroad, I witnessed the BBC host self-proclaimed “experts” from the US predicting the catastrophic impacts deepfakes would play on the 2024 US election, and yet, nothing. It seems deepfakes are having a bad time in the information manipulation arena.
There also seems to be massive confusion between deepfakes, slop, and memes. Reporting on the topic lumps all of these into the same category just because AI was used to generate them, but this misses the point entirely because they are not the same thing and have different goals.
Since this will likely be the last piece I write about deepfakes before the 2024 US election, let’s break some of this down. I’ve covered this territory multiple times since 2020, and everything I’ve written, including my predictions, has held up. However, this article isn’t a victory lap. These previous posts remain valid, but we have some new terms now. Let’s take them piece by piece.
My Position
My position has always been that deepfakes won’t affect the outcome of elections or change a person’s mind on polarizing topics. This should be relatively obvious because the truth doesn’t change people’s minds in similar situations. Deepfakes do fool people when there are no stakes. When people don’t care about the topic, think Pope in a puffer jacket or Katy Perry’s dress at the Met Gala.
For more information on my position, I’ve written multiple pieces going back to 2020. I won’t retread the same ground in this post.
We’ve had multiple major elections across the globe, and deepfakes have played no role in the outcome. The 2024 US election has been dubbed the “deepfake election,” and yet it’s been a big nothing burger. Despite all of this, the threat of deepfakes related to elections remains wildly overhyped, ultimately because of perverse reasons. Fear gets clicks.
For some recent evidence proving this point, look at the report OpenAI released just this month on Influence and Cyber operations. None of the operations they tracked had any meaningful engagement. Although it’s true that OpenAI isn’t the only game in town, this report does seem to track with other observations.
Deepfakes, Slop, and Meme Confusion
Let’s start by defining the difference between deepfakes, slop, and memes. People can be forgiven for this confusion since it’s all generated by AI, and the definitions of these terms can be confusing. For example, let’s start with deepfakes.
an image or recording that has been convincingly altered and manipulated to misrepresent someone as doing or saying something that was not actually done or said
Well, that’s not very helpful. Wikipedia’s definition is a bit better, but it’s still not good.
Deepfakes (a portmanteau of 'deep learning' and 'fake') are images, videos, or audio which are edited or generated using artificial intelligence tools, and which may depict real or non-existent people. They are a type of synthetic media.
Although technically accurate, any definition of deepfake can’t be considered without the intention. The mental picture of deepfakes in the public eye is that they are generated to support a larger narrative or trick someone into believing something. So, they either provide the evidence for a narrative or are used in a social engineering attack. Back in 2020, I referred to these as Narrative Evidence attacks. Simply put, people consider deepfakes as evidence used to convince people that something happened that didn’t.
Let’s look at a simple example. I may have a narrative that you stole a car, but I have no evidence to convince people that you stole it. To support my narrative, I create a deepfake video of you stealing the car as the “proof” supporting my narrative. This usage makes the scenario a deepfake and not purely slop or a meme.
AI Slop
AI slop has filled the internet. It’s unavoidable, slathered across every corner of the web, and is one of the lasting legacies of generative AI. If you’ve ever heard of things like Shrimp Jesus, that’s what we are talking about. It’s a sort of low-quality AI-generated content that fills a digital placeholder.
Here is a great comment from Twitter user @nearcyan summing up YouTube shorts with AI Slop.
There is no confusion about the truthfulness of slop. Nobody thinks The Rock is part of any of these videos on YouTube. Its purpose is merely as content to fill a placeholder and try to get clicks. In the context of elections, people may leverage slop with a twist of propaganda, using generated counterfactuals to elicit an emotional response.
Take this image, for instance.
After Hurricane Helene, a bunch of AI-generated images flooded social media. People shared these images to try and get an emotional response. This is more like propaganda. There’s no doubt that some people may have believed the image was real, but a vast majority of people who shared, shared it because it aligned with their biases and supported their message. The image was merely an emotional placeholder. Oddly enough, they would have shared it regardless of the image.
Memes
Memes need no introduction and far predate the generative AI era. However, Generative AI has generated memes of its own. Take, for instance, the videos of Will Smith eating spaghetti. The poor quality of the video made it a meme of its own. So when the AI-generated video of Will Smith eating spaghetti with Donald Trump surfaced, everyone should have gotten the joke.
Just like AI slop, with memes, there is no confusion about the truthfulness of the content.
In the context of elections, deepfakes are AI-generated content meant to fool people into believing something happened, while AI slop and memes are more akin to propaganda.
Spot The Deepfake is Pointless
By now, you’ve undoubtedly run across many of the spot-the-deepfake challenges where you cycle through a series of images or videos and try to determine which is real and which is AI-generated. Other than creating some very basic awareness of the capability of the technology, these exercises are pointless.
Here is a recent example I saw posted online of someone telling people what to look for. It’s the same kind of inane advice parroted repeatedly that won’t hold up.
The reality is that asking which image is real or fake gives an irrelevant answer to the wrong question. These spot-the-deepfake challenges are misleading because it doesn’t matter which one is real or fake. Also, training people to “spot” characteristics in images and videos that are disappearing or changing rapidly doesn’t set them up for future success.
Deepfakes Aren’t In Isolation
Deepfakes and other fake content meant to fool people aren’t encountered in isolation. They are provided as evidence to support a larger narrative. This means you’ll never just have the fake content with which to make your decision.
Consider a doctor making a diagnosis and prescribing a treatment. It would be rare for a doctor to look at an image, make a diagnosis, and prescribe a treatment. The doctor will use additional context in diagnosis and treatment. First, they may order additional tests for verification. Also, they consider other contextual information such as medical history, family history, allergies, and a slew of other information before moving forward.
The real question has little to do with the deepfake itself. What we are evaluating is the message. So, given the source and surrounding context, can the overall message be believed? With misinformation and disinformation, Deepfakes aren’t the message; they are the proof.
Below are questions you can ask to mentally focus on the message. This is not an all-encompassing list of questions, but it can get you thinking.
Who is sharing?
Are they credible?
What’s their motive?
What have they said in the past?
Are there conflicting or contradictory accounts?
Have claims been fact-checked?
The fact of the matter is that getting to reality takes work. It’s work that, unfortunately, many won’t put in. It’s far easier to like and share.
Bad Reporting
The lousy reporting on deepfakes is constant, but let’s look at a couple of recent examples.
Take this article called: What Happened to the Deep Fake Election. Given the title, it would seem that it would be a step in the right direction. However, this person draws all the wrong conclusions.
Then there’s the article Welcome to the AI Election, which is really bad. This article is a whole bunch of fear-based nonsense. This is hilarious since the title claims this is the AI election but claims that the real AI election will be the 2026 mid-term elections.
By the time we get to the 2026 midterms, AI will be so much more advanced that in the hands of the right (or wrong) people, it’ll be able to generate hyper-realistic video content, which could be used to create personalized political narratives tailored to each voter’s psychological profile.
This is just straight-up nonsense. This person has no idea how any of this works. First of all, the cost of individual generation for each person would be an astronomical expense. You’d have the expense of the data collection and personalization components and the cost to generate video clips for each person. Second, this would require massive collusion, requiring tech companies and social media to play a part in the data collection and dissemination of the content.
For instance, they might be used to target us individually based on our biomarkers. Sorry, I forgot to mention AIs will soon have more information about us on a biological level, including our health and behavior. Why? you might ask. Because you’ll give it to them through apps and programs you’ll engage with, or are already engaging with.
What planet does this guy live on? Certainly not Earth. In all seriousness, I think I agree with the odd underlying point he’s trying to make. The privacy implications here are indeed very concerning, and we do risk being manipulated by these systems. So, I’m on board with that. However, shoehorning that into election manipulation, especially by 2026, is patently ridiculous. Who is the big bad person pulling those strings with all of the access to that data? In 2026, we’ll still have disparate systems and individual collections of data. Even if this were possible, it would still require a bunch of collusion between providers. This is a conspiracy theory dressed up like a technology prediction.
Just Not That Stupid
It’s difficult to grasp that other people aren’t that stupid. I think this is one factor that keeps the fear of deepfakes stoked. Ultimately, people believe what they want to believe, real or not. It’s been this way since the dawn of civilization.
As I mentioned in a previous article, we don’t see ourselves in other people. We see ourselves as outliers instead of the mean. This often gets warped by algorithms that keep us in a bubble and promote the most outlandish content. As someone who lives in Florida, this hits close to home with all of the nonsense about people thinking Hurricane Milton was not only manufactured but controlled. This resulted in meteorologists getting death threats.
If you tried to write a book on stupidity, this is so stupid that nobody would believe it, but then again, many conspiracy theories turn out this way. Conspiracy theory has morphed into a cult or a religion.
Back in 2021 I wrote, “Conspiracy theorists are like cult members, only worse. Worse, because a cult has a leader, but conspiracy theories make you the leader.” This everyone is a leader concept is incredibly empowering and addictive, making people both the hero and the victim. Despite this, most people don’t have deep conspiracy beliefs, yet they can receive outsized attention in both social media and traditional media alike. Don’t fall for it.
How Are They Being Used
So, how are deepfakes being used in the 2024 US Election? They aren’t, well, to be more precise, not with any relevance. Primarily, you see memes and slop, precisely as I predicted in previous articles. There was a boatload of bad reporting on the topic, but it doesn’t match reality. An obvious example was the Taylor Swift AI endorsement of Trump. These were reported as though they were deepfakes, but the whole incident was silly. If you went and reviewed these images, one is an AI-generated image of Taylor Swift dressed as Uncle Sam. The rest are Swifties for Trump. Comically enough, one of them is literally labeled as “Satire.” Where is the facepalm emoji when you need it?
Campaigns and political action committees (PACs) have used AI to generate content used as counterfactuals for campaign ads. This is something I think is disgusting, but they are hardly deepfakes. They are slop with a propaganda twist.
There was the fake Joe Biden voice call in New Hampshire, which was a legitimate deepfake. However, nobody believed it, and it had no impact. This isn’t unlike every other deepfake this election cycle.
Interestingly, Trump claimed that Kamala Harris’ crowd in Detroit was AI-generated. I also predicted this type of accusation behavior back in 2020 in my initial deepfakes article. Okay, so maybe now I’m taking a small victory lap.
Conclusion
It’s time to take a deep breath. We are less than two weeks away from Election Day in the United States. Undoubtedly, the AI election is here, and it’s more silly (and sometimes pathetic) than terrifying. Despite this, it will not be the last we hear of the fear-mongering. Fear sells, and fear gets clicks. Now, get out and vote.
In August, I sat down on the show floor of DEF CON to discuss a variety of topics with Vivek Ramachandran, the founder of SquareX. Our conversation covered a variety of topics, including AI at the intersection of humanity and technology.
You can listen to the podcast using your favorite platform, or feel free to watch the video below. It was a great conversation that certainly brought back some memories from the early days.
Also, to clarify one of my old school references, we were discussing the old days of wireless hacking, and I brought up a reference to the Proxim Orinoco Gold cards. If you’ve never seen the cards they feature a man with a briefcase far too happy to get WiFi.
I also forgot that they used the larger image of him on the box.
In just a few short years, we’ll not only have achieved AGI but live in a world of abundance. Physical goods will be so cheap that they’ll basically be free, and we’ll be able to 3D print anything we’d like. One can only assume with their free 3D printer. We’ll connect our brains to the cloud and have seemingly endless compute, which will also essentially be free. We’ll have cured our illnesses, created replicants, and become immortal. This is but a taste of the nonsense peddled by Ray Kurzweil.
In his new book, The Singularity is Nearer: When We Merge with AI, he pushes a couple of themes. One is that all technological advancements will be universally positive. Second, the only way humans can ever hope to compete is by fully merging with technology. I have issues with both of these themes. This post attempts to address only a tiny amount of the BS in the book.
Ray Kurzweil
If you are unfamiliar with Ray Kurzweil, he’s someone propped up by many as the preeminent futurist. I recently caught one of his appearances, and his ramblings elicited a noticeable grimace from me. I must admit, I wasn’t familiar with his unique brand of absurdity. I knew he’d said some wacky stuff in the past and had a book about the singularity, but I didn’t pay him much attention. After this interview, I purchased his new book, The Singularity Is Nearer: When We Merge with AI.
The strange thing about Kurzweil is the way people treat him during interviews. I haven’t really seen anyone push him on his asininity. When interviewers attempt to question him, he makes up more nonsense and says things like if people can live 20 more years, they’ll be able to live indefinitely or that things will be free in the future, avoiding the question altogether.
The book demonstrates how disconnected and out of touch Kurzweil is from reality. But it also highlights a bigger problem. As long as tech is allowed to be presented as magic, charlatans and hucksters will run rampant. This is the playbook that Kurzweil exploits. Unfortunately, I don’t have the time to address all of the issues with the book, but I will point out a few things that stood out to me.
As long as tech is allowed to be presented as magic, charlatans and hucksters will run rampant.
Before We Start
A few thoughts before we begin. If you read the reviews of this book, they are overwhelmingly positive. I’m sure many people won’t care for this post. Kurzweil is a famous tech personality with multiple books, TV appearances, and impressive credentials. I’m a nobody security researcher. I’ll never be in demand like Kurzweil or sell as many books as him, so I’ll have to cry myself to sleep at night with my integrity intact.
After all, he was just listed on the Time100/AI list, which caused me to laugh out loud. Then again, we live in a performative age, and Kurzweil is a performer.
However, I’ve spent my entire career analyzing risks and envisioning future threats to technology, something Kurzweil is oblivious to or completely ignores. Neither is a good scenario.
It’s also important to know that Kurzweil has been wrong many, many times before. I stumbled upon this old Newsweek article from 2009, which had an amazing quote.
P. Z. Myers, a biologist at the University of Minnesota, Morris, who has used his blog to poke fun at Kurzweil and other armchair futurists who, according to Myers, rely on junk science and don't understand basic biology. "I am completely baffled by Kurzweil's popularity, and in particular the respect he gets in some circles, since his claims simply do not hold up to even casually critical examination," writes Myers. He says Kurzweil's Singularity theories are closer to a deluded religious movement than they are to science. "It's a New Age spiritualism—that's all it is," Myers says. "Even geeks want to find God somewhere, and Kurzweil provides it for them."
The author even made a midlife crisis joke and another person accused him of trying to start a religion. Fifteen years, and not much has changed.
Let me also say that given enough time and technological progress, just about anything is possible. I think this is something that everyone innately knows. However, people like Kurzweil exploit this instinct for their benefit, running up the clock and leveraging the hype. We should be aware of this trick when evaluating claims.
Why Write This?
You might ask, why would I dedicate time to writing this article out of all the other things I could be writing? Indeed, I’d rather be writing something else, but as I was sketching my thoughts for this post, I read an article with the following quote.
“A colleague of mine, without a hint of irony, claimed that because of AI, high school education would be obsolete within five years, and that by 2029 we would live in an egalitarian paradise, free from menial labor. This prediction, inspired by Ray Kurzweil’s forecast of the “AI Singularity,” suggests a future brimming with utopian promises.”
THIS is why I’m writing it. These predictions powered by Kurzweil are fabricated bullshit. Let me go on record and say we won’t have AGI by 2029 or a utopia. Now, I’m not delusional in thinking that I would have nearly the reach needed to make a dent in Kurzweil’s impact, but I’ll reach a few people and get this off my chest. So, let’s dive in and call it like it is.
Kurzweil is a BS Artist
If I had to summarize The Singularity Is Nearer, I’d say it’s the ramblings of an aging gentleman confronted with his mortality, hoping that wishful thinking and vibes are enough to speed the tech he imagines into existence. It’s a book of absurdity wrapped in historical and disconnected examples attempting to give Kurzweil’s bullshit credibility. Even the title of the book is a sleight of hand. Sure, everything is nearer than when his previous book was written, but that doesn’t mean it’s close.
Another obvious fact on display is that if Kurzweil found himself down at the crossroads, we know exactly what he’d sell his soul for. He wants to become a robot so badly that he’s willing to shed every bit of his humanity to get it. Oddly enough, this doesn’t seem to be the bright, wavy red flag it should be. He’s so scared of death that he’s willing to discontinue being human for a small taste of an extended life.
Why Are People Convinced?
So, if my statements are true, then why is the book so convincing and the reviews universally positive? It’s not because people are stupid, but something far more simple. The book doesn’t continually and all at once slap you in the face with his flatulence. The pungent aroma is layered between positive messages (a utopia, immortality, etc.), topics that have nothing to do with this title, and historical examples of technological progress. This layering is a mental sleight of hand that has a reinforcing effect. Let me give you an example.
Imagine I wrote a book claiming that within ten years, humans would be exploring and populating the cosmos outside of our solar system. Rather than go into the specifics of my claim and address the real risks and challenges, I spend most of the pages talking about other things. I discuss at length the history of NASA and the challenges conquered to put humans on the moon, all in the span of a decade. I talk about the potential of solar sails and other propulsion technologies. I even go off on a tangent imagining the impact on humanity of having a working Dyson sphere. Kurzweil employed the same distraction techniques instead of making points or providing supporting evidence in his book.
The book itself has little to do with its title. He dedicates only a small portion of the text to this topic. He really wants you to know that, based on his vibes, utopia, and immortality are just a few years away. Kurzweil claims we’ll have a utopia in the next 20 years. It’s an easy sell since many people reading his book will still be alive. The entire book consists of telling people what they want to hear. He spends no time talking about challenges or issues. He knows that you sell a lot more books telling people what they want to hear rather than confronting hard truths. This sums up so much of our current age.
He knows that you sell a lot more books telling people what they want to hear rather than confronting hard truths.
That said, the book is an informative glimpse into the mindset of a certain type of person. These would be people with the transhumanist, posthumanist, or e/acc mindset. So much of the transhumanist argument is framed around making us better humans, but it’s really about making us into machines. I’m sure Kurzweil believes he describes a utopia. But like so many utopias, it’s just a thin layer of cheap wallpaper over a dystopia.
So much of the transhumanist argument is framed around making us better humans, but it’s really about making us into machines.
Disconnected From Reality
Kurzweil gives some of the most absurd examples in this book, proving that he has no idea how the world works and is disconnected from reality altogether. For example, after connecting our brains to the cloud, he imagines entertainment where we don’t merely watch a movie but feel the actor’s complex and disorganized emotions. Uhm… Can someone please tell him that actors are… well… acting? He doesn’t seem to realize that people acting in movies are expressing emotions, not feeling emotions. When we insist a tortured character in a movie needs to actually be tortured for entertainment, whose utopia are we living in?
Virtual Experiences
The book has an obsession with virtual experiences. He imagines scenarios such as a virtual beach vacation for your family, where you have the sights and smells of an actual beach. Nothing like taking a vacation with your family while, in reality, not taking a vacation. It reminds me of the company Rekal from the Philip K. Dick short story We Can Remember It For You Wholesale, which became the movie Total Recall for those who never read the story. I don’t know what it is with these people who seem to look at a dystopian SciFi and say, “Yes, that’s the technology we need.” These are cheap illusions that don’t have the impact of the real thing, but not to Kurzweil.
He claims simulations will be so good that there will be no point in doing the real thing and uses the example of climbing Mount Everest, which demonstrates he doesn’t understand what stakes are or what the point of doing something challenging is in the first place. In many cases, the point of performing the activity is the friction and difficulty. We just had the Olympics. Imagine telling Simone Biles, “Why put all of that hard work into competition? Soon, you’ll be able to ‘experience’ Olympic competition.” What Kurzweil doesn’t understand is that when experiences become easy, they lose their value.
When experiences become easy, they lose their value.
Confronted With His Aging
In many ways Kurzweil is keenly aware of his aging. This is obvious in his obsession with simple technology like replicants, which are merely trained on your writings. He discusses the replicant he made of his deceased father and rambles on about how fooled he was by his creation. This experiment was supposed to demonstrate the impressive capabilities of today’s technology, but it ended up just being sad.
However, what’s the point of having a chatbot trained on your writings and other material that exists after you pass away? It’s not you. It doesn’t have your identity or your true thoughts, nor does it encapsulate the complexities that make up your true identity. Even if you could create a more exact replicant, what’s the point? It still won’t be you. It could be a perfect copy of you, but it isn’t you. This is the kind of thing a narcissist would want. I don’t want a copy of myself running around, and I’m sure the world thanks me for that.
When you think more deeply about them, replicants have another problem. They are a get-out-of-jail-free card for not doing the right thing. Why spend time with your loved ones when they are alive if you can create a cheap copy to chat with at your convenience after they are gone? More time doing what you want and less time spending with the ones you love.
Things Will Cost Nothing
Not only will things be better in the future, goods will basically cost nothing. Kurzweil says that everything will become information technology, and the cost will go to zero or nearly zero, even basic necessities like food and clothing. He uses this transformation to say people won’t fight over resources anymore and uses a silly example, such as people fighting over a PDF. The whole premise is absurd. Vertical farming won’t drive food costs to zero, and people will fight over information. People get into fights over social media posts all the time.
Speaking of social media, he makes more ridiculous claims about social media and the cost/value tradeoffs. For example, he says it costs companies like Facebook, Google, and TikTok nothing after they’ve built their infrastructure, suspiciously omitting the energy costs and maintenance to run the infrastructure and the veritable army of people these organizations employ. He justifies his claim by stating that there’s no difference in cost between connecting you to a hundred people or a thousand people, as though the connection between people is where the cost is, but that’s not the stupidest part.
He says that if you could make $20 mowing a lawn but choose to spend that time on TikTok instead, then TikTok is worth $20 to you. This is asinine. Not every action you take in life is in service to make money, and not every free moment is a lost opportunity, either. So, in Kurzweil’s logic, if you could make $5 on Fiverr by designing a logo for someone but decide to sleep instead, then sleep is worth $5. You could make that $5 the next morning with no money lost. None of this even considers algorithms, the addictive nature of social media, and humans just wasting time.
Another spit-take moment is his discussion of radical life extension technology, which he states will not be available solely to the wealthy but also to the less fortunate worldwide. To prove this point, he uses the mobile phone as an example. Nope, you read that right.
Kurzweil says that since most people on the planet have a mobile phone, radical life extension technology will be available to them in much the same way due to extremely low cost. However, I think the mobile phone analogy is worth a deeper look. There’s a big difference between the iPhone in my pocket and an adware-riddled cheap cell phone subsidized by some company squeezing every drop of data from a user that it can. Tack onto this subsidized connectivity like Facebook’s Free Basics program meant to provide free internet to users in developing countries, which ultimately traps them in a Facebook hellscape, and you have the blueprint for something fairly dystopian.
Continuing his cost-nothing crusade, Kurzweil states that using robotics, cheap energy, and automation to replace labor outright in the 2030s would make it relatively inexpensive to live at a level considered luxurious. Telling people things will be cheaper, but you won’t be able to afford them because you don’t have a job is a contradiction that apparently didn’t dawn on him when he wrote that passage.
And… I’m not even going to get into his Bitcoin comments.
Jobs and Wages
Kurzweil has odd claims about jobs and wages. For example, he claims that more jobs will be created than lost, but he can’t answer what those jobs will be because they haven’t been invented yet. He uses examples like farming and the textile industry to prove his point. But this doesn’t make sense since AI is a far more generalized technology than a tractor or the power loom and can cross many different industries.
On wage stagnation, he boasts about how stagnated wages can buy more compute. Imagine that conversation with your family when having to skip a meal because you can’t afford food. “I know you are hungry, kids, but just think about how much more compute we have!”
I know you are hungry, kids, but just think about how much more compute we have!
One of Kurzweil’s favorite scare tactics is claiming there won’t be jobs for unenhanced humans and stating that until we fully merge with AI, there will be almost no jobs left. He makes multiple claims throughout the book on this point, saying biological brains cannot keep up with non-biological precision nanoengineering. Whatever the f—k that word salad means. This is another one of Kurzweil’s tactics on display. He knows most people know nothing about nanoengineering, so he bloviates on the topic. For good measure, he also mentions a world where we watch political ads or share personal data to get free nano-manufactured products. Ah, yes. The utopia we were all hoping for.
When it comes to automation replacing and disrupting the job market, he brings up a silver lining. The gig economy. He mentions the gig economy offers people more flexibility, autonomy, and leisure time. Kurzweil is so out of touch he doesn’t realize these aren’t the same thing. Once again, imagine that conversation. Telling someone who delivers for DoorDash, “Sure, you don’t have a regular job that pays well enough or has benefits, but isn’t all that leisure time great?” When you can’t pay your bills, downtime isn’t leisure time.
When you can’t pay your bills, downtime isn’t leisure time.
Being Human
In one part of the book, he questions what being human even means when introducing non-biological components and brain-computer interfaces. This is actually a great question, which, of course, Kurzweil doesn’t answer. Instead of answering, he vomits more of his pontification about inevitability, saying the non-biological component will grow exponentially while our biological intelligence will stay the same, providing a more specific prediction that in the 2030’s our thinking itself will be largely non-biological. Kurzweil has a way of stating questions as though he’ll answer them but never answering them. This is how a con artist operates, appearing to be upfront.
It should be obvious to anyone reading the book that Kurzweil really doesn’t like being human and yearns for the day to transform into something else. It doesn’t even matter to him what he becomes as long as it isn’t human.
For example, it’s uncomfortable (but necessary) to think about how replacing our biological components with synthetic ones may change us, especially when it’s not for the better. Instead of addressing this complicated reality, he makes the point that we remain the same person despite our cells going through a replacement process and our brains being almost completely replaced over the span of a few months. The implication he hopes you draw is that this non-biological replacement shouldn’t bother us. Once again, more absurdity.
Bodily regenerative processes are not the same as a wholesale replacement by synthetic alternatives. This holds true for both physical and cognitive functions. This irritates me to no end, and it’s one of the most obvious flaws in his logic. Kurzweil hopes to smother us with a pillow while he whispers, “Just let the singularity happen.”
No Downsides
One of the most apparent aspects to readers of the book is Kurzweil’s failure to mention nearly any negative aspects or potential adverse outcomes in his book. Either he’s oblivious to them or feels that adverse outcomes don’t align with his message. My guess is it’s a mixture of both.
I’ve discussed many of these downsides already, but one in particular is his presentation of simulation and self-driving cars as though they’re magic. To support this, he mentions the success of companies like Waymo. There is never a mention of Waymo’s issues, such as how these cars have been found driving down the wrong side of the road or mysteriously honking their horns. We don’t have capable Level 5 self-driving cars on the road today, and this problem is not solved. Every company working on self-driving features, from Waymo to Tesla, has issues they cannot solve today.
These are undoubtedly solvable issues, and we will have full driverless technology in the future, possibly even in the near future, but today, these companies can’t solve the problems. It’s undoubtedly disingenuous to talk about driverless cars as though they are a solved problem today.
Okay Not Knowing
Another of Kurzweil’s comfortabilities is agreeableness to not knowing how AI works or comes to its conclusions. He mentions that we may not know or understand even if explanations were provided. It’s odd that he mentions this while talking about the judicial system, an area that’s been plagued with algorithmic issues. Even outside of the judicial system and policing, there have been so many instances where algorithms have unfairly discriminated against people, denying them benefits and even entry into schools. Recently, it was announced that Nevada will use Google’s AI to determine whether people get benefits. People have a right to know why they were denied benefits, and it can’t be, “because the algorithm says so.”
Imagine an air traffic control AI that instructs pilots to fly figure 8’s around the airport before landing. Will we question this or receive it as some sort of hidden knowledge that the AI system has that we can’t fathom? This would be an obvious example that the system has an issue, but countless hidden issues wouldn’t surface in the same way. When we don’t understand how a system came to its conclusions, we set ourselves up for confounders to run rampant.
As I read the section on the judicial system, I wondered how you would ever get a fair trial by jury in the future. When everyone is permanently connected and has access to data that biases them, it may be possible for anyone to get away with a crime purely by spending enough money to taint the data. Or will you be forced to install the JuryBlocker software directly into your cognitive processes? I’m sure Kurzweil would think this thought exercise is silly because the goal is to remove humans from the judicial process altogether, but as we know, we don’t live in a perfect world. Our technology is rarely that good, and humans have a habit of not making the right decisions.
Not The Whole Story
There were so many parts of the book where Kurzweil would bring something up, and I’d be left with the thought, “That’s not the whole story.”
For example, he references things like ChatGPT passing the Bar Exam or AlphaGo beating the best Go player in the world but never tells the whole story. For example, when ChatGPT passed the Bar exam, it also passed other similar standardized tests. Researchers reworded the questions to ask the same question differently, and ChatGPT failed, proving that it had memorized data in its training data. Kurzweil wants you to believe that because of this, a lawyer’s days as a profession are numbered, but his exercise misses the more significant point that lawyers don’t sit around answering Bar exam questions all day.
AlphaGo was a truly amazing accomplishment, but Kurzweil leaves out that even average Go players can beat superhuman Go AIs these days. They can exploit these systems through adversarial policy attacks. These attacks are highly concerning if the technology is deployed in high-risk scenarios outside the game of Go.
In his discussion about disruption from AI, he claims that sometimes there aren’t any losers. For example, a revenue stream from treating a particular illness. He says there are many areas of technological change where losers don’t exist and gives the example of creating a cure for a disease. In this scenario, companies and individuals lose a long-term stream of revenue. This is another one of those sleight-of-hand things Kurzweil does. The cure is indeed more beneficial to society, but that’s certainly not how things play out in practice as large pharmaceutical companies hang on to revenue streams. No matter how cloud-connected your brain is, you won’t be able to compete with large organizations and a mobilized workforce. There may be occasional exceptions, but it’s hardly the rule.
Conclusion
This lengthy post didn’t scratch the surface of the nonsense hawked by Ray Kurzweil in his book. There are so many points I take issue with, most specifically the arguments of inevitability and the aggressive timelines he’s attached. Given all of his bullshit, you might be surprised that prominent people continue to hold him in high regard, but I’m not. Kurzweil’s tech spirituality aligns with their larger goals.
We need to ask more serious questions of people trying to sell us things, even those selling us ideas, because these things have consequences.
Look For Yourself
Before writing this article, I didn’t look for other articles or takedowns of Ray Kurzweil. I didn’t want these pieces to taint my impressions of the claims made in the book. The only exception was the Newsweek article from 2009, which I stumbled upon while looking up a specific piece of information about him. After writing the article, I was curious about what others had to say, and I promised myself I wouldn’t go back and reword anything in this article based on what I had read.
If you are still unconvinced, I’ve highlighted a few articles you can read for yourself below. Some of these are older articles, proving that nothing has changed. These are worth the read.
How Ray Kurzweil Sells His Junk Science – Geoffrey James – June 17, 2010 This is an old article, but it’s worth the read for the rules of selling junk science.
Ray Kurzweil Does Not Understand the Brain – PZ Myers – August 17, 2010 This is PZ Myers smashing Kurzweil for making the claim that by 2020, we’ll have reversed engineered the human brain. Obviously, that didn’t happen.
The singularity is not near: The intellectual fraud of the “Singularitarians” – Corey Pein – May 13, 2018. This article has an amazing quote. “Science begins with doubt. Everything else is sales.” This is something we should all keep in mind as we blindly take as fact the drivel of the AI Hype Bros. For some more notable bangers by Corey Pein, check out Cyborg Soothsayers of the High-Tech Hogwash Emporia. “Ray Kurzweil’s Singularity is an overheated white paper by a zealot for the American dream of luxury and convenience.” There were a whole lot of references to this type of thing in the book.
Anyone digging even mildly beneath the surface will see that Ray Kurzweil is a charlatan and a huckster. He’s not someone to be taken seriously. Despite this, many tech people will continue to genuflect for Kurzweil because he says what they want to hear. I also mentioned in a previous post that in our short attention span existence, we reward people for being bold, not for being accurate. Something that Kurzweil happily exploits. Welcome to the age of post-reality.
I had a great conversation with Aseem Jakhar for CIO.inc and iSMG. We covered topics surrounding AI Safety and Security as well as deepfakes. I explained why I don’t think the misinformation aspect of deepfakes will affect the outcome of elections and provided my opinion on deepfake detectors. We also discuss how we think we need to throw out the rulebook every time a new technology comes along instead of applying lessons learned.
Humans are social creatures, and friendship and love are relationships that run deep in our history, predating Homo sapiens as a species. We associate these relationships as core features of our humanity, but companies are attempting to change this. Every time a new technology comes along, people try to use it to solve complex social issues that have nothing to do with technology, and with AI, it’s happening again. Would you have a chatbot friend? Would you marry a chatbot? There are companies developing products that hope you will. Welcome to the attempted dehumanization of friendship and love.
Solving Non-Problems
There are few things that I can say for sure, but I will say with certainty that the world won’t be a better place when both friendship and love are simulated, and we treat apps like humans and humans like apps.
The world won’t be a better place when both friendship and love are simulated, and we treat apps like humans and humans like apps.
When we take a step back, one thing that should be obvious in the current generative AI craze is that solving non-problems is far easier than solving real problems. This makes sense. There’s a low cost of failure in addressing non-problems. Hell, you don’t even need to _solve_ non-problems to be successful. Let’s think about it: it’s not like the world has a shortage of writers, artists, and musicians. However, those specific non-problems are a topic for another day.
Speaking of solving non-problems, rather than using generative AI capabilities for well-suited tasks, we’ve witnessed an abundance of what I call shitty AI gadgets. What makes them “shitty” is the fact that they don’t actually solve a problem. The focus for many is how “cool” the technology is without emphasis on whether it solves a problem or does anything at all.
This joke by @plibin on Twitter sums up what every single one of these gadgets looks like to me.
Shove generative AI into every technological crevice possible and hope that money sprouts. These products are only good for setting fire to VC money.
When AI is Your Friend, You’ve Got No Friends
The latest shitty AI gadget is called Friend. No, not a joke. And apparently, they spent most of their raised money on their domain name.The Friend gadget also exhibits higher levels of cringe than other gadgets. Other gadgets at least pretend to do something useful. Friend is happy to do nothing at all.
A glance at their commercial is all that’s needed to address doubts about peak-level cringe.
If you want some faith restored in humanity, read the comments. The people writing the comments are human, and they get it—something that the Friend team doesn’t.
Watching the Friend commercial shows just how disconnected these people are from reality. If they are trying to shed conspiracy theories about how they are secretly unfeeling reptilian aliens, they are failing. I mean, what date is going to put up with this? Oh, what is that around your neck? Yeah… I’m sorry, I just realized I have something else to do.
Of course, all of these miss the larger point that someone invested in the Friend device wouldn’t be on a date in the first place, nor would they be out enjoying time with “real” friends.
In looking to optimize everything, including our personal lives, AI friends make sense. It can be all about us. We’ll never have to listen to them tell us about their problems or need to be a shoulder for them to cry on. We may even enter an era where many people don’t know what true friendship feels like.
However, it’s not just loneliness that would drive someone to AI friends or AI lovers. Part of the problem stems from people wanting sure things. There is no perceived risk, fear of rejection, or potential pain. A chatbot will not reject us or tell us things we don’t want to hear—well, unless we don’t pay the bill. This is a powerful pull that some will find attractive.
Isolating Effects
An AI friend or lover wouldn’t have us out living our best lives in the real world because they have an isolating effect. These gadgets provide users with a false sense of companionship and exacerbate the very issues they purport to solve. Rather than going out, we stay home. We stay home and play it safe rather than going on a date and taking a chance on love. If gadgets like Friend were to take off, this would be a net negative for health and wellbeing.
An AI friend or lover doesn’t care if we live or die. It doesn’t care if we are happy or sad. Subconsciously, even if we fool ourselves, we know this.
I’ve mentioned when AI is your friend, you’ve got no friends. I’m not just referring to the uncaring stochastic companion we haul around, but it’s the fact that it makes people not want to interact with us. This aspect further isolates us from the real world. I mean, which of our real-world friends would put up with this?
If I wore the Friend device to a get-together with my actual friends, they would launch a merciless onslaught of insults and fun at my expense, and that’s why they’re my friends. Real friends keep us honest. They don’t let us get full of ourselves, and they don’t just tell us everything you want to hear. This feedback helps us grow and have greater life satisfaction.
Nothing easy is satisfying or worth having. This applies to friendship and love as well. The modern world promises that we don’t need to delay gratification. There’s no sense of investment. Everything needs to be an instantaneous hit of dopamine. There are very few things in life where instant gratification is nearly as satisfying as a delayed gratification activity.
In a recent interview with Eugenia Kuyda, the CEO of Replika (An AI friend company), she said, “It’s okay if we end up marrying chatbots.”
Here’s her response to a very good question:
Question: “When we started out this conversation, you said Replika should be a complement to real life, and we’ve gotten all the way to, “It’s your wife.” That seems like it’s not a complement to your life if you have an AI spouse. Do you think it’s alright for people to get all the way to, “I’m married to a chatbot run by a private company on my phone?”
Kuyda: “I think it’s alright as long as it’s making you happier in the long run. As long as your emotional well-being is improving, you are less lonely, you are happier, you feel more connected to other people, then yes, it’s okay.”
Feel more connected to other people? Really? This is disconnected, disingenuous, or outright stupid. Sure, it could be simple disingenuousness. After all, her job is to hawk her company’s wares. But it should be obvious that being married to a chatbot won’t make us more connected to other people. This situation reminds me of a documentary I watched years ago about people in love with their RealDolls. They’d take them out for drives, sit down for dinner, and watch TV with them, just like another human. You know what the documentary didn’t show? Their friends!
We highlight this disconnect by examining something simple between real friends, like laughter. Is our LLM-powered friend going to make us laugh? I mean, a real guttural laugh that sticks with us? Or will it try to entertain us with a mindless video it thinks we’ll like, generating a momentary chuckle that gets lost in the din of distraction? This and many more cheap substitutions await us, beaten into submission, until we won’t remember the real thing.
More Cringe
With the Friend device, there’s a supreme disconnection from reality, but this isn’t the exception. This is becoming the rule. The Friend gadget is the most obvious incarnation of this, but this disconnection is everywhere in the AI space. This is on full display when we hear the AI tech crowd talking about creativity and creative arts. You can tell these people have never been creative in their life and understand nothing about art. Not even a little bit.
I can’t remember who said this, but someone commented about this situation, saying that the Silicon Valley crowd is just a bunch of people having fun with their friends. There’s some truth to this. It’s like a Silicon Valley garage band, but instead of music, it’s tech. So, it’s not about art or creativity at all. The point is to make “cool” tech, whether it solves a problem or not. It’s a familiar theme.
However, startups are not the only ones exhibiting this cringe factor and disconnection. Google’s new Gemini video mixes both cringe and dehumanization, all in the name of optimization.
There are so many things wrong with this commercial. All these tech types fail to realize that some things are supposed to have friction. Friction is how we grow and become better. Friction is how we challenge ourselves. Even things like second-guessing and self-reflection are a form of friction. We are optimizing all the wrong things, a topic I’ve covered twice before, in Optimizing Away Human Interactions With AI and Outsourcing Simulated Emotional Connections To Bots.
Now, do you think Sydney would rather get a letter from a little girl who struggled to put her words to paper, leaving every imperfection as evidence of her effort and caring, or from Gemini? Which scenario do you also think would be better for the little girl? The answer is so blatantly obvious, well, obvious to us humans, at least. (I’ll avoid making a second alien joke here.)
Technical Issues
So far, I’ve only discussed the human aspects of technology, but there’s a lot more when considering the technical risks. There’s far too much to cover, but I’ll highlight two. For more information, you can read my post introducing SPAR.
Privacy is one of the obvious issues. This is because all of that data collected and shared with our AI friend is valuable. If there’s one thing we’ve learned from recent history, it’s that data available is data exploited with all of our personal thoughts and interactions monetized and weaponized against us. Even if the startup creating the AI friend application claims to respect your privacy, when they get acquired (possibly specifically for this type of data) all bets are off.
At least the people in the documentary I watched years ago didn’t have to worry about their RealDoll harvesting sensitive data and snitching back to the company.
Perverse Alignment
Can we be sure that our AI friend is aligned with our best interests?
A perverse alignment is the alignment of a system to serve the best interest of the company or organization that created it over the user using the system. There is the potential to nudge and push users to do all sorts of things. This may be to buy products or spend more time on the platform. In the AI friend scenario, spending more time on the platform leads to less time with real friends.
It may be difficult to identify when a system is aligned like this. It’s not like our AI friend will respond, “You’ve been worried about car insurance. Do you know who has great car insurance? GEICO.” I made the same GIECO joke back in February 2023 about AI-powered search engines. I gotta get some new material.
Loneliness
I don’t mean any of this to discount the loneliness epidemic happening with younger people. This epidemic is something Jonathan Haidt covers at length and is infinitely more qualified to address than I am. I’ll give you a hint, though. Do you know what he doesn’t recommend? More technology.
This crisis is, at least in part, fueled by technology. There’s something perverse about layering even more technology to solve a human problem. An old saying about treating the symptoms instead of the cause applies here.
There’s a problem with a device that is basically a super-powered inspirational quotes machine, telling us everything we want to hear. We never get better, we never challenge ourselves, and we never encounter real satisfaction. We get stuck in a loneliness loop, with only momentary relief. It’s like if we had an excruciating headache every day, we wouldn’t put up with it, chewing ibuprofen like it was candy to gain temporary relief every day. We’d try to find the cause and address it. This situation is no different.
The AI Religion
Part of the problem is that AI has turned into a religion. I’ve joked about how these devices often resemble communion wafers, but I don’t believe the Catholic Church has had any influence on them. People have talked about AI in more religious contexts, attacking people without enough faith and elevating people they believe are prophets. AI has died, AI has risen, AI will come again.
Religions seldom involve questions, at least not questions that have answers, which is perfect for our current AI moment and aligns with hype. We have to take it on faith that things will get better, and the sermons from AI prophets aren’t merely an attempt to get more profit.
Read Ray Kurzweil’s new book The Singularity is Nearer for more religion-related disconnections from reality. I swear, I’ve pulled a muscle in my neck, shaking my head at all the misperceptions and misunderstandings contained within the book. But Kurzweil is a prophet in the church of AI, and what I’m saying now is blasphemous. If Kurzweil says something, it requires taking it on faith.
When we dig into it, people like Kurzweil, Chalmers, and Clark push a transhumanist vision for humanity that converts us into the Borg, stripping away our humanity and turning us into machines. Resistance will most likely be futile.
What happens when we evolve not to know or have true love and friendship? Will we be better or worse off? Evolving into a machine doesn’t sound appealing to me, but the transhumanist figureheads push the opposite perspective. Transhumanists push the perspective that merging with machines will make us superior humans, but it will most likely make us average machines. That’s not a good trade. I’ll expand upon this in a different post.
Transhumanists push the perspective that merging with machines will make us superior humans, but it will most likely make us average machines.
Don’t fret over my immortal digital soul. I’ve already prayed my five Hail Turings for the day.
Conclusion
As we navigate the sea of innovation porn, let’s not set our course away from humanity. Core features of our humanity make us unique on this planet, not our processing capabilities. We can have technology that works for us and maintains our humanity. Don’t believe those who tell you it’s a tradeoff. They are selling something.
Also, let’s use LLMs for what they are good for, not for friends or lovers. There are plenty of tasks for which you can apply LLMs to boost efficiency and actually solve problems. Do that. Friendship isn’t a technology problem. Neither is love.
If you are hopeful about the future and of technology but remain skeptical of BS claims and other nonsense, hang in there. More and more people are voicing their opinions, and it’s no longer a lonely hill to stand on.
So, you are attending a security conference. That’s great news. Every year, more and more people join the security community and explore the experience of attending a security conference. If you are new or relatively new to security conferences, getting the most out of your experience isn’t obvious. There are things people don’t tell you that you learn through trial and error. I hope to make this a bit less painful.
I’ve been attending and speaking at conferences for over twenty years. I’ve given my share of advice to new attendees, but it’s always been in person or as a couple of pointers to someone else’s social media posts. I wanted to share some insights since Black Hat USA and DEF CON are next week. As our community grows, new people begin to attend conferences and much of the advice they are given isn’t helpful. I hope to rectify that. If I can help even a few people make the most out of their experience and avoid some pitfalls, then this post served its purpose.
Bad Advice
Let’s start with bad advice. There’s no shortage of truly unhelpful advice that people seem to dish out as though they dispense ancient wisdom. You’ll hear things like pace yourself, drink water, and wear deodorant. It’s the type of condescending, childish response you get from people who pretend they have some secret knowledge and don’t want to share it with you. Yes, Las Vegas is in the desert. It’s also hot. Anyone with two brain cells to rub together can put as much together on their own.
Worst of all, we all know this isn’t the type of advice people seeking it want to hear. It’s like someone stepping up to the high dive for the first time and asking for advice, only to be confronted with someone telling them to make sure they bring a swimsuit. So, let’s all agree to stop doing this.
Understanding Security Conferences
I absolutely love security conferences, even though I may jokingly tell you I despise them. You see, I’m not the biggest people person in the world, but at a security conference talking shop with others, I feel like I’m in my element. I’m more likely to strike up a conversation with complete strangers and make new friends and acquaintances. This is why I want to encourage people to attend, interact, and engage. Now, we are starting to get to the heart of what security conferences are.
The first thing to understand is that security conferences are events or experiences if we are being dramatic. These events are only partly about the presentations delivered on stage. Thinking that security conferences are only about presentations is like thinking that county fairs are only about Ferris wheels. Conferences of all sizes typically have other activities, such as meetups, CTFs, vendors, contests, etc. This is what builds the experience of attending.
It surprises people to discover that presentations may not be the most valuable part of the conference. Given this perspective, you’ll want to maximize your conference experience. Hopefully, the rest of this post will assist with this goal.
It surprises people to discover that presentations may not be the most valuable part of the conference.
Start Here
What do you hope to get out of your conference experience? What would a successful conference experience look like when you return from the conference? Think of this as your destination on your map. However, instead of planning a turn-by-turn route or creating a script you need to follow, you’ll want to think of it as throwing out some waypoints and figuring the route out later. You’ll also want to leave some options open, as well as the ability to capitalize on serendipity.
If the conference has an official app, use it to build a schedule. If you’d prefer to use another calendar with which to build a schedule, that’s fine too, but just have a way to track events and activities that interest you
Sessions and Presentations
There will undoubtedly be sessions you won’t want to miss. Add these most important sessions to your schedule. Additionally, If you don’t have other conflicting activities, fill up the remaining conference hours with potential sessions that interest you. These will be your optional sessions. You aren’t locked into going to any of these, but at least you aren’t standing in the hallway trying to figure out what session you want to see or figuring out where it is while the sessions have already started.
Again, this is just a high-level plan and not some script you must follow, so fill the time slots. Okay, that’s it for the content and presentations. If you believe conferences are only about presentations, you can stop here. Wasn’t that easy? But if you really want to understand security conferences, keep reading.
The Security Community
It’s called the security community for a reason. The security community is among the weirdest, quirky, and welcoming communities of any industry. Full stop. I can’t think of another community where you could walk up on a technical conversation between a person in a sports coat, a furry, and someone wearing a tinfoil hat and shirt that reads “birds aren’t real.” All three of them have a great conversation, respecting each other’s perspective. If you wrote this into a TV show, nobody would believe it.
The community aspect of our industry is arguably the most valuable asset you can cultivate. It can launch your career into a whole new trajectory, allow you to make new friends, and introduce you to new avenues you didn’t know were open to you. All of this is on full display at security conferences.
The community aspect of our industry is arguably the most valuable asset you can cultivate.
This community aspect is why you’ll want to maximize your participation in networking opportunities. These include meet-ups, gatherings, and events that are both official and unofficial.
Hallway Con
Hallway Con is the security community’s conference. It’s the con within a con. Hallway Con is the reference to all of the conversations that happen outside of the presentation area. It’s the meetups, networking, catching up with old friends, and discussions with strangers, experts, and peers.
Hallway Con is where the real information is shared. People disclose details they wouldn’t share on the stage or over a video conference. This less filtered sharing allows for a more accurate picture and perspective of realities on the ground. People will be more honest in their assessments and give candid responses.
A couple of notes about Hallway Con. The Hallway Con experience is impossible to virtualize. You will get information from a virtual conference, but you won’t get the experience. This is a face-to-face, in-person activity only. Despite attempts to recreate this experience, nobody has successfully done it yet. This means if you attend a virtual conference, you miss out on some of the most important value of the conference.
You will get information from a virtual conference, but you won’t get the experience.
Hallway Con has a randomness aspect, and although you can’t purposefully plan it, you can enhance your odds of being successful. Here are a couple of tips.
Catch up with people you know
Are there people you want to talk to or catch up with? Reach out to them and see if you can catch up. You could meet in a common area or have lunch. Anything that might get a conversation started. These people will often introduce you to new people.
Scope out any interesting meetups
Meetups are specifically for this networking purpose. Maximize your opportunities by attending them. It’s good to remember that not all of these meetups are published on the conference schedule. Be on the lookout through various social media platforms as well as with conversations onsite. You can also ask about any meetups through your social media platforms and see what responses you get.
Add known meetups you’d like to attend to your schedule for tracking purposes.
Roam around a common area and sit with strangers
I like to roam around the common area to see if there are people I recognize. These common areas are collecting points. You probably won’t know many people if you are new to the community, but it can still be a good exercise. You can also sit with strangers. I often sit with strangers during conference lunches. It’s a good way to be forced to introduce yourself and spark conversations. You typically find that you have much more in common with these strangers than you thought.
Chat with presenters after their session concludes
Presenters are often also conference attendees. They don’t just deliver their talk and fly out the door. You can talk with them directly after their presentation to ask additional questions or get further clarification, but you can also see them roaming around common areas and having additional conversations. Remember, speakers will likely be more candid off-stage than on, so take advantage of this.
Be prepared to share
Security people absolutely LOVE to talk. So, don’t worry if you are talking to an expert and are only asking questions. However, strangers are more open to sharing with you if you share something with them. This doesn’t have to be in-depth technical information. It could be challenges you are having or things that aren’t working.
Plan Your Evenings
There will undoubtedly be after-hours activities. The bigger conferences have afterparties sponsored by organizations and vendors. Sometimes, you can just walk up and enter, but many of these events require pre-registration. This isn’t something to put off. Research these events and register ahead of time.
These are also opportunities to maximize your networking. You’ll be there with other attendees and speakers. No speaker has ever not enjoyed someone walking up and telling them they enjoyed their presentation. Use it as an opportunity to spark a conversation.
Events and Contests
Security conferences are also about entertainment. There are many reasons you may want to watch Hacker Jeopardy and not actually participate. However, if you want to participate in a contest, you’ll probably need some preparation. There may be pre-registration or qualifying rounds. This may also include other preparations like bringing your computer with tools installed. Do a little research to increase your success.
Remember
Every security conference is different. Just because you attended one and didn’t like it doesn’t mean you won’t like another one. Taking it all in your first year at a larger conference is also okay. Don’t be too hard on yourself. It can be difficult to know what you are in for your first time at a large event. Plan the things you can and learn from the things you can’t. There’s always next year.
Misconceptions Persist
Okay… I’m gonna rant.
It irks me to no end to hear sales and marketing people who have never attended these security events talk about them like they know something about them. You hear things like security leaders and CISOs don’t attend DEF CON, that there’s no value in having someone speak at conferences because it doesn’t directly lead to sales, that security conferences are just an excuse to party, and many other completely out-of-touch statements. These are nonsense perspectives from people who are out of touch with the very community they are supposed to serve. If you are one of these people reading this and think I’m talking about you, then I’m absolutely talking about you.
Throughout my entire career, I’ve tried to educate people about the security community and the value of interaction at conferences. I’ve had both successes and failures. Some salespeople only want to sell firewalls to customers who, in turn, could care less if they bought them from a chatbot. There’s no competing with that.
If you think I’m being harsh, you should see the original draft wording of this section. I’m feeling generous today, so this is me being nice 😊
Rant complete.
Finally
Above all, have fun and enjoy yourself. Don’t stress about trying to make everything you want to see. Oh, and your head hurting from the knowledge you gained and your vocal cords a little raspy from talking so much are symptoms you’re doing it right. Remember, it’s not about what you learn but how you modify and apply it to your own challenges. It’s how you take the information in new directions and make it your own. That’s what this is all about.
In my haste to publish this post, I’m sure I’ve missed things. I’ll leave that to others to fill in any obvious blanks. See you at a security conference soon!
