By now, you’ve probably heard of the letter [https://futureoflife.org/open-letter/pause-giant-ai-experiments/] from the Future of Life Institute, signed by experts calling for a pause on AI experiments. Odds are, you’ve seen reporting on it but didn’t read it yourself. You should go read it before forming too much of an opinion. Some, like Eliezer Yudkowsky even argue that the letter didn’t go far enough and we should shut it all down.

The criticism surrounding the letter hasn’t been intellectually honest either. It seems everyone wants to get their hot takes in and is looking more for one-liners than solutions. Critics attack the institute or the individual authors without acknowledging the concerns outlined, committing a logical fallacy known as the genetic fallacy

The criticism seems to fall into one of three camps:

• I don’t like “X” therefore, nothing is valid.
• My concern wasn’t addressed. Therefore, nothing is valid
• There are no dangers
  

If you don’t believe me, feel free to read the rebuke letter.

Welcome to the hot mess that is AI alignment.

We can be harmed by both good and bad AI

I’ve previously weighed in on the alignment topic and expressed my concerns about the current development trend. So if you are looking for definitions of alignment and the paperclip maximizer, you can see that here.

As a security researcher, I’m much more concerned with near-term AI risks. These are the risks typically caused by bad AI, for example, the velocity at which we see companies shoehorning chatbots into their application is concerning. But, there are risks with good AI as well. Good AI has the potential to displace many workers, negatively affect how humans communicate, and create other societal damage. Of course, both good AI and bad AI can have privacy issues as well.

Transformer-based Large Language Models (LLMs) won’t lead to AGI. They aren’t actually reasoning, and they don’t have specific goals to maximize, but what comes next just might.

My Problems with the Letter

My issues with the letter have to do with the optics of it, and I think it’s a bad look overall. The letter makes it far too easy to attack by making the signers look irrational or out of touch, something which, if you look at the names on the list, they are not.

The first issue I have is the mention of GPT-4, which I think is a mistake. This mention makes it seem like either GPT-4 itself is the problem or was some sort of catalyst for the letter. If you read the letter, you can tell that’s not the case, but if you look at the criticism lobbed at the letter, this is used because some of the signers have criticized the capabilities of GPT-4.

The second issue I have has to do with the six-month pause on training more capable AI systems. The duration of six months looks arbitrarily chosen. There is a list of what the hopes would be during these six months, but it seems far too heavy a lift. Companies could use the pause to gain a competitive advantage, or bad-faith actors like nation-states could continue their development.

Finally, the letter calls for an unrealistic recommendation that has no way of coming to fruition. These companies aren’t going to stop their work. There’s no incentive for them to do so. In fact, there is precisely the opposite. These companies are locked in an arms race.

This seems like an issue where we are going to have the build the airplane while we are flying it. However, it’s hard to deny that the letter is having a positive effect. The Biden administration discussed AI dangers with the President’s Council of Advisors on Science and Technology (PCAST) yesterday. It’s hard to believe that conversation happening when it did without the attention around the letter.

AI Alignment and Ethics

AI alignment and ethics are two different areas, and researchers focused on one aren’t researching the same things as the other. At a high level, alignment folks are focused on the intended goals of a system, and ethics folks on the harms from systems, intended or unintended.

However, when the topic of alignment comes up, ethics is always lumped in there. AI ethics organizations don’t have the best track record of proposing realistic approaches either. Recommendations are often overly academic, disconnected from the real world, and sometimes even illegal. That is unless you think the way to stop a ruthless dictator from using deepfakes is to publish a usage standard, or you think the way to make systems fairer is to collect even more sensitive personal data. These out-of-touch recommendations make it easier to discount the larger AI ethics community as well as the alignment community. This is a huge mistake because we need both communities for a successful future.

Additional Reading

Like everything else in our culture, AI Alignment has now become a spicy, politicized topic. In reality, we need to separate ourselves from the hype and arguments and focus on the very real problems.

For some additional reading on this topic, Dan Hendrycks published a paper called Natural Selection Favors AIs Over Humans. It’s a great read with some solid food for thought, especially the section on Value Erosion, which is something I’m incredibly concerned about and have covered in various aspects on my blog.