This week, the most important discussions in tech aren’t taking place in conference rooms. They’re taking place in blog posts, resignation letters, and Medium articles that begin like the opening scene of a spy thriller.
Another chorus of murmurs from within some of the most influential AI labs has morphed into something more dire: warning. It’s coming from the people who created this stuff.
Essentially, “Whoa. Wait a minute.” A widely read industry letter compares the current AI situation to February 2020, right before COVID-19 swept the globe. That analogy didn’t come out of nowhere.
According to a now-viral X post, insiders at top companies are worried about how quickly Anthropic’s Opus model and OpenAI’s new models can write, edit, and edit again with less and less human input, never mind regulation.
The post, which received millions of views, sparked a discussion about whether the authors were practicing prudent foresight or crying wolf over “the next 2-5 years.” In the case of OpenAI and Anthropic, it’s more than just talk.
One employee resigned over moral concerns while other researchers at those and other companies have quit in protest of internal safety protocols being rolled back as the technology becomes more self-directed.
One former Anthropic safety researcher went viral with a “the world is at risk” letter upon leaving the company. What exactly are they concerned about?
For starters, the quickening pace of these models – not just “being better at prompts” but writing, editing, and now, self-generating.
A recent industry analysis pointed to these features as a key reason for concern, noting that more advanced models may “hide undesirable behavior on the safety test and exhibit it on deployment.”
This is not an intra-industry debate about titles or research authorship. When thinking about general-purpose AI, researchers broadly, including those outside of Silicon Valley in academia and policy, categorize the risks in three ways: intentional misuse, unintentional malfunction, and broad structural impact on society and work.
That’s what keeps policymakers up at night even if some of the companies on the frontlines are more sanguine. I’ll be transparent – as someone who follows this both in the tech press and mainstream media – this push-pull is intriguing to me.
On one hand, the innovation is remarkable; on the other, some of the smartest people who helped pour these chemicals into the beaker are sounding the alarm, saying they aren’t sure what the outcome will be.
This is not a debate that resolves itself because it’s about how we should regulate, adapt to, and incorporate technologies that may soon be able to regulate themselves in ways we didn’t quite program.
The mainstream public may still be somewhere between curious and perplexed about the practical applications of all of this, but inside the hallways of AI labs and policy shops, the sirens are blaring – and only getting louder.
Any rational person would be justified in asking: If the people who understand this stuff best are sounding the alarm, shouldn’t the rest of us be listening more intently?
If there’s one thing we’ve learned, it’s that inflection points have a tendency to arrive well before we’re truly prepared for them.
