Skip to content

Anthropic warns in its IPO prospectus that it is close to AI that improves itself: it admits it once noticed an acceleration in its own development only after it had happened

1 min read
Share
Anthropic warns in its IPO prospectus that it is close to AI that improves itself: it admits it once noticed an acceleration in its own development only after it had happened

Anthropic has published its revised filing to go public (the S-1/A document), and attention went straight to the numbers: revenue is growing, but so are costs. Deeper in the document, in the risk section, sits a sentence more important than any balance sheet. The company writes that it may be on the threshold of artificial intelligence that improves itself.

The term is recursive self-improvement (RSI) - the point at which AI becomes so good at training itself that development starts to accelerate without brakes. For some, that's the moment humans lose control and there's no way back. For others, it's a promise of productivity and scientific discoveries we can hardly imagine. Anthropic, according to its own document, thinks it's both.

There are two new sentences in the risk section. The first: the company increasingly uses AI for its own development, so models may advance faster than it can test and secure them. And it admits that in the past it did not recognise a significant acceleration in development until it had already happened. The second: internal assessments of future models indicate that Anthropic may "in the near future" reach the threshold that its own Responsible Scaling Policy describes as "dramatically accelerated progress attributable to AI".

The document immediately reassures. None of the current models has been determined to have crossed that threshold, and the data is still being analysed. The company writes that it does not expect to be able to safely let Claude design and build new generations of models entirely on its own without "meaningful human oversight or understanding", and that it is considering policy changes to keep humans in control.

This is where you have to stop and read who is writing and why. An IPO prospectus is both a mandatory risk disclosure and marketing material ahead of a share sale. A company telling investors it's close to AI that builds itself is also sending another message: that it's ahead of everyone else. Is this a warning, or the most expensive advert of the year?

On the other hand, Claude's progress this year in programming, maths and cybersecurity is visible even without a prospectus. If the very company building the system says it's approaching the line, the question is no longer technological. Who will guard that line - the shareholders who are only now coming in, or the lawmakers in Washington and Brussels?