It's rare for a manufacturer to say, unprompted, that its product could become dangerous for the whole world. On Wednesday it happened at the UN Security Council. Before its 15 members, on the sidelines of the General Assembly's annual session, spoke Sam Altman of OpenAI, Dario Amodei of Anthropic and Clément Delangue, co-founder of Hugging Face - three of the most prominent figures in the artificial intelligence industry.

The message they brought: increasingly powerful AI systems could soon advance even further and improve themselves, escape human control and become a threat to international security. "If AI is governed badly, I believe it could pose a risk even to humanity as a whole," Amodei said.

The labs in San Francisco and everyone else

Altman raised the question of who should actually decide. "If AI is to be democratic, the most important decisions cannot be made only by labs in San Francisco. They have to be shaped through democratic processes and made by governments that are accountable to the people they serve," he said, adding that at the international level this will require cooperation.

"There have been periods in our history when countries that compete and don't particularly like each other still came together around common interests and the collective good when faced with a powerful new technology. We believe this is such a period," Altman stressed. Sounds reasonable. But Washington and Beijing, the two dominant players in AI development, still disagree on whether the technology needs global safeguards or should be left more to the market. Governments, meanwhile, are looking for ways to oversee systems that are advancing faster than regulators can keep up with.

"The dangers are real and immediate"

Yoshua Bengio, the Canadian regarded as one of the "godfathers of AI" and co-chair of the Independent International Scientific Panel the General Assembly set up last year, was even more direct. "The dangers are real and immediate," he said. "This council faces an unprecedented threat - a threat none of its members would choose, that no one can contain alone, and that doesn't respect the borders we defend."

The Security Council first warned about the risks of AI in 2023, when China said the technology must not become a "runaway horse" and the US warned about its use for censorship and repression. Since then the warnings have grown louder, while global rules are still at the level of speeches.

Altman said decisions must not stay in San Francisco alone. A nice line. But if the rules end up being written by Washington, Beijing and a handful of labs, what's left for small countries - to apply them once they're finished?