A chatbot invented an intelligence report, and American jets nearly struck a Chinese ship

The military aircraft were already in the air when American officials realised what they were holding. The intelligence report driving an armed operation against a Chinese ship was not an analyst's mistake, nor a bad source. It was hallucinated by a chatbot. The operation was called off at the last moment, and with it a possible clash with China.

The details are more interesting than the conclusion. The report claimed the ship was carrying components for a nuclear weapons programme, and it circulated while the war with Iran was under way. Its origin lies with an analyst at Special Operations Command, who asked a chatbot to merge publicly available data with classified signals intelligence. The tool mixed up the ship's cargo manifest. Then the same analyst used it a second time - this time to give the error a tidy, official appearance. Packaged that way, the falsehood set off through the command channels.

The second step is the one to read twice. The problem is not only that the machine invented something. The problem is that the same machine then dressed the invention in a format that travels up a hierarchy without resistance. Nobody stops a document that looks like a real document halfway down the line.

The American military is racing to embed this technology in order to decide faster and keep its edge over China. The Pentagon openly says the gain lies in speeding up the decision chain, so commanders can react in time. The very speed that makes the tool attractive is also what lets a hallucination through without enough human checking. The price of the advantage and the price of the error are paid from the same purse.

"It is important that service members understand the uncertainty built into these models," said Jake Steckler, a researcher at GovAI and a former US Army officer. "But it is especially critical for any decision that could lead to the use of force - targeting, intelligence analysis, operational planning. Those decisions have life-and-death consequences."

Steckler is not asking for the tool to be thrown out. He says the incident should mean more safeguards, not withdrawal. "These tools can be useful in the right context and with the right protections. But putting speed of adoption ahead of everything else will most likely produce incidents that only strip service members of their trust in these systems - which in the end only slows adoption down."

Notice what is missing from the whole story: a mechanism that caught the problem. The report was not stopped by a check, but by a last-minute discovery, with aircraft already in flight. That is not a system that works - it is a system that got lucky this time.