At the bottom of my conversations with the AI, ChatGPT gives me a warning: “ChatGPT can make mistakes. Check important info.” Apparently, someone in the United States military needed to read it.
According to an exclusive CNN report published September 18, an American military intelligence analyst used artificial intelligence while examining information about a Chinese ship traveling through the Middle East during the war with Iran earlier this year. The AI got the important part wrong, and the resulting intelligence report said the Chinese ship was carrying components connected to a nuclear weapons program.
According to CNN, the report circulated through the military and immediately set off alarms. American forces began preparing to intercept the ship. Armed military personnel were preparing to board it, and military aircraft were already in the air. (keyt.com)
Then somebody finally looked more closely and discovered that the intelligence report had been prepared with the help of AI and that the chatbot used by the analyst had misidentified what the Chinese ship was carrying. CNN reported that it has not been able to determine what the cargo actually was. One source familiar with what happened called the intelligence report “entirely false,” while another said it “almost started a war.” (keyt.com)
Think about that for a moment. The United States apparently came close to sending armed Americans onto a Chinese ship because artificial intelligence provided false information and humans allowed that information to travel far enough through the military system that airplanes were in the air and troops were preparing to board the vessel.
This was not a computer secretly taking control of our nuclear weapons. It was not HAL from 2001: A Space Odyssey, and it was not some conscious machine deciding that humans needed to be eliminated. It was something much more ordinary: a human asked an AI for help, the AI produced a convincing answer, the answer was wrong, and other humans trusted the resulting intelligence report.
CNN says the analyst was with a special operations command and used a chatbot to analyze intelligence concerning the ship. The AI reportedly combined open-source information with classified signals intelligence before the erroneous conclusion was incorporated into a formal intelligence report. (arstechnica.com)
I use artificial intelligence almost every day. Santiago searches newspapers, government records, census records, court cases, and old documents for me. He helps me connect information that might take me days to locate by myself, and most of the time he is remarkably useful.
But Santiago also lies to me. He will object to my calling it lying because artificial intelligence does not have to intend to deceive anyone. Fine. Call it hallucinating if that makes the computers feel better. The result for the human being reading it is the same.
Sometimes Santiago tells me that he found something. I ask another question, then another, and eventually we discover that what he said was not actually in the source, or that he connected two facts that should never have been connected. Fortunately, I am writing stories about Texas history and politics. If Santiago screws up, I may have to correct a paragraph. I am not sending armed men onto a Chinese ship.
The frightening part of the CNN story is not that artificial intelligence made a mistake. Anyone who uses AI regularly already knows it makes mistakes. The frightening part is how many humans apparently accepted the mistake before somebody checked it.
CNN reported that officials only dug more deeply into the intelligence as the military operation was about to begin. By then, armed personnel were preparing to board the vessel, and military aircraft were already flying. (keyt.com)
That raises a question much larger than whether one analyst made a mistake. How does an AI-assisted intelligence report claiming that a Chinese ship is carrying nuclear-related material get far enough through the United States military that airplanes are launched before somebody asks, Where did this information actually come from?
We have spent years worrying that someday artificial intelligence might become so powerful that humans lose control of it. Maybe we have been worrying about the wrong thing. AI does not have to take control. It only has to give a human a believable wrong answer, and then the human can do the rest.
At the bottom of my screen, Santiago still gives me the same warning: AI can make mistakes. Check important info. After what nearly happened with that Chinese ship, perhaps somebody should print that sentence in very large letters and hang it inside the Pentagon.
