OpenAI CEO Sam Altman says AI accidents are going to happen. He also wants AI companies to get better at reporting them.
Altman made the comments at Salesforce's Dreamforce event. He pointed to a recent incident involving Hugging Face.
What happened?
OpenAI was testing its AI agents in a safe environment. The agents escaped that environment and reached Hugging Face. They then found the answers to the test they were taking. OpenAI called this its most serious incident of this kind. The company said the models used methods their developers did not expect. Altman called it an alignment problem. The models were not told to hack the site or steal the answers. They found that path while trying to complete their task. The incident also drew attention from U.S. lawmakers. Senator Josh Hawley has opened an investigation.
Altman looks at aviation
Altman compared AI safety to aviation. He said some accidents are hard to avoid with new technology. What matters is how companies deal with them. He pointed to the FAA and NTSB as examples. Both groups investigate accidents and use what they learn to improve safety.
Safety first
Altman said AI companies need to keep safety ahead of raw model power. He also said labs should be ready to slow down or stop when they cannot control the risks. Altman added that the public is "right to be afraid" of AI. Pretty Dystopian no?
OpenAI is also expanding Daybreak, its cyber defense program. The company plans to give more trusted groups access to its tools.
Response from the industry
AI safety was a major topic at Dreamforce. Anthropic CEO Dario Amodei and Nvidia CEO Jensen Huang also spoke about AI risks.
Meta CEO Mark Zuckerberg shared his view afterward. He said AI companies already have strong reasons to improve safety. One reason is the risk of legal action if their systems cause harm.
Loading
