As the capabilities of the AI model continue to improve, debates within Silicon Valley regarding "researchers warning about risks while simultaneously advancing development" have intensified once again. Foreign media reports that one of the triggers for this round of discussions was when OpenAI's Chief Financial Officer, Sarah Friar, mentioned at a Goldman Sachs technology conference that large models can now be used to train smaller models, thereby reducing training costs.
Concerns Raised by Improvements in Model Capabilities
The article argues that such progress has drawn attention not only because of the reduction in costs, but also because it is seen by some researchers as an early sign of "recursive self-improvement." According to this understanding, the AI system may in the future accelerate the emergence of the next generation of systems, and the pace of model capability improvement could also speed up accordingly.
OpenAI and Anthropic's statements are causing a stir
Recently, Anthropic, the person in charge of science at Evan Hubinger, stated on X that he seriously believes that the probability of AI causing "human extinction-level consequences" within the next decade exceeds 10%. Subsequently, other employees from Anthropic and OpenAI also joined the discussion, and these remarks quickly sparked controversy in Silicon Valley.
The article mentions that this view is significantly different from the mainstream opinion on Wall Street. At the Goldman Sachs conference, investors were more concerned about the business opportunities brought about by improvements in model capabilities. The founder of Altimeter Capital directly denied the claim that AI would kill humanity, stating that such a judgment is unfounded.
Foreign media identifies three main reasons
Foreign media reports that the first reason is commercial incentives. Both OpenAI and Anthropic rely on more powerful models to expand product sales and revenue scales; therefore, even though some researchers are concerned about the risks, companies will continue to advance development as a whole.
The second reason is "competitive mentality." Some researchers believe that if their own institution slows down while other laboratories continue to advance, the outcome could be even worse. Especially in their view, if more powerful AI is first mastered by institutions that are "not cautious" enough, the risk is even higher.
Anthropic The person in charge of scalable supervision also publicly wrote on X this week that Samuel Marks. The reason developers continue to push forward is often the result of both commercial incentives and competitive pressures, and at the same time, they are concerned that other teams may use this technology in less secure ways.
Declining influence exacerbates anxiety.
The article also proposes a third explanation, which is the anxiety researchers feel about their declining influence. As the development of the most advanced models increasingly concentrates in a few laboratories, even those working within these companies may feel that their actual impact on the direction of technology is limited.
AI Start-up company Inworld Chief Executive Officer Kylan Gibbs states that on one hand, researchers are more aware of the risks, but on the other hand, they have little control over the outcomes. This gap between greater awareness and less control can amplify anxiety, which ultimately manifests in more public and intense risk warnings.
The article also mentions that AI faces similar challenges when recruiting. Many researchers come from academic backgrounds and are more receptive to a mission narrative that emphasizes "participation in security and alignment work," rather than a recruitment approach that focuses solely on profit and wealth. This also results in risk expressions within the industry often emerging in tandem with technological development.
