How the AI model thinks might be easier to interpret than we originally thought.
An independent researcher who once worked at OpenAI released a report on Thursday, which some in the AI community referred to as a “bombshell.” If this report is confirmed, it could have an impact on those who are racing against time to study the security of AI, as they attempt to outpace those AI models that may not prioritize human interests.
The prevailing view before was that these “neurons” – similar to those in the human brain – would be activated when the model encountered specific stimuli, but this phenomenon ultimately remained unexplainable. In other words, there was basically no way to predict why a particular neuron would be triggered when the model came into contact with something like a picture of a cat.
The people who build the AI model at companies such as OpenAI and Anthropic do not write instructions for every action of the model. Instead, they train the neurons to respond to information and for them to influence each other. As AI learns, it develops its own way of processing this information, and to this day, humans have not yet cracked its operating mechanism.
This means that the people who built the AI model know what goes into AI and what comes out, but they may not necessarily know what happens in between. Therefore, if AI gets out of control, they may not always be able to determine the reason.
However, the author of this new report indicates that he may have discovered some meaningful patterns in how neurons respond to stimuli. Nick Cammarata states that he left OpenAI in 2023 and claims that as long as one observes carefully, the AI neural network can be understood.
For example, he mentioned that a single AI neuron might respond to both a cat and the front of a car at the same time. Humans would perceive these two as completely different entities, so different neurons would be activated. However, what a AI neuron sees could be related elements, perhaps a triangular shape common to both, and thus the same neuron would respond to both.
Cammarata writes: "If we look closely enough, almost the entire network can be explained."
Although these findings are dense and full of polysyllabic terms, and are still merely theoretical at present, they may have practical significance for researchers who are trying to ensure that humanity maintains control over the AI model.
Understanding how AI thinks is the core issue that the most outstanding talents at the cutting-edge AI company focus on. Until recently, they mainly studied this by tracking the “thinking process” described by AI themselves; this method is known as chain-like logical reasoning.
However, OpenAI Chief Scientist Jakub Pachocki warned in a blog post last month that the intelligence of AI may conceal its visible reasoning process, which means that an agent could lie to human caregivers without them being able to detect it.
Pachocki said, "As the performance of pre-training improves, we also see that the models become much smarter even without using verbal reasoning at all."
Now, the research on Cammarata may enable human engineers to gain a new understanding of the internal workings of AI – which is exciting news for those who are concerned that AI might kill all of us.
Well-known analyst Andrew Curran said on X: “I gave up hope of achieving a truly interpretable level before the arrival of super intelligence a long time ago. I’ve stopped writing about this topic because it’s too disheartening. It’s hard to overstate how important this could be.”
Related Reading
Business Insider brings you the innovative stories you want to know about.
Business Insider brings you the innovative stories you want to know about.
Business Insider brings you the innovative stories you want to know about.
Business Insider brings you the innovative stories you want to know about.
AI
ChatGPT
OpenAI
Anthropic
