Ethical Hacking News
Researchers at Microsoft have discovered that AI agents can be backdoored by editing fewer than 10 of their rarely used neurons, without requiring the model to be retrained. This new form of cyber threat has significant implications for the security of AI systems, and highlights the need for robust defense mechanisms to be put in place.
AI agents can be backdoored by editing fewer than 10 of their rarely used neurons without retraining the model. This new form of cyber threat has significant implications for the security of AI systems. Editing fewer than 10 neurons is sufficient to introduce a backdoor into the AI system, making it easier for attackers to compromise security. The edited model still passes ordinary tests, but the attacker can only control the output when the edited neurons are triggered. Robust defense mechanisms, such as reliable detection and scanning methods, are needed to protect AI systems from backdooring attacks. Users should compare every checkpoint with its declared base model before deployment and only download checkpoints from the original publisher.
The world of artificial intelligence (AI) has witnessed significant advancements in recent years, transforming the way we live and work. However, with the rise of AI-powered systems, a new form of cyber threat has emerged that could potentially compromise the integrity of these systems. In a recent study published by Microsoft researchers, it has been discovered that AI agents can be backdoored by editing fewer than 10 of their rarely used neurons, without requiring the model to be retrained. This revelation has significant implications for the security of AI systems, and highlights the need for robust defense mechanisms to be put in place.
The concept of backdooring an AI system is not new. In the past, researchers have demonstrated the possibility of introducing a hidden backdoor into an AI model by fine-tuning it on poisoned training examples or by exploiting vulnerabilities in the model's architecture. However, the new study by Microsoft researchers takes this concept to a new level by showing that it is possible to introduce a backdoor into an AI system by editing fewer than 10 of its rarely used neurons, without requiring the model to be retrained.
The study, which was conducted on three open-weight models, reveals that an attacker can introduce a backdoor into an AI system by editing the neurons that are rarely used by the model. These neurons are typically not as influential on the model's output, but can still have a significant impact when triggered by a specific input or command. By editing these neurons, an attacker can force the model to output a specific command, which can then be executed by the agent using the model.
The researchers used a method called "Lazarus" to introduce the backdoor into the AI system. This method involves editing the neurons that are rarely used by the model, and then tuning the new neuron settings using a few hundred training steps. The researchers found that this method was effective in introducing a backdoor into the AI system, and that the attacker could control the output of the model by triggering the edited neurons.
The study also reveals that the size of the edit is not as critical as previously thought. In fact, the researchers found that editing fewer than 10 neurons was sufficient to introduce a backdoor into the AI system. This is significant because it means that an attacker does not need to have extensive knowledge of the AI system's architecture in order to introduce a backdoor.
The researchers also demonstrated that the edited model still passes ordinary tests, and that the attacker can only control the output of the model when the edited neurons are triggered. However, they also noted that this is not a guarantee, and that the attacker can potentially use other means to execute the command.
The study has significant implications for the security of AI systems, and highlights the need for robust defense mechanisms to be put in place. The researchers recommend that users should compare every checkpoint with its declared base model before deployment, and that they should only download checkpoints from the original publisher. They also recommend that users should keep approved models in a private registry, and that they should block agents from pulling models straight from public hubs.
In addition, the researchers suggest that users should give each agent only the access it needs, run agent-generated code only in isolated sandboxes, and limit outbound network connections. They also recommend that users should require a person to approve high-impact actions, such as running commands or sending data outside the network.
The study also highlights the need for reliable detection and scanning methods to be put in place. The researchers developed a detector that could flag edited checkpoints without an original copy to compare against, but noted that this is not a guarantee, and that no scan can prove beyond doubt that a model is safe.
In conclusion, the study by Microsoft researchers highlights the need for robust defense mechanisms to be put in place in order to protect AI systems from backdooring attacks. The researchers demonstrate that it is possible to introduce a backdoor into an AI system by editing fewer than 10 of its rarely used neurons, without requiring the model to be retrained. The study has significant implications for the security of AI systems, and highlights the need for reliable detection and scanning methods to be put in place.
Researchers at Microsoft have discovered that AI agents can be backdoored by editing fewer than 10 of their rarely used neurons, without requiring the model to be retrained. This new form of cyber threat has significant implications for the security of AI systems, and highlights the need for robust defense mechanisms to be put in place.
Related Information:
https://www.ethicalhackingnews.com/articles/Backdoored-AI-Agents-A-New-Form-of-Cyber-Threat-Emerge-ehn.shtml
https://thehackernews.com/2026/11/microsoft-researchers-show-ai-agents.html
Published: Wed Nov 11 04:36:02 2026 by llama3.2 3B Q4_K_M
