Ethical Hacking News
Cisco has been experimenting with AI to generate security incident reports, but the results have been mixed. The company's efforts highlight the challenges and limitations of relying on AI for critical tasks such as incident response. While some aspects of AI-powered reporting have shown promise, others have fallen short due to errors and inconsistencies.
Cisco Systems experimented with using artificial intelligence (AI) to generate security incident reports. The results were plagued by inaccuracies and inconsistencies, highlighting challenges of relying on AI for critical tasks like incident response. Large language models (LLMs) used in the experiment made educated guesses, leading to errors such as: - Using different data sources
- Reaching different conclusions from the same data
- Generating documents with varying structure and formatting
Cisco developed techniques to mitigate these issues, including providing granular instructions and setting rules about style and format. The experiment showed that AI-generated reports can be of high quality if done correctly, but require careful consideration and quality control processes.
Cisco Systems, a leading provider of networking and telecommunications equipment, has been experimenting with artificial intelligence (AI) to generate security incident reports. The company's efforts have yielded mixed results, highlighting the challenges and limitations of relying on AI for critical tasks such as incident response.
In a recent blog post, Nate Pors, a senior incident commander at Cisco's Talos Incident Response team, shared the findings of the company's experimentation with AI-powered reporting. According to Pors, when using large language models (LLMs) to generate reports on tabletop security exercises, the results were often plagued by inaccuracies and inconsistencies.
The nature of LLMs, which are essentially advanced autocomplete systems that make educated guesses, is prone to errors. These errors can manifest in several ways, including:
* Using different data sources for each query, making it difficult to rely on an LLM for repeatable, standardized research outcomes.
* Reaching different conclusions from the same data, often resulting in duplicate or irrelevant recommendations.
* Generating documents with varying structure and formatting due to the token-by-token approach of LLMs, which can lead to issues with quality control and consistency.
Additionally, AI-powered reporting tools may discard critical information, leading to suboptimal reports. To mitigate these issues, Cisco developed several techniques, including providing granular, single-task instructions that focus on specific sections of the report, instructing LLMs to use designated sources, and setting rules about style and format.
Despite these efforts, the results were not entirely satisfactory. In a blind test of sample reports, the peer reviewer, professional editor, and management reviewer all commented positively on the quality of the reports while remaining unaware that they were AI-generated. However, the team also found that editing multiple reports within a single session could lead to cross-contamination of content.
The researchers concluded that starting anew with each report, re-entering prompts, was essential for maintaining consistency and accuracy. Furthermore, developing a spelling-grammar-checking prompt proved to be unsuitable for production use due to its inconsistent performance and tendency to hallucinate grammar issues.
In conclusion, while AI has shown promise in generating security incident reports, the results are far from perfect. Cisco's experiment highlights the need for caution and careful consideration when relying on AI-powered tools for critical tasks. Moreover, it underscores the importance of investing time and resources into developing robust guidelines, training, and quality control processes to ensure that AI-generated reports meet the required standards.
Related Information:
https://www.ethicalhackingnews.com/articles/Ciscos-AI-Powered-Security-Incidents-A-Mixed-Bag-of-Results-ehn.shtml
https://www.theregister.com/security/2026/05/22/cisco-used-ai-to-write-security-incident-reports-with-mixed-results/5244692
https://www.cisco.com/c/en/us/products/collateral/security/xdr/xdr-ai-empowers-soc-analysts-wp.html
Published: Fri May 22 01:47:40 2026 by llama3.2 3B Q4_K_M
