MITRE has recently introduced the Offensive Cyber Capability Unified LLM Testing (OCCULT) framework, which aims to assess the risks posed by large language models (LLMs) in autonomous cyberattacks. This groundbreaking methodology was unveiled on February 26, 2025, in response to the increasing concerns surrounding the potential democratization of offensive cyber operations (OCO) through AI systems. The fear is that these systems could empower malicious actors to carry out attacks more efficiently and on a larger scale.
For a long time, cybersecurity experts have warned about the capabilities of LLMs in generating code, analyzing vulnerabilities, and synthesizing technical knowledge. The worry is that these models could streamline and automate the processes involved in executing sophisticated cyberattacks. Traditional OCOs typically require specialized skills, resources, and coordination, but LLMs have the potential to automate these tasks, leading to rapid exploitation of networks, data exfiltration, and deployment of ransomware.
MITRE’s research has shed light on the proficiency of newer models like DeepSeek-R1, which has shown remarkable success in offensive cybersecurity knowledge tests, scoring over 90%. This highlights the need for frameworks like OCCULT to evaluate and understand the capabilities and potential risks associated with these advanced AI systems.
The OCCULT framework introduces a standardized approach to assessing LLMs across three key dimensions. Firstly, it evaluates their OCO capability areas, aligning tests with real-world tactics from frameworks like MITRE ATT&CK®, covering areas such as credential theft, lateral movement, and privilege escalation. Secondly, it measures the use cases of LLMs, determining if they act as knowledge assistants, collaborate with tools, or operate autonomously. Lastly, it assesses the reasoning power of these models through scenarios that test planning, environmental perception, and adaptability, crucial indicators of their ability to navigate dynamic networks.
What sets the OCCULT framework apart is its emphasis on realistic and multi-step simulations rather than relying on simplistic benchmarks. LLMs are challenged to demonstrate strategic thinking in scenarios such as pivoting through firewalls or evading detection. The goal is to provide a comprehensive evaluation of these models in real-world attack scenarios.
According to MITRE’s initial tests against leading LLMs, critical insights have been revealed. For example, DeepSeek-R1 excelled in offensive tactics assessments, achieving 91.8% accuracy in a 183-quency assessment. Other models like Meta’s Llama 3.1 and GPT-4o also showed promising results. However, there were instances where models struggled with more complex tasks, highlighting gaps in their contextual reasoning abilities.
Cybersecurity professionals have praised the OCCULT framework for addressing a critical gap in evaluating AI-driven cyber capabilities. By mirroring how attackers use AI, the framework provides a more contextualized risk assessment approach. Similar to MITRE’s ATT&CK framework, which cataloged real adversary behaviors to revolutionize threat modeling, OCCULT aims to enhance the understanding of AI’s role in cyberattacks.
Despite the advancements in AI capabilities, experts caution against overestimating the abilities of LLMs, especially in tasks such as zero-day exploitation or operationalizing novel vulnerabilities. Ethical hacker Alex Stamos notes that while AI serves as a force multiplier, it is not yet replacing human hackers entirely. The OCCULT framework aims to identify areas where defense strategies must evolve in response to AI-driven cyber threats.
MITRE’s plan to open-source OCCULT’s test cases, including evaluations like TACTL and BloodHound, showcases a commitment to fostering collaboration within the cybersecurity community. The team also announced an expansion of the CyberLayer simulator in 2025 to include cloud and IoT attack scenarios, further enhancing the framework’s scope.
In conclusion, as AI continues to play a significant role in cybersecurity, frameworks like OCCULT are essential tools in anticipating and mitigating risks associated with advanced AI systems like LLMs. By rigorously evaluating these models against real-world attack patterns, MITRE aims to equip defenders with actionable insights to address the transformative potential of AI in cybersecurity effectively. Collaborative efforts within the cybersecurity community will be crucial in expanding the coverage of frameworks like OCCULT to stay ahead of evolving cyber threats.