Agentic AI,
Artificial Intelligence & Machine Learning,
Governance & Risk Management
Find Evil! Contest Winners Show That Evidence Controls Matter More Than AI Speed
In a striking turn of events, an individual with a background solely in artificial intelligence and machine learning has emerged victorious in the SANS Institute’s highly competitive Find Evil! hackathon. This contest focused on the creation of autonomous agents capable of conducting thorough digital forensic investigations, while rigorously demonstrating the evidence supporting their conclusions. The victory highlights the significance of evidence controls over mere speed in artificial intelligence capabilities.
The hackathon drew an impressive number of participants, with 4,413 registrants over the span of 60 days. Participants were tasked with building the necessary controls, verification, and audit mechanisms to enhance the trustworthiness of autonomous forensic analysis. The SANS Institute initiated this contest to push back against the prevailing narrative that defenders are helpless against AI-powered attacks unless they use AI themselves. Rob T. Lee, the chief AI officer and research lead at SANS, expressed his frustration at the ongoing comparison of AI systems on opposite sides of the battle. He stated, “I’m kind of sick of everyone saying AI versus AI. Let’s go do something.” This sentiment underscored the need for practical solutions rather than theoretical debates.
Ultimately, out of 291 submitted projects, only 123 met the requisite technical and documentation standards to progress to the judging phase. A dedicated panel of 90 incident responders evaluated these submissions, performing a total of 1,775 assessments to gauge each agent’s accuracy, safeguards, audit trails, and self-correction capabilities. The stakes were high, with more than $22,000 in cash and prizes up for grabs for the top participants. The first, second, and third place winners would receive $10,000, $7,500, and $4,500, respectively.
The grand prize went to a creation called Mulder, a “fully autonomous forensic investigator” ingeniously developed by Caleb Evans. This innovative tool adopts a comprehensive five-stage process to organize an investigation, execute necessary tools, analyze results, challenge its own conclusions, and produce reports. In a notable demonstration, Mulder logged 773 tool calls while sifting through 11 systems and examining 120 gigabytes of evidence. The AI conducted its analysis adeptly, facilitated by a meticulously designed control mechanism. This system ensured that the agent’s access to evidence and tools remained supervised, allowing for thorough checks of its outputs and accurate recording of how conclusions were reached. Moreover, an evidence-reference validator was crucial in dismissing any conclusions that could not be substantiated with actual artifacts, thus preventing erroneous findings.
A judge who utilized Mulder on their own test disk image reported that the tool successfully identified 119 out of 143 artifacts of compromise, showcasing its remarkable accuracy and effectiveness. Heather Barnhart, a SANS fellow and judge at the Find Evil! contest, praised Mulder’s design, particularly its robust safeguards against “hallucinations” and the requirement for the tool to substantiate its conclusions. “The hallucination precautions and sources being required and challenged are fantastic. It compelled the tool to work harder before simply jumping to conclusions,” she noted.
In second place, the project TRUDI, developed by incident responder Trinity Harrison, made waves by effectively challenging its own initial assumptions when evidence did not corroborate the indicators presented in an incident briefing. Meanwhile, the third-place entry, Camel, crafted by a dedicated team led by Allister Beharry, utilized a unique sandboxed architecture to trace its findings back to the scripts and commands that generated them, further reinforcing the importance of evidence integrity in digital forensics.
Judges rigorously tested finalists against various challenges, including evidence-spoliation attempts, path traversal, command injections, and more, to assess the effectiveness of each project’s safeguards. Every finalist endured an additional verification process using evidence that had not been previously seen by the teams, ensuring a stringent evaluation standard. The SANS Institute noted that projects that documented their errors and testing limitations tended to outperform those that claimed to have achieved flawless accuracy, reinforcing the notion that transparency is crucial in forensic analysis. Almost every successful finalist maintained rigorous evidence protections through their architecture instead of relying solely on prompts to guide behavior.
Interestingly, Lee had initially anticipated the strongest entries would come from teams comprising both AI technicians and seasoned incident responders. To his surprise, the top two projects were the result of solo participants who had to familiarize themselves with the required knowledge independently. Evans, who won first place, entered the contest with expertise in AI engineering but no experience in cybersecurity. Harrison, the second-place winner, was still studying at the SANS Technology Institute and had incident response experience but limited exposure to AI and machine learning.
Hackathons present a unique opportunity for participants to tackle concrete problems, utilizing real tools under tight deadlines while receiving valuable feedback—experiences that can be challenging to achieve through self-directed learning. Lee articulated the essence of these contests by highlighting that participants, regardless of their starting point, simply need enough foundational knowledge to begin building their projects. “You just need to know enough about how to get the stuff going, and then you just start building from there,” he remarked.
In addition to recognizing the top three projects with cash prizes, SANS also acknowledged two additional finalists—FindEvil and Protocol SIFT++. All five projects will be made available as open source tools, and SANS revealed plans to integrate them into the SIFT Workstation, a significant stride forward in digital forensic innovation.