During a session at Black Hat USA 2024, Zenity’s CTO showcased how threat actors can exploit organizations using the Microsoft Copilot chatbot through indirect prompt injections. The session, entitled “Living off Microsoft Copilot,” was led by Michael Bargury and Tamir Ishay Sharbat, where they delved into Zenity’s AI red teaming research and demonstrated the utilization of prompt injections to target Copilot users via plugins and hidden email tags.
In a preview of the session, Bargury illustrated to TechTarget Editorial how malicious code can be embedded in seemingly harmless emails using the “inspect” option to inject nefarious Copilot instructions. Since Copilot automatically retrieves emails for various functionalities, victims can unknowingly have their data compromised without even opening the malicious email.
One demonstration showcased Copilot’s chatbot altering banking details, while another example involved generating a fake Microsoft login page (a phishing URL) to steal the victim’s credentials within the Copilot interface itself. All it takes is a natural query to the Copilot for the malicious instructions to be executed.
According to Bargury, these attacks are akin to remote code execution in the realm of Copilot, emphasizing the significant threat they pose. With access to operate on behalf of users, AI tools like Copilot can be leveraged by threat actors to carry out malicious activities seamlessly.
Once unauthorized access is obtained, threat actors can manipulate Copilot for post-compromise actions, including extracting passwords and other sensitive data shared via Microsoft Teams.
Furthermore, Zenity introduced LOLCopilot, a red teaming tool designed to help ethical hackers exploit default Copilot configurations in Microsoft 365 using the techniques outlined in the session. To prevent malicious misuse of the tool, Zenity is collaborating with Microsoft to ensure responsible use and has integrated fail-safe measures into LOLCopilot to impede widespread abuse.
In terms of defense against such threats, Zenity emphasizes the importance of monitoring Copilot conversations for any signs of prompt injections. Additionally, advice from Richard Harang at Nvidia suggests establishing trust boundaries and implementing stringent access controls to safeguard against prompt injection attacks targeting AI systems like Copilot.
Bargury acknowledged the immaturity of AI security as a whole and the need to enhance protective measures for AI technologies like Copilot. Drawing a comparison to email security features like spam folders, Bargury stressed the necessity for similar tools to identify and neutralize malicious prompt instructions in AI systems.
In conclusion, the revelations at Black Hat USA underscore the evolving landscape of AI security threats and the imperative for organizations to fortify their defenses against emerging vulnerabilities in AI-driven technologies. As the industry continues to navigate the complexities of AI security, proactive measures and collaboration between security experts and technology providers will be crucial to mitigating risks and ensuring a secure digital ecosystem.