In a recent discovery by researchers at Wiz, a critical vulnerability was found in the Replicate AI platform that had the potential to allow attackers to execute a malicious AI model within the platform for a cross-tenant attack. This could potentially lead to unauthorized access to the private AI models of customers, putting proprietary knowledge and sensitive data at risk of exposure.
The flaw was uncovered as part of a series of partnerships with AI-as-a-service providers to investigate the security of their platforms. This discovery sheds light on the challenges of maintaining tenant separation across AI-as-a-service solutions, especially in environments that involve running AI models from untrusted sources.
According to Shir Tamari and Sagi Tzadik of Wiz, the exploitation of this vulnerability could have granted unauthorized access to the AI prompts and results of all Replicate platform customers, potentially allowing malicious actors to alter those results. This poses a serious threat to the integrity of AI-driven outputs and the decision-making processes of these models.
Ami Luttwak, Wiz CTO and co-founder, emphasized the importance of verifying the origin of AI models and scanning their content for malicious payloads. He noted that the exploitation of vulnerabilities like this highlights the risks associated with running AI models from untrusted sources in cloud environments.
The researchers at Wiz responsibly disclosed the vulnerability to Replicate in January 2023, prompting the company to promptly mitigate the flaw to ensure no customer data was compromised. This swift action prevented any further damage, and customers are not required to take any additional steps at this time.
Exploiting the flaw involved achieving remote code execution on the Replicate platform by creating a malicious container in the Cog format, a proprietary format used to containerize models on Replicate. By uploading the container to the platform and executing code with root privileges, the researchers were able to investigate the environment, move laterally, and conduct a cross-tenant attack that allowed them to query other models and modify their outputs.
The ability to alter prompts and responses of AI models presents a significant risk to the functionality of AI applications, potentially compromising the accuracy and reliability of automated decisions. This underscores the need for new forms of mitigation to address the security implications of malicious AI models.
Luttwak suggested that using secure formats like safetensors for production workloads can reduce the attack surface and prevent attackers from taking over the AI model instance. Security teams are advised to monitor for unsafe models and transition to safetensors or similar secure formats to enhance their defenses against potential threats.
In addition, cloud providers running customer models in a shared environment should enforce tenant-isolation practices to prevent attackers who manage to execute malicious models from accessing other customers’ data or the service itself. By implementing these measures, organizations can better protect their AI models and mitigate the risks associated with vulnerabilities like the one found in the Replicate platform.