Researchers recently made a troubling discovery on the popular Hugging Face Hub, an online repository for datasets and pre-trained models used in machine learning (ML). Two ML models containing malicious code were found on the platform, raising concerns about the security of users who download and execute these models on their machines.
The malicious payload embedded in these models is designed to create a reverse shell that connects to a hardcoded IP address once executed. It has the ability to determine whether it is being run on a Windows, Linux, or macOS system and adjust its programming logic accordingly. This sophisticated attack highlights the risks associated with downloading and running unverified models from online repositories.
The Hugging Face Hub serves as a valuable resource for software developers and researchers, allowing them to access and collaborate on ML models that can be integrated into various applications. However, the recent discovery of malicious models underscores the importance of vigilance when using repositories like Hugging Face Hub.
The malicious models identified by researchers were stored in PyTorch format, utilizing the pickle module for serialization and deserialization. This method, while convenient, poses security risks as it allows for the execution of Python code during model deserialization. By storing the models in a non-standard format (7z instead of ZIP), the attackers were able to evade detection by tools like Picklescan, which are designed to identify suspicious pickle files.
The development of a defense evasion technique named “NullifAI” allowed the attackers to exploit the execution of serialized code within pickle files, enabling the execution of malicious code. This technique highlights the ongoing arms race between threat actors and security researchers in the realm of AI and ML security.
In response to this incident, the Hugging Face security team promptly removed the malicious models from the platform and enhanced the detection capabilities of Picklescan. Indicators of compromise have been published to help users identify and mitigate potential threats. It is alarming to note that these malicious models evaded detection for over eight months, underscoring the need for continuous monitoring and threat detection in online repositories.
The discovery of these malicious models also raises questions about the intentions of the individuals behind them. While some indicators suggest that researchers may have been responsible, the harmful nature of the payloads points to a malicious intent. The creation of reverse shells on users’ machines can lead to serious security breaches and unauthorized access.
Overall, this incident serves as a reminder of the evolving landscape of cybersecurity threats in the AI and ML domain. As researchers continue to innovate and push boundaries in these fields, it is essential to prioritize security and mitigate risks associated with the deployment of ML models. The Hugging Face Hub and other online repositories must remain vigilant against malicious actors seeking to exploit vulnerabilities in the ecosystem.