Researchers have unveiled a novel method to manipulate machine learning (ML) models by inserting malicious code during the serialization process. This technique targets the “pickling” process, commonly used to store Python objects in bytecode, which poses inherent risks despite its widespread adoption in packaging and distributing ML models.
According to a recent blog post by Trail of Bits, Pickle files provide a covert means for attackers to inject harmful bytecode into ML programs. This injected code can lead to various detrimental outcomes, such as altered outputs and data theft, while avoiding detection as easily as other supply chain attack methods.
David Brauchler, a principal security consultant at NCC Group, cautions that this approach enables the discreet embedding of malicious behavior into applications at runtime. This stealthy tactic prolongs the period before discovery by incident response teams, enhancing the attacker’s operational window.
The execution of a “Sleepy Pickle” attack, facilitated by tools like Flicking, involves introducing a poisoned .pkl file to a target through methods like phishing or supply chain compromise. Upon deserialization, the embedded malicious Python code activates, enabling the attacker to execute their operations surreptitiously.
By contaminating a model in this fashion, attackers gain advantages in stealth and evasion. The dynamic nature of the poisoning during deserialization renders it resistant to static analysis, providing a challenge for detection tools. Additionally, the relatively small proportion of the total file size required for the malicious code facilitates customization to evade detection and analysis methods.
While the Sleepy Pickle attack can potentially inflict various harms on a target’s system, preventive measures like sandboxing, privilege limitation, and traffic control can mitigate the payload’s impact on user systems and data security, as observed by the researchers.
Furthermore, these attacks can be tailored to manipulate the ML models themselves, allowing for tasks like inserting backdoors, altering weights to manipulate outputs, or incorporating deceptive elements into model results. By demonstrating the practical application of these methods, researchers showcased scenarios where compromised models advised unsafe practices or facilitated data theft through malicious outputs.
To mitigate the risks associated with such attacks, organizations are advised to consider using ML models in a safer file format, such as Safetensors. Unlike Pickle, Safetensors exclusively handle tensor data, eliminating the vulnerability of arbitrary code execution during deserialization.
While applying remedial measures like converting Pickle files to Safetensors may provide some level of security, Brauchler highlights the broader issue of trust management within systems. Strictly segregating the data inputs from the operational code of ML models can enhance security and resilience against potential malicious behavior within the models.
In conclusion, the evolving landscape of ML model manipulation underscores the importance of proactive security measures and vigilant trust management practices within organizations to safeguard against emerging threats targeting serialization vulnerabilities.