Experts are warning that companies using private instances of large language models (LLMs) to make their business data searchable through conversational interfaces may be at risk of data poisoning and potential data leakage if proper security controls are not in place. The recent disclosure of a cross-site request forgery (CSRF) flaw affecting applications based on the EmbedAI component by AI provider SamurAI highlights the potential dangers, according to Synopsys, an application security firm.
The vulnerability in the EmbedAI component could allow attackers to deceive users into uploading poisoned data into their language model, potentially compromising the security of the platform. Synopsys security researcher Mohammed Alshehri, who discovered the flaw, emphasizes the importance of implementing the same security controls for AI applications as those used for traditional web applications.
The rush to integrate AI into business processes poses unique risks, especially for companies giving LLMs and other generative AI applications access to large repositories of data. While only 4% of US companies have adopted AI in their operations, certain industries like information and professional services show higher adoption rates, according to a survey by the US Census Bureau.
Dan McInerney, lead AI threat researcher at Protect AI, points out that the vulnerabilities lie not in the AI models themselves but in the software components and tools used to develop AI applications and interfaces. Attackers have actively targeted vulnerabilities in popular AI frameworks like Ray, exploiting security issues to compromise systems.
Despite the belief that private instances of AI-powered LLMs and chatbots may be safer from exploitation, recent research by Protect AI revealed vulnerabilities that could still be exploited, including critical remote exploits and other security flaws. The discovery of a CSRF issue by Synopsys underscores the potential risks of data poisoning and system compromise through malicious attacks.
Companies utilizing AI systems need to be vigilant about security, as many platforms are built on open-source components with limited oversight. Recent incidents like the discovery of malicious code-execution models in the Hugging Face AI model repository emphasize the importance of thorough security reviews and testing.
To mitigate risks, companies should segment data and LLM instances based on employee access levels and implement strict controls to prevent unauthorized access. Regular updates to software assets and minimizing the components used in AI tools can also help enhance security and make exploitation more difficult.
In conclusion, the evolving landscape of AI technology presents both opportunities and challenges for businesses. By prioritizing security measures and implementing best practices, companies can harness the power of AI while safeguarding against potential risks of data poisoning and leakage.