Share research related to AI & machine learning security

Research

“Retrieval-augmented generation (RAG) has been shown vulnerable to retrieval corruption attacks: an attacker can inject malicious passages into retrieval results to induce inaccurate responses. In this paper, we propose RobustRAG as the first defense framework against retrieval corruption attacks. The key insight of RobustRAG is an isolate-then-aggregate strategy: we get LLM responses from each passage in isolation and then securely aggregate these isolated responses. To instantiate RobustRAG, we design keyword-based and decoding-based algorithms for securely aggregating unstructured text responses. Notably, RobustRAG can achieve certifiable robustness: we can formally prove and certify that, for certain queries, RobustRAG can always return accurate responses, even when the attacker has full knowledge of our defense and can arbitrarily inject a small number of malicious passages. We evaluate RobustRAG on open-domain QA and long-form text generation datasets and demonstrate its effectiveness and generalizability across various tasks and datasets.”

"Certifiably Robust RAG against Retrieval Corruption" by Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, Prateek Mittal

Welcome to the MLSecOps Community, the premier hub for educational information about building security into AI and machine learning life cycles end-to-end. We look forward to meeting you!

MLSecOps Community

Job Title

Login to the MLSecOps Community community to connect with others, attend community events, and more!

Login or Sign Up for even more

To unlock all parts of the community and get the best experience, complete your profile.