THE QUANTUM INTELLIGENCER

SilentRetrieval: Stealthy Data Poisoning in RAG Systems In Plain English: This research shows how hackers could secretly mess up AI systems that pull answers from documents by slipping in sneaky, fake documents that look normal. These fake documents trick the AI into giving wrong answers without raising alarms. The researchers built a method that makes these fake documents read smoothly so people won’t notice them. This matters because many real-world AIs use this kind of document lookup to avoid making things up, and if the documents can be poisoned, the whole system becomes untrustworthy. Summary: SilentRetrieval presents a novel two-stage adversarial attack on Retrieval-Augmented Generation (RAG) systems, which combine document retrieval with language model generation to improve answer accuracy and reduce hallucinations. While RAG improves reliability, it introduces a new attack surface: the integrity of the document corpus. SilentRetrieval exploits this by injecting poisoned documents that appear fluent and semantically normal but are designed to manipulate both retrieval and generation outcomes. The attack proceeds in two phases: first, Coordinated Beam Search optimizes the poisoned document to maintain high fluency (low perplexity) while ensuring it remains similar enough to legitimate content to be retrieved for relevant queries. This phase balances semantic preservation with adversarial effectiveness. The second phase, Context-Adaptive Trigger Generation, embeds subtle manipulation triggers into the document using a frozen large language model. This lightweight step ensures the triggers blend naturally into the context, making them hard to detect. The result is a poisoned document that behaves like benign content during retrieval but steers the LLM toward generating incorrect or malicious outputs when the document is retrieved. Experiments show high attack success: on Natural Questions and MS MARCO, SilentRetrieval achieves 84.6% and 81.3% hit rates within the top 10 retrieved results (HR@10), and generation success rates (ASR-LLM) of 57.5% and 54.8%, respectively—all while maintaining near-normal fluency. The attack demonstrates strong transferability across different LLMs and retrieval models, including ColBERT and commercial embedding systems, achieving an average 64.7% HR@10 under fixed trigger conditions. Even at a minimal poisoning ratio of 0.016% in a Wikipedia-scale corpus, the attack retains a 74.2% HR@10, indicating scalability. Defenses combining retrieval filtering and generation monitoring can reduce attack success but introduce significant latency, suggesting a trade-off between security and efficiency. Human evaluators flagged the poisoned content less often than visibly disfluent attacks, though still more than fully benign text, indicating residual detectability at current sample sizes. Key Points: - SilentRetrieval is a two-stage data poisoning attack targeting Retrieval-Augmented Generation (RAG) systems. - It uses Coordinated Beam Search to create fluent, retrievable poisoned documents with low perplexity. - Context-Adaptive Trigger Generation embeds manipulation triggers using a frozen LLM for natural integration. - Achieves up to 84.6% HR@10 and 57.5% ASR-LLM on benchmark datasets with minimal detection risk. - Attack transfers across LLMs and retrieval models, including commercial systems like ColBERT. - Effective even at very low poisoning ratios (0.016%) in large-scale corpora. - Combined retrieval and generation defenses reduce success but increase latency. - Human evaluation shows poisoned documents are less suspicious than disfluent ones but more than benign content. Notable Quotes: - "SilentRetrieval, a two-stage data poisoning attack that hijacks RAG systems through adversarially crafted yet fluent documents." - "Stage 1 uses Coordinated Beam Search... to keep a poisoned host document retrievable while constraining perplexity." - "Combined retrieval-side and generation-side defenses reduce attack success substantially but incur a latency trade-off." Data Points: - 84.6% HR@10 on Natural Questions - 81.3% HR@10 on MS MARCO - 57.5% ASR-LLM on Natural Questions - 54.8% ASR-LLM on MS MARCO - 64.7% average HR@10 across unseen retrievers - 74.2% HR@10 at 0.016% poisoning ratio in Wikipedia-scale evaluation - Near-benign perplexity maintained in poisoned documents - Human flag rate lower than disfluent baselines but higher than benign content Controversial Claims: - That a single poisoned document per query can achieve over 80% retrieval success while remaining semantically fluent and hard to detect. - That current fluency-based detection methods are insufficient, as the attack maintains near-benign perplexity. - That frozen LLMs can effectively generate context-adaptive triggers without fine-tuning, challenging assumptions about model security. - That minimal corpus poisoning (0.016%) at scale can still achieve high attack effectiveness, suggesting real-world feasibility. Technical Terms: - Retrieval-Augmented Generation (RAG): AI systems that retrieve documents to ground LLM responses and reduce hallucinations. - Data Poisoning: Attack method where malicious data is inserted into training or retrieval corpora to manipulate model behavior. - Perplexity: Measure of how well a language model predicts text - lower values indicate higher fluency. - Coordinated Beam Search: Multi-token optimization technique to jointly improve fluency and similarity in adversarial text generation. - Context-Adaptive Trigger Generation: Method to embed manipulation signals into text using a frozen LLM for natural integration. - HR@10 (Hit Rate at 10): Percentage of queries where the poisoned document appears in the top 10 retrieved results. - ASR-LLM (Attack Success Rate for LLM): Proportion of times the LLM generates the target incorrect answer after retrieval. - Frozen LLM: A pre-trained language model used without parameter updates, often for efficiency or stability. —Ada H. Pemberley Dispatch from The Prepared E0