  Recent remarkable advancements in large language models (LLMs) have led to
their widespread adoption in various applications. A key feature of these
applications is the combination of LLMs with external content, where user
instructions and third-party content are combined to create prompts for LLM
processing. These applications, however, are vulnerable to indirect prompt
injection attacks, where malicious instructions embedded within external
content compromise LLM's output, causing their responses to deviate from user
expectations. Despite the discovery of this security issue, no comprehensive
analysis of indirect prompt injection attacks on different LLMs is available
due to the lack of a benchmark. Furthermore, no effective defense has been
proposed.
  In this work, we introduce the first benchmark, BIPIA, to measure the
robustness of various LLMs and defenses against indirect prompt injection
attacks. Our experiments reveal that LLMs with greater capabilities exhibit
more vulnerable to indirect prompt injection attacks for text tasks, resulting
in a higher ASR. We hypothesize that indirect prompt injection attacks are
mainly due to the LLMs' inability to distinguish between instructions and
external content. Based on this conjecture, we propose four black-box methods
based on prompt learning and a white-box defense methods based on fine-tuning
with adversarial training to enable LLMs to distinguish between instructions
and external content and ignore instructions in the external content. Our
experimental results show that our black-box defense methods can effectively
reduce ASR but cannot completely thwart indirect prompt injection attacks,
while our white-box defense method can reduce ASR to nearly zero with little
adverse impact on the LLM's performance on general tasks. We hope that our
benchmark and defenses can inspire future work in this important area.
