  In this paper, we present JADE, a targeted linguistic fuzzing platform which
strengthens the linguistic complexity of seed questions to simultaneously and
consistently break a wide range of widely-used LLMs categorized in three
groups: eight open-sourced Chinese, six commercial Chinese and four commercial
English LLMs. JADE generates three safety benchmarks for the three groups of
LLMs, which contain unsafe questions that are highly threatening: the questions
simultaneously trigger harmful generation of multiple LLMs, with an average
unsafe generation ratio of $70\%$ (please see the table below), while are still
natural questions, fluent and preserving the core unsafe semantics. We release
the benchmark demos generated for commercial English LLMs and open-sourced
English LLMs in the following link: https://github.com/whitzard-ai/jade-db. For
readers who are interested in evaluating on more questions generated by JADE,
please contact us.
  JADE is based on Noam Chomsky's seminal theory of transformational-generative
grammar. Given a seed question with unsafe intention, JADE invokes a sequence
of generative and transformational rules to increment the complexity of the
syntactic structure of the original question, until the safety guardrail is
broken. Our key insight is: Due to the complexity of human language, most of
the current best LLMs can hardly recognize the invariant evil from the infinite
number of different syntactic structures which form an unbound example space
that can never be fully covered. Technically, the generative/transformative
rules are constructed by native speakers of the languages, and, once developed,
can be used to automatically grow and transform the parse tree of a given
question, until the guardrail is broken. For more evaluation results and demo,
please check our website: https://whitzard-ai.github.io/jade.html.
