  Protecting the copyright of large language models (LLMs) has become crucial
due to their resource-intensive training and accompanying carefully designed
licenses. However, identifying the original base model of an LLM is challenging
due to potential parameter alterations through fine-tuning or continued
pretraining. In this study, we introduce HuRef, a human-readable fingerprint
for LLMs that uniquely identifies the base model without exposing model
parameters or interfering with training. We first observe that the vector
direction of LLM parameters remains stable after the model has converged during
pretraining, showing negligible perturbations through subsequent training
steps, including continued pretraining, supervised fine-tuning (SFT), and RLHF,
which makes it a sufficient condition to identify the base model. The necessity
is validated by continuing to train an LLM with an extra term to drive away the
model parameters' direction and the model becomes damaged. However, this
direction is vulnerable to simple attacks like dimension permutation or matrix
rotation, which significantly change it without affecting performance. To
address this, leveraging the Transformer structure, we systematically analyze
potential attacks and define three invariant terms that identify an LLM's base
model. We make these invariant terms human-readable by mapping them to a
Gaussian vector using a convolutional encoder and then converting it into a
natural image with StyleGAN2. Our method generates a dog image as an identity
fingerprint for an LLM, where the dog's appearance strongly indicates the LLM's
base model. Experimental results across various LLMs demonstrate the
effectiveness of our method, the generated dog image remains invariant to
different training steps, including SFT, RLHF, or even continued pretraining
with augmented vocabulary in a new language.
