d is theoretically sound and empirically effective. They demonstrate that the interpolated attention score (a measure of how much the model pays attention to different parts of the input) is much more stable than the extrapolated one, which means the model can adapt more easily to the interpolated position encodings.

The authors also show that the Position Interpolation method can extend the context window size up to 32768 tokens with minimal fine-tuning. They demonstrate that the extended models perform w