How is Qwen3's inv_freq computed from scratch?

#13

by ghostplant - opened Jul 24

Jul 24

The cos and sin values are computed based on inv_freq and position_ids, however, the inv_freq seems changed after rope_theta's changing. Is there a forma to present how to compute inv_freq from scratch?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment