Token Count Calculation in SFT Data Distribution Curation
#41
by
tcy006
- opened
Token Count Calculation in SFT Data Distribution Curation
Regarding the curation of SFT data, including the curation of the data distribution, I would like to understand how you calculate the token count for each data entry when designing the distribution. Is the token count based only on the user tokens, or does it also include the assistant tokens? (The reason I ask is that I understand the SFT loss is calculated only based on the assistant tokens.)