yandex/wmt24-en-ru-rate
Preview
•
Updated
•
17
Ermakov Petr
ermakovpetr
AI & ML interests
LLM, Search, Diffusion
Recent Activity
published
a dataset
19 days ago
yandex/wmt24-en-ru-rate
liked
a model
2 months ago
yandex/YandexGPT-5-Lite-8B-instruct-GGUF
liked
a model
2 months ago
yandex/YandexGPT-5-Lite-8B-instruct
Organizations
published
a
dataset
19 days ago
liked
2
models
2 months ago
liked
2
datasets
2 months ago
upvoted
a
paper
5 months ago
liked
a
dataset
6 months ago
Огромное вам спасибо. Продолжение?
10
#5 opened 9 months ago
by
Makar7
liked
a
model
9 months ago
reacted to
artnitolog's
post with 🤝🤗🚀🔥❤️👍
9 months ago
Post
2619
Recently, we open-sourced YaFSDP, Yandex’s tool for efficient distributed training of LLMs.
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
• Move gradient division to the very end of the backward pass.
To learn more about how YaFSDP works, check out our latest blog post: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
• Move gradient division to the very end of the backward pass.
To learn more about how YaFSDP works, check out our latest blog post: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
upvoted
a
paper
12 months ago
liked
a
model
about 1 year ago
