yen-av/olmo-3-7b-butterfly-refusal
7B
•
Updated
•
13
Avery Yen
yen-av
·
AI & ML interests
None yet
Recent Activity
updated
a model
4 days ago
yen-av/olmo-3-7b-butterfly-refusal
published
a model
5 days ago
yen-av/olmo-3-7b-butterfly-refusal
published
a dataset
about 1 month ago
yen-av/gemma3-reasoning-sft
Organizations
updated
a
model
4 days ago
published
a
model
5 days ago
published
a
dataset
about 1 month ago
updated
a
dataset
about 1 month ago
reacted to
codelion's
post with 🔥
about 1 month ago
Post
6070
Introducing Dhara-70M: A diffusion language model that achieves 3.8x higher throughput than autoregressive models!
Key findings from our research on optimal architectures for small language models:
→ Depth beats width: 32 layers outperforms 12 layers at the same parameter count
→ Best-in-class factuality: 47.5% on TruthfulQA
→ 10x training efficiency using WSD (Warmup-Stable-Decay) conversion
→ Canon layers add only 0.13% parameters but improve reasoning
We trained on 1B tokens using the optimal 50-30-20 dataset mix (PDFs + filtered web + educational content), then converted to diffusion with just 100M additional tokens.
Blog: https://huggingface.co/blog/codelion/optimal-model-architecture
Model: codelion/dhara-70m
Key findings from our research on optimal architectures for small language models:
→ Depth beats width: 32 layers outperforms 12 layers at the same parameter count
→ Best-in-class factuality: 47.5% on TruthfulQA
→ 10x training efficiency using WSD (Warmup-Stable-Decay) conversion
→ Canon layers add only 0.13% parameters but improve reasoning
We trained on 1B tokens using the optimal 50-30-20 dataset mix (PDFs + filtered web + educational content), then converted to diffusion with just 100M additional tokens.
Blog: https://huggingface.co/blog/codelion/optimal-model-architecture
Model: codelion/dhara-70m
- 1 reply
upvoted
a
paper
about 2 months ago
updated
a
dataset
2 months ago
published
a
dataset
2 months ago
updated
a
model
3 months ago
published
a
model
3 months ago
updated
a
model
4 months ago
published
a
model
4 months ago
updated
a
model
4 months ago
published
a
model
4 months ago