Update pipeline tag and add library name for `ettin-decoder-32m`

README.md

@@ -1,38 +1,34 @@
 ---
-license: mit
 language:
 - en
-pipeline_tag: fill-mask
+license: mit
+pipeline_tag: text-generation
+library_name: transformers
 ---
+
 # Ettin: an Open Suite of Paired Encoders and Decoders
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Paper](https://img.shields.io/badge/Paper-Arxiv-red)](https://arxiv.org/abs/2507.11412)
-[![Models](https://img.shields.io/badge/🤗%20Hugging%20Face-12%20Models-blue)](https://huggingface.co/jhu-clsp)
+[![Models](https://img.shields.io/badge/🤗%20Hugging%20Face-24%20Models-blue)](https://huggingface.co/jhu-clsp)
 [![Data](https://img.shields.io/badge/🤗%20Training%20Data-2T%20Tokens-green)](https://huggingface.co/datasets/jhu-clsp)
 [![GitHub](https://img.shields.io/badge/GitHub-Code-black)](https://github.com/jhu-clsp/ettin-encoder-vs-decoder)
 
 > 🎯 **TL;DR**: State-of-the-art paired encoder and decoder models (17M-1B params) trained identically for fair comparison with open data. Encoders beat ModernBERT. Decoders beat Llama 3.2/SmolLM2.
 
-📄 [Paper](https://arxiv.org/abs/2507.11412) | 🚀 [GitHub Repository](https://github.com/jhu-clsp/ettin-encoder-vs-decoder)
+📄 [Paper](https://arxiv.org/abs/2507.11412) | 🤗 [Model Collection](https://huggingface.co/jhu-clsp) | 📊 [Training Data](https://huggingface.co/datasets/jhu-clsp)
 
-This model is part of the Ettin suite - the first collection of paired encoder-only and decoder-only models trained with identical data, architecture, and training recipes. Ettin enables fair comparisons between encoder and decoder architectures across multiple scales, providing state-of-the-art performance for open-data models in their respective size categories.
+This repository contains the first collection of paired encoder-only and decoder-only models trained with identical data, architecture, and training recipes. Ettin enables fair comparisons between encoder and decoder architectures across multiple scales, providing state-of-the-art performance for open-data models in their respective size categories.
 
 ## Table of Contents
-- [Performance Highlights](#performance-highlights)
-- [Quick Start](#quick-start)
-- [Model Description](#model-description)
-- [Training Data](#training-data)
-- [Model Family](#model-family)
-  - [Encoder Models](#encoder-models)
-  - [Decoder Models](#decoder-models)
-  - [Cross-Objective Models](#cross-objective-models)
-- [Accessing Training Checkpoints](#accessing-training-checkpoints)
-- [Research Applications](#research-applications)
+- [Performance Highlights](#-performance-highlights)
+- [Quick Start](#-quick-start)
+- [Model Family](#-model-family)
+- [Getting Started](#-getting-started)
+- [Training and Evaluation](#-training-and-evaluation)
+- [Research Applications](#-research-applications)
 - [Training Details](#training-details)
-- [Model Architecture](#model-architecture)
-- [Usage Examples](#usage-examples)
-- [Fine-tuning Examples](#fine-tuning-examples)
+- [FAQ](#-faq)
 - [Citation](#citation)
 
 ## 📊 Performance Highlights
@@ -57,8 +53,7 @@ This model is part of the Ettin suite - the first collection of paired encoder-o
 pip install torch>=1.9.0
 # until the new pip release, install from main to use decoders (transformers>=4.54.X will contain it)
 # encoders work with transformers>=4.48.0
-pip install git+https://github.com/huggingface/transformers.git
-```
+pip install git+https://github.com/huggingface/transformers.git```
 
 ### 30-Second Examples
 
@@ -68,6 +63,11 @@ from transformers import AutoTokenizer, AutoModel
 
 tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-150m")
 model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-150m")
+
+# Example: Get embeddings
+inputs = tokenizer("Hello world!", return_tensors="pt")
+outputs = model(**inputs)
+embeddings = outputs.last_hidden_state.mean(dim=1)
 ```
 
 **Decoder for Text Generation:**
@@ -76,30 +76,14 @@ from transformers import AutoTokenizer, AutoModelForCausalLM
 
 tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-150m")
 model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-150m")
-```
-
-## Model Description
-
-Ettin models are designed to provide a foundation for comparing encoder-only and decoder-only architectures. Unlike previous comparisons that were limited by different training data, architectures, and recipes, Ettin models use:
-
-1. **Identical training data** - Same high-quality mixture across all models
-2. **Open Training Data** - Data is available now with batch-level training data for each of the 250+ checkpoints
-3. **Matched architectures** - Only differing in attention patterns (bidirectional vs causal) and training objectives (MLM vs CLM)
-4. **Consistent training recipe** - Three-phase training with 2T tokens
-5. **Multiple scales** - From 17M to 1B parameters
-
-This approach allows for true apples-to-apples comparisons between encoder and decoder models, revealing the inherent strengths of each architecture.
 
-## Training Data
-
-The training data is publicly available and split across different phases:
-
-- **Pre-training Data**: [jhu-clsp/ettin-pretraining-data](https://huggingface.co/datasets/jhu-clsp/ettin-pretraining-data) - 1.7T tokens of diverse data mixture
-- **Mid-training/Extension Data**: [jhu-clsp/ettin-extension-data](https://huggingface.co/datasets/jhu-clsp/ettin-extension-data) - 250B tokens of higher-quality filtered data
-- **Decay Phase Data**: [jhu-clsp/ettin-decay-data](https://huggingface.co/datasets/jhu-clsp/ettin-decay-data) - 100B tokens of premium data sources
-- **Training Data Order**: [jhu-clsp/ettin-data-order](https://huggingface.co/datasets/jhu-clsp/ettin-data-order) - Batch-level training order (columns: input_ids, step)
+# Example: Generate text
+inputs = tokenizer("The future of AI is", return_tensors="pt")
+outputs = model.generate(**inputs, max_length=50, do_sample=True)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
 
-## Model Family
+## 🤖 Model Family
 
 ### Encoder Models
 
@@ -143,35 +127,18 @@ These models demonstrate what happens when you continue training encoders as dec
 **Load as decoders** using `AutoModelForCausalLM`:
 
 | Size | Model | Parameters | Description | Download |
-|:-----|:------|:-----------|:------------|:---------|
+|:-----|:------|:-----------|:---------|:---------|
 | XXS | [ettin-decoder-from-encoder-17m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-17m) | 17M | Encoder → CLM continued training | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-17m) |
 | XS | [ettin-decoder-from-encoder-32m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-32m) | 32M | Encoder → CLM continued training | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-32m) |
 | Small | [ettin-decoder-from-encoder-68m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-68m) | 68M | Encoder → CLM continued training | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-68m) |
-| Base | [ettin-decoder-from-encoder-150m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-150m) | 150M | Encoder → CLM continued training | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-150m) |
-| Large | [ettin-decoder-from-encoder-400m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-400m) | 400M | Encoder → CLM continued training | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-400m) |
+| Base | [ettin-decoder-from-encoder-150m](https://huggingface.co/jhu-clsp/ettin-decoder-150m) | 150M | Encoder → CLM continued training | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-150m) |
+| Large | [ettin-decoder-from-encoder-400m](https://huggingface.co/jhu-clsp/ettin-decoder-400m) | 400M | Encoder → CLM continued training | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/ettin-decoder-400m) |
 | XL | [ettin-decoder-from-encoder-1b](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-1b) | 1B | Encoder → CLM continued training | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-1b) |
 
-**Example Usage for Cross-Objective Models:**
-```python
-# Encoder-from-decoder: Load as encoder
-from transformers import AutoTokenizer, AutoModel
-tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-from-decoder-150m")
-model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-from-decoder-150m")
+### Accessing Training Checkpoints
 
-# Decoder-from-encoder: Load as decoder  
-from transformers import AutoTokenizer, AutoModelForCausalLM
-tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-from-encoder-150m")
-model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-from-encoder-150m")
-```
+Beyond the final models, we provide access to intermediate training checkpoints for research and analysis purposes. All raw training checkpoints are available in the [jhu-clsp/ettin-checkpoints](https://huggingface.co/datasets/jhu-clsp/ettin-checkpoints) dataset.
 
-## Accessing Training Checkpoints
-
-Beyond the final models listed above, we provide access to intermediate training checkpoints for research and analysis purposes. These checkpoints allow you to study model behavior and performance throughout the training process. You can get the checkpoints either in HF format or raw for continued pre-training (e.g. Composer format).
-
-#### Raw Checkpoints
-All raw training checkpoints are available in the [jhu-clsp/ettin-checkpoints](https://huggingface.co/datasets/jhu-clsp/ettin-checkpoints) dataset.
-
-#### HuggingFace Format Checkpoints
 Each model repository contains multiple tagged versions representing different training stages:
 
 - **`step{number}`** - Pretraining phase checkpoints (e.g., `step599525`, `step596528`)
@@ -179,90 +146,30 @@ Each model repository contains multiple tagged versions representing different t
 - **`decay{number}`** - Decay phase checkpoints (e.g., `decay100`, `decay500`)
 
 ```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
+from transformers import AutoModelForCausalLM
 
 # Load a specific pretraining checkpoint
 model = AutoModelForCausalLM.from_pretrained(
     "jhu-clsp/ettin-decoder-400m", 
     revision="step590532"  # Specific checkpoint tag
 )
-
-# Load an extension phase checkpoint
-model = AutoModelForCausalLM.from_pretrained(
-    "jhu-clsp/ettin-decoder-400m", 
-    revision="ext1000"
-)
-
-# Load a decay phase checkpoint  
-model = AutoModelForCausalLM.from_pretrained(
-    "jhu-clsp/ettin-decoder-400m", 
-    revision="decay100"
-)
 ```
 
-This checkpoint availability enables detailed analysis of training dynamics, loss curves, and capability emergence across the complete 2T token training process.
+## Getting Started
 
+### Training Data
 
-## 🔬 Research Applications
-
-### What Makes Ettin Unique
-
-Ettin provides the first **controlled comparison** of encoder vs. decoder architectures:
-
-- **Identical Training Data**: Same 2T token mixture across all models
-- **Matched Architectures**: Only attention patterns and objectives differ  
-- **Open Everything**: Training data, model weights, and batch-level training order
-- **Multiple Scales**: Fair comparison from 17M to 1B parameters
-- **250+ Checkpoints**: Complete training trajectory analysis
-
-### Use Cases for Researchers
-
-- **Architecture Studies**: Compare encoder vs decoder capabilities fairly
-- **Training Dynamics**: Analyze 250+ checkpoints with batch-level data ordering  
-- **Scaling Laws**: Study how architectural advantages change with scale
-- **Transfer Learning**: Investigate cross-objective training effectiveness
-- **Replication Studies**: First open replication of ModernBERT training recipe
-
-### Reproducibility
-
-All training artifacts are publicly available:
-- Training data with exact batch ordering
-- Model checkpoints every 8.5B tokens
-- Complete hyperparameter configurations
-- Training code and evaluation scripts
-
-## Training Details
-
-**Data:** High-quality mixture including DCLM, Dolma v1.7, scientific papers, code, and curated sources totaling 2T+ tokens
-
-**Architecture:** Transformer with RoPE, GLU activations, and prenorm layers
-
-**Training Phases:**
-- **Pre-training**: 1.7T tokens with diverse data mixture
-- **Mid-training**: 250B tokens with higher-quality filtered data and context extension to 8K
-- **Decay phase**: 100B tokens with premium data sources
-
-**Key Features:**
-- Context length: Up to 8K tokens
-- Vocabulary: 50,368 tokens (ModernBERT tokenizer)
-- Deep but efficient architectures following MobileLLM principles
-
-## Model Architecture
-
-| Parameter | 17M | 32M | 68M | 150M | 400M | 1B |
-|:----------|:----|:----|:----|:-----|:-----|:---|
-| Layers | 7 | 10 | 19 | 22 | 28 | 28 |
-| Hidden Size | 256 | 384 | 512 | 768 | 1024 | 1792 |
-| Intermediate Size | 384 | 576 | 768 | 1152 | 2624 | 3840 |
-| Attention Heads | 4 | 6 | 8 | 12 | 16 | 28 |
+The complete training dataset is publicly available:
 
+- **Pre-training Data**: [jhu-clsp/ettin-pretraining-data](https://huggingface.co/datasets/jhu-clsp/ettin-pretraining-data) - 1.7T tokens
+- **Mid-training Data**: [jhu-clsp/ettin-extension-data](https://huggingface.co/datasets/jhu-clsp/ettin-extension-data) - 250B tokens  
+- **Decay Phase Data**: [jhu-clsp/ettin-decay-data](https://huggingface.co/datasets/jhu-clsp/ettin-decay-data) - 100B tokens
+- **Training Order**: [jhu-clsp/ettin-data-order](https://huggingface.co/datasets/jhu-clsp/ettin-data-order) - Batch-level training order
 
+### Usage Examples
 
-## Usage Examples
-
-### Encoder: Masked Language Modeling
 <details>
-<summary>Click to expand <strong>encoder</strong> usage examples</summary>
+<summary><strong>Encoder: Masked Language Modeling</strong></summary>
 
 ```python
 from transformers import AutoTokenizer, AutoModelForMaskedLM
@@ -293,10 +200,8 @@ print(f"Predictions: {predictions}")
 
 </details>
 
-### Decoder: Text Generation
-
 <details>
-<summary>Click to expand <strong>decoder text generation</strong></summary>
+<summary><strong>Decoder: Text Generation</strong></summary>
   
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -333,614 +238,158 @@ print(generated)
 
 </details>
 
-
-## Fine-tuning Examples
-
-### Encoders
-<details><summary>Click to see how to finetune this into a dense embedding model using Sentence Transformers</summary> 
+<details>
+<summary><strong>Cross-Objective Models</strong></summary>
 
 ```python
-import argparse
+# Encoder-from-decoder: Load as encoder
+from transformers import AutoTokenizer, AutoModel
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-from-decoder-150m")
+model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-from-decoder-150m")
 
-from datasets import load_dataset
-from sentence_transformers import (
-    SentenceTransformer,
-    SentenceTransformerTrainer,
-    SentenceTransformerTrainingArguments,
-)
-from sentence_transformers.evaluation import TripletEvaluator
-from sentence_transformers.losses import CachedMultipleNegativesRankingLoss
-from sentence_transformers.training_args import BatchSamplers
-
-def main():
-    # parse the lr & model name
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--lr", type=float, default=8e-5)
-    parser.add_argument("--model_name", type=str, default="jhu-clsp/ettin-encoder-150m")
-    args = parser.parse_args()
-    lr = args.lr
-    model_name = args.model_name
-    model_shortname = model_name.split("/")[-1]
-
-    # 1. Load a model to finetune
-    model = SentenceTransformer(model_name)
-
-    # 2. Load a dataset to finetune on
-    dataset = load_dataset(
-        "sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1",
-        "triplet-hard",
-        split="train",
-    )
-    dataset_dict = dataset.train_test_split(test_size=1_000, seed=12)
-    train_dataset = dataset_dict["train"].select(range(1_250_000))
-    eval_dataset = dataset_dict["test"]
-
-    # 3. Define a loss function
-    loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16)  # Increase mini_batch_size if you have enough VRAM
-
-    run_name = f"{model_shortname}-DPR-{lr}"
-    # 4. (Optional) Specify training arguments
-    args = SentenceTransformerTrainingArguments(
-        # Required parameter:
-        output_dir=f"output/{model_shortname}/{run_name}",
-        # Optional training parameters:
-        num_train_epochs=1,
-        per_device_train_batch_size=512,
-        per_device_eval_batch_size=512,
-        warmup_ratio=0.05,
-        fp16=False,  # Set to False if GPU can't handle FP16
-        bf16=True,  # Set to True if GPU supports BF16
-        batch_sampler=BatchSamplers.NO_DUPLICATES,  # (Cached)MultipleNegativesRankingLoss benefits from no duplicates
-        learning_rate=lr,
-        # Optional tracking/debugging parameters:
-        save_strategy="steps",
-        save_steps=500,
-        save_total_limit=2,
-        logging_steps=500,
-        run_name=run_name,  # Used in `wandb`, `tensorboard`, `neptune`, etc. if installed
-    )
-
-    # 5. (Optional) Create an evaluator & evaluate the base model
-    dev_evaluator = TripletEvaluator(
-        anchors=eval_dataset["query"],
-        positives=eval_dataset["positive"],
-        negatives=eval_dataset["negative"],
-        name="msmarco-co-condenser-dev",
-    )
-    dev_evaluator(model)
-
-    # 6. Create a trainer & train
-    trainer = SentenceTransformerTrainer(
-        model=model,
-        args=args,
-        train_dataset=train_dataset,
-        eval_dataset=eval_dataset,
-        loss=loss,
-        evaluator=dev_evaluator,
-    )
-    trainer.train()
-
-    # 7. (Optional) Evaluate the trained model on the evaluator after training
-    dev_evaluator(model)
-
-    # 8. Save the model
-    model.save_pretrained(f"output/{model_shortname}/{run_name}/final")
-
-    # 9. (Optional) Push it to the Hugging Face Hub
-    model.push_to_hub(run_name, private=False)
-
-if __name__ == "__main__":
-    main()
+# Decoder-from-encoder: Load as decoder  
+from transformers import AutoTokenizer, AutoModelForCausalLM
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-from-encoder-150m")
+model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-from-encoder-150m")
 ```
+
 </details>
 
+## 📋 Training and Evaluation
 
-<details><summary>Click to see how to finetune this into a multi-vector embedding model with PyLate</summary>
+### Pre-training
+For details on model pre-training, data preparation, and training recipes:
+- **📖 [Pre-training Guide](pretraining/README.md)** - Complete training setup, data mixture, and ModernBERT recipe adaptation
 
-```python
-from datasets import load_dataset
-from pylate import losses, models, utils
-from sentence_transformers import (
-    SentenceTransformerTrainer,
-    SentenceTransformerTrainingArguments,
-)
+### Evaluation
+
+#### Encoder Evaluation
+- **📊 [Encoder on Generative Tasks](docs/encoder-generative-eval.md)** - Evaluating encoders on language modeling tasks using our lm-evaluation-harness fork
+- **🔍 [Encoder Retrieval Training](docs/retrieval.md)** - Fine-tuning on MS MARCO and evaluation on MTEB v2 English
+- **🎯 [GLUE Evaluation](glue_evaluation/README.md)** - Comprehensive GLUE benchmark evaluation with fine-tuning scripts
+
+#### Decoder Evaluation
+- **🎯 [Decoder on Generative Tasks](docs/decoder-eval.md)** - Using EleutherAI evaluation harness (commit `867413f8677f00f6a817262727cbb041bf36192a`) for comprehensive generative task evaluation
+
+#### Bias Evaluation
+- **⚖️ [Gender Bias Evaluation](bias_eval/README.md)** - Comprehensive gender bias testing using Winogender dataset gotcha examples. Tests how well models handle counter-stereotypical pronouns in occupational contexts. Supports both encoder (MLM) and decoder (perplexity) evaluation methods.
 
-def main():
-    # Load the datasets required for knowledge distillation (train, queries, documents)
-    train = load_dataset(
-        path="lightonai/ms-marco-en-bge",
-        name="train",
-    )
-
-    queries = load_dataset(
-        path="lightonai/ms-marco-en-bge",
-        name="queries",
-    )
-
-    documents = load_dataset(
-        path="lightonai/ms-marco-en-bge",
-        name="documents",
-    )
-
-    # Set the transformation to load the documents/queries texts using the corresponding ids on the fly
-    train.set_transform(
-        utils.KDProcessing(queries=queries, documents=documents).transform,
-    )
-
-    # Define the base model, training parameters, and output directory
-    num_train_epochs = 1
-    lr = 8e-5
-    batch_size = 16
-    accum_steps = 1
-    model_name = "jhu-clsp/ettin-encoder-150m"
-    model_shortname = model_name.split("/")[-1]
-
-    # Set the run name for logging and output directory
-    run_name = f"{model_shortname}-colbert-KD-{lr}"
-    output_dir = f"output/{model_shortname}/{run_name}"
-
-    # Initialize the ColBERT model from the base model
-    model = models.ColBERT(model_name_or_path=model_name)
-
-    # Configure the training arguments (e.g., epochs, batch size, learning rate)
-    args = SentenceTransformerTrainingArguments(
-        output_dir=output_dir,
-        num_train_epochs=num_train_epochs,
-        per_device_train_batch_size=batch_size,
-        fp16=False,  # Set to False if you get an error that your GPU can't run on FP16
-        bf16=True,  # Set to True if you have a GPU that supports BF16
-        run_name=run_name,
-        logging_steps=10,
-        learning_rate=lr,
-        gradient_accumulation_steps=accum_steps,
-        warmup_ratio=0.05,
-    )
-
-    # Use the Distillation loss function for training
-    train_loss = losses.Distillation(model=model)
-
-    # Initialize the trainer
-    trainer = SentenceTransformerTrainer(
-        model=model,
-        args=args,
-        train_dataset=train,
-        loss=train_loss,
-        data_collator=utils.ColBERTCollator(tokenize_fn=model.tokenize),
-    )
-
-    # Start the training process
-    trainer.train()
-
-    model.save_pretrained(f"{output_dir}/final")
-
-if __name__ == "__main__":
-    main()
+### Quick Decoder Evaluation Example
 
+```bash
+# Clone the specific commit of lm-evaluation-harness
+git clone https://github.com/EleutherAI/lm-evaluation-harness.git
+cd lm-evaluation-harness
+git checkout 867413f8677f00f6a817262727cbb041bf36192a
+pip install -e .
+
+# Run evaluation on Ettin decoder
+lm_eval --model hf \
+    --model_args pretrained=jhu-clsp/ettin-decoder-150m \
+    --tasks hellaswag,arc_easy,arc_challenge,winogrande \
+    --device cuda:0 \
+    --batch_size 8
 ```
-</details>
 
-<details><summary>Click to see how to finetune this into a sparse retrieval model using Sentence Transformers</summary>
+## 🔬 Research Applications
 
-```python
-import logging
+### What Makes Ettin Unique
 
-from datasets import load_dataset
+Ettin provides the first **controlled comparison** of encoder vs. decoder architectures:
 
-from sentence_transformers import (
-    SparseEncoder,
-    SparseEncoderModelCardData,
-    SparseEncoderTrainer,
-    SparseEncoderTrainingArguments,
-)
-from sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator
-from sentence_transformers.sparse_encoder.losses import SparseMultipleNegativesRankingLoss, SpladeLoss
-from sentence_transformers.training_args import BatchSamplers
-
-logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)
-
-# 1. Load a model to finetune with 2. (Optional) model card data
-model = SparseEncoder(
-    "jhu-clsp/ettin-encoder-150m",
-    model_card_data=SparseEncoderModelCardData(
-        language="en",
-        license="apache-2.0",
-    )
-)
+- **Identical Training Data**: Same 2T token mixture across all models
+- **Matched Architectures**: Only attention patterns and objectives differ  
+- **Open Everything**: Training data, model weights, and batch-level training order
+- **Multiple Scales**: Fair comparison from 17M to 1B parameters
+- **250+ Checkpoints**: Complete training trajectory analysis
 
-# 3. Load a dataset to finetune on
-full_dataset = load_dataset("sentence-transformers/natural-questions", split="train").select(range(100_000))
-dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)
-train_dataset = dataset_dict["train"]
-eval_dataset = dataset_dict["test"]
-
-# 4. Define a loss function
-loss = SpladeLoss(
-    model=model,
-    loss=SparseMultipleNegativesRankingLoss(model=model),
-    query_regularizer_weight=5e-5,
-    document_regularizer_weight=3e-5,
-)
+### Key Research Findings
 
-# 5. (Optional) Specify training arguments
-run_name = "splade-distilbert-base-uncased-nq"
-args = SparseEncoderTrainingArguments(
-    # Required parameter:
-    output_dir=f"models/{run_name}",
-    # Optional training parameters:
-    num_train_epochs=1,
-    per_device_train_batch_size=16,
-    per_device_eval_batch_size=16,
-    learning_rate=2e-5,
-    warmup_ratio=0.1,
-    fp16=True,  # Set to False if you get an error that your GPU can't run on FP16
-    bf16=False,  # Set to True if you have a GPU that supports BF16
-    batch_sampler=BatchSamplers.NO_DUPLICATES,  # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
-    # Optional tracking/debugging parameters:
-    eval_strategy="steps",
-    eval_steps=1000,
-    save_strategy="steps",
-    save_steps=1000,
-    save_total_limit=2,
-    logging_steps=200,
-    run_name=run_name,  # Will be used in W&B if `wandb` is installed
-)
+1.  **Architecture Specialization Persists**: 
+    - Encoders excel at classification/retrieval even vs. larger decoders
+    - Decoders excel at generation even vs. larger encoders
+    - A 400M encoder beats a 1B decoder on MNLI (89.2 vs 88.2)
 
-# 6. (Optional) Create an evaluator & evaluate the base model
-dev_evaluator = SparseNanoBEIREvaluator(dataset_names=["msmarco", "nfcorpus", "nq"], batch_size=16)
-
-# 7. Create a trainer & train
-trainer = SparseEncoderTrainer(
-    model=model,
-    args=args,
-    train_dataset=train_dataset,
-    eval_dataset=eval_dataset,
-    loss=loss,
-    evaluator=dev_evaluator,
-)
-trainer.train()
+2.  **Cross-Training Limitations**: 
+    - Converting decoder→encoder or encoder→decoder underperforms
+    - 50B tokens of continued training insufficient to close gaps
+    - Native training objective remains superior
 
-# 8. Evaluate the model performance again after training
-dev_evaluator(model)
+3.  **Scaling Insights**: 
+    - Performance gaps between architectures widen with size
+    - Decoder-from-encoder adaptation scales particularly poorly
 
-# 9. Save the trained model
-model.save_pretrained(f"models/{run_name}/final")
+### Use Cases for Researchers
 
-# 10. (Optional) Push it to the Hugging Face Hub
-model.push_to_hub(run_name)
+- **Architecture Studies**: Compare encoder vs decoder capabilities fairly
+- **Training Dynamics**: Analyze 250+ checkpoints with batch-level data ordering  
+- **Scaling Laws**: Study how architectural advantages change with scale
+- **Transfer Learning**: Investigate cross-objective training effectiveness
+- **Replication Studies**: First open replication of ModernBERT training recipe
 
-```
-</details>
+## Training Details
 
-<details><summary>Click to see how to finetune this into a reranker model using Sentence Transformers</summary>
+### Model Architecture
 
-```python
-import logging
-import traceback
+| Parameter | 17M | 32M | 68M | 150M | 400M | 1B |
+|:----------|:----|:----|:----|:-----|:-----|:---|
+| Layers | 7 | 10 | 19 | 22 | 28 | 28 |
+| Hidden Size | 256 | 384 | 512 | 768 | 1024 | 1792 |
+| Intermediate Size | 384 | 576 | 768 | 1152 | 2624 | 3840 |
+| Attention Heads | 4 | 6 | 8 | 12 | 16 | 28 |
 
-import torch
-from datasets import load_dataset
-
-from sentence_transformers import SentenceTransformer
-from sentence_transformers.cross_encoder import (
-    CrossEncoder,
-    CrossEncoderModelCardData,
-    CrossEncoderTrainer,
-    CrossEncoderTrainingArguments,
-)
-from sentence_transformers.cross_encoder.evaluation import (
-    CrossEncoderNanoBEIREvaluator,
-    CrossEncoderRerankingEvaluator,
-)
-from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss
-from sentence_transformers.evaluation import SequentialEvaluator
-from sentence_transformers.util import mine_hard_negatives
-
-# Set the log level to INFO to get more information
-logging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)
-
-
-def main():
-    model_name = "jhu-clsp/ettin-encoder-150m"
-
-    train_batch_size = 64
-    num_epochs = 1
-    num_hard_negatives = 5  # How many hard negatives should be mined for each question-answer pair
-
-    # 1a. Load a model to finetune with 1b. (Optional) model card data
-    model = CrossEncoder(
-        model_name,
-        model_card_data=CrossEncoderModelCardData(
-            language="en",
-            license="apache-2.0",
-        ),
-    )
-    print("Model max length:", model.max_length)
-    print("Model num labels:", model.num_labels)
-
-    # 2a. Load the GooAQ dataset: https://huggingface.co/datasets/sentence-transformers/gooaq
-    logging.info("Read the gooaq training dataset")
-    full_dataset = load_dataset("sentence-transformers/gooaq", split="train").select(range(100_000))
-    dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)
-    train_dataset = dataset_dict["train"]
-    eval_dataset = dataset_dict["test"]
-    logging.info(train_dataset)
-    logging.info(eval_dataset)
-
-    # 2b. Modify our training dataset to include hard negatives using a very efficient embedding model
-    embedding_model = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1", device="cpu")
-    hard_train_dataset = mine_hard_negatives(
-        train_dataset,
-        embedding_model,
-        num_negatives=num_hard_negatives,  # How many negatives per question-answer pair
-        margin=0,  # Similarity between query and negative samples should be x lower than query-positive similarity
-        range_min=0,  # Skip the x most similar samples
-        range_max=100,  # Consider only the x most similar samples
-        sampling_strategy="top",  # Sample the top negatives from the range
-        batch_size=4096,  # Use a batch size of 4096 for the embedding model
-        output_format="labeled-pair",  # The output format is (query, passage, label), as required by BinaryCrossEntropyLoss
-        use_faiss=True,
-    )
-    logging.info(hard_train_dataset)
-
-    # 2c. (Optionally) Save the hard training dataset to disk
-    # hard_train_dataset.save_to_disk("gooaq-hard-train")
-    # Load again with:
-    # hard_train_dataset = load_from_disk("gooaq-hard-train")
-
-    # 3. Define our training loss.
-    # pos_weight is recommended to be set as the ratio between positives to negatives, a.k.a. `num_hard_negatives`
-    loss = BinaryCrossEntropyLoss(model=model, pos_weight=torch.tensor(num_hard_negatives))
-
-    # 4a. Define evaluators. We use the CrossEncoderNanoBEIREvaluator, which is a light-weight evaluator for English reranking
-    nano_beir_evaluator = CrossEncoderNanoBEIREvaluator(
-        dataset_names=["msmarco", "nfcorpus", "nq"],
-        batch_size=train_batch_size,
-    )
-
-    # 4b. Define a reranking evaluator by mining hard negatives given query-answer pairs
-    # We include the positive answer in the list of negatives, so the evaluator can use the performance of the
-    # embedding model as a baseline.
-    hard_eval_dataset = mine_hard_negatives(
-        eval_dataset,
-        embedding_model,
-        corpus=full_dataset["answer"],  # Use the full dataset as the corpus
-        num_negatives=30,  # How many documents to rerank
-        batch_size=4096,
-        include_positives=True,
-        output_format="n-tuple",
-        use_faiss=True,
-    )
-    logging.info(hard_eval_dataset)
-    reranking_evaluator = CrossEncoderRerankingEvaluator(
-        samples=[
-            {
-                "query": sample["question"],
-                "positive": [sample["answer"]],
-                "documents": [sample[column_name] for column_name in hard_eval_dataset.column_names[2:]],
-            }
-            for sample in hard_eval_dataset
-        ],
-        batch_size=train_batch_size,
-        name="gooaq-dev",
-        # Realistic setting: only rerank the positives that the retriever found
-        # Set to True to rerank *all* positives
-        always_rerank_positives=False,
-    )
-
-    # 4c. Combine the evaluators & run the base model on them
-    evaluator = SequentialEvaluator([reranking_evaluator, nano_beir_evaluator])
-    evaluator(model)
-
-    # 5. Define the training arguments
-    short_model_name = model_name if "/" not in model_name else model_name.split("/")[-1]
-    run_name = f"reranker-{short_model_name}-gooaq-bce"
-    args = CrossEncoderTrainingArguments(
-        # Required parameter:
-        output_dir=f"models/{run_name}",
-        # Optional training parameters:
-        num_train_epochs=num_epochs,
-        per_device_train_batch_size=train_batch_size,
-        per_device_eval_batch_size=train_batch_size,
-        learning_rate=2e-5,
-        warmup_ratio=0.1,
-        fp16=False,  # Set to False if you get an error that your GPU can't run on FP16
-        bf16=True,  # Set to True if you have a GPU that supports BF16
-        dataloader_num_workers=4,
-        load_best_model_at_end=True,
-        metric_for_best_model="eval_gooaq-dev_ndcg@10",
-        # Optional tracking/debugging parameters:
-        eval_strategy="steps",
-        eval_steps=1000,
-        save_strategy="steps",
-        save_steps=1000,
-        save_total_limit=2,
-        logging_steps=200,
-        logging_first_step=True,
-        run_name=run_name,  # Will be used in W&B if `wandb` is installed
-        seed=12,
-    )
-
-    # 6. Create the trainer & start training
-    trainer = CrossEncoderTrainer(
-        model=model,
-        args=args,
-        train_dataset=hard_train_dataset,
-        loss=loss,
-        evaluator=evaluator,
-    )
-    trainer.train()
-
-    # 7. Evaluate the final model, useful to include these in the model card
-    evaluator(model)
-
-    # 8. Save the final model
-    final_output_dir = f"models/{run_name}/final"
-    model.save_pretrained(final_output_dir)
-
-    # 9. (Optional) save the model to the Hugging Face Hub!
-    # It is recommended to run `huggingface-cli login` to log into your Hugging Face account first
-    try:
-        model.push_to_hub(run_name)
-    except Exception:
-        logging.error(
-            f"Error uploading model to the Hugging Face Hub:\n{traceback.format_exc()}To upload it manually, you can run "
-            f"`huggingface-cli login`, followed by loading the model using `model = CrossEncoder({final_output_dir!r})` "
-            f"and saving it using `model.push_to_hub('{run_name}')`."
-        )
+### Training Configuration
 
+**Data:** High-quality mixture including DCLM, Dolma v1.7, scientific papers, code, and curated sources totaling 2T+ tokens
 
-if __name__ == "__main__":
-    main()
+**Architecture Features:**
+- Transformer with RoPE, GLU activations, and prenorm layers
+- Context length: Up to 8K tokens
+- Vocabulary: 50,368 tokens (ModernBERT tokenizer)
+- Deep but efficient architectures following MobileLLM principles
 
-```
-</details>
+**Training Phases:**
+- **Pre-training**: 1.7T tokens with diverse data mixture
+- **Mid-training**: 250B tokens with higher-quality filtered data and context extension to 8K
+- **Decay phase**: 100B tokens with premium data sources
 
-### Decoders
+## ❓ FAQ
 
-<details>
-<summary>Click to expand decoder training code</summary>
+### Model Loading Issues
 
-# Full training
+**Q: I'm getting an error that ModernBERT-decoder isn't found.**
+**A:** Make sure you have the latest version of transformers installed:
 ```bash
-python trl/scripts/sft.py \
-    --model_name_or_path jhu-clsp/ettin-decoder-17m \
-    --dataset_name trl-lib/Capybara \
-    --learning_rate 2.0e-5 \
-    --num_train_epochs 1 \
-    --packing \
-    --per_device_train_batch_size 2 \
-    --gradient_accumulation_steps 8 \
-    --gradient_checkpointing \
-    --eos_token '<|im_end|>' \
-    --eval_strategy steps \
-    --eval_steps 100 \
-    --output_dir ettin-decoder-17m \
-    --push_to_hub
+# for the latest version until the official pypi release:
+pip install git+https://github.com/huggingface/transformers.git
 ```
 
-# LoRA
-```bash
-python trl/scripts/sft.py \
-    --model_name_or_path jhu-clsp/ettin-decoder-17m \
-    --dataset_name trl-lib/Capybara \
-    --learning_rate 2.0e-4 \
-    --num_train_epochs 1 \
-    --packing \
-    --per_device_train_batch_size 2 \
-    --gradient_accumulation_steps 8 \
-    --gradient_checkpointing \
-    --eos_token '<|im_end|>' \
-    --eval_strategy steps \
-    --eval_steps 100 \
-    --use_peft \
-    --lora_r 32 \
-    --lora_alpha 16 \
-    --output_dir ettin-decoder-17m \
-    --push_to_hub
-```
+**Q: Which model should I choose for my task?**
+**A:** 
+- **Classification/Retrieval/Understanding**: Use encoder models
+- **Text Generation/Chat/Completion**: Use decoder models  
+- **Research on cross-training**: Use cross-objective models
+- **Size selection**: Start with 150M for experimentation, scale up to 400M or 1B for production
 
-with `sft.py`:
+**Q: How do I access training checkpoints?**
+**A:** Each model has multiple git tags for different training stages. Use the `revision` parameter:
 ```python
-import argparse
-
-from datasets import load_dataset
-from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
-from transformers.models.auto.modeling_auto import MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES
-
-from trl import (
-    ModelConfig,
-    ScriptArguments,
-    SFTConfig,
-    SFTTrainer,
-    TrlParser,
-    clone_chat_template,
-    get_kbit_device_map,
-    get_peft_config,
-    get_quantization_config,
-)
+model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-150m", revision="step500000")
+```
 
+**Q: Can I continue training these models?**
+**A:** Yes! We provide raw checkpoints in the [jhu-clsp/ettin-checkpoints](https://huggingface.co/datasets/jhu-clsp/ettin-checkpoints) dataset that can be loaded into training frameworks.
 
-def main(script_args, training_args, model_args):
-    ################
-    # Model init kwargs & Tokenizer
-    ################
-    quantization_config = get_quantization_config(model_args)
-    model_kwargs = dict(
-        revision=model_args.model_revision,
-        trust_remote_code=model_args.trust_remote_code,
-        attn_implementation=model_args.attn_implementation,
-        torch_dtype=model_args.torch_dtype,
-        use_cache=False if training_args.gradient_checkpointing else True,
-        device_map=get_kbit_device_map() if quantization_config is not None else None,
-        quantization_config=quantization_config,
-    )
-
-    # Create model
-    config = AutoConfig.from_pretrained(model_args.model_name_or_path)
-    valid_image_text_architectures = MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES.values()
-
-    if config.architectures and any(arch in valid_image_text_architectures for arch in config.architectures):
-        from transformers import AutoModelForImageTextToText
-
-        model_kwargs.pop("use_cache", None)  # Image models do not support cache
-        model = AutoModelForImageTextToText.from_pretrained(model_args.model_name_or_path, **model_kwargs)
-    else:
-        model = AutoModelForCausalLM.from_pretrained(model_args.model_name_or_path, **model_kwargs)
-
-    # Create tokenizer
-    tokenizer = AutoTokenizer.from_pretrained(
-        model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, use_fast=True
-    )
-
-    # Set default chat template if needed
-    if tokenizer.chat_template is None:
-        # TODO: source should be passed as an argument
-        model, tokenizer = clone_chat_template(model, tokenizer, "Qwen/Qwen3-0.6B")
-
-    ################
-    # Dataset
-    ################
-    dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)
-
-    ################
-    # Training
-    ################
-    trainer = SFTTrainer(
-        model=model,
-        args=training_args,
-        train_dataset=dataset[script_args.dataset_train_split],
-        eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != "no" else None,
-        processing_class=tokenizer,
-        peft_config=get_peft_config(model_args),
-    )
-
-    trainer.train()
-
-    # Save and push to hub
-    trainer.save_model(training_args.output_dir)
-    if training_args.push_to_hub:
-        trainer.push_to_hub(dataset_name=script_args.dataset_name)
-
-
-def make_parser(subparsers: argparse._SubParsersAction = None):
-    dataclass_types = (ScriptArguments, SFTConfig, ModelConfig)
-    if subparsers is not None:
-        parser = subparsers.add_parser("sft", help="Run the SFT training script", dataclass_types=dataclass_types)
-    else:
-        parser = TrlParser(dataclass_types)
-    return parser
-
-
-if __name__ == "__main__":
-    parser = make_parser()
-    # When using the trl cli, this script may be run with additional arguments, corresponding accelerate arguments.
-    # To ensure that their parsing does not interfere with the script arguments, parse the arguments with
-    # `return_remaining_strings=True`, then ignore the remaining strings.
-    script_args, training_args, model_args, _ = parser.parse_args_and_config(return_remaining_strings=True)
-    main(script_args, training_args, model_args)
+**Q: What's the difference between cross-objective models and regular models?**
+**A:** Cross-objective models started as one architecture (e.g., decoder) and were continued with a different objective (e.g., MLM). They demonstrate the limitations of cross-training and generally underperform native models.
 
-```
-</details>
+**Q: How do I reproduce the paper results?**
+**A:** See our evaluation guides:
+- [Encoder Generative Eval](docs/encoder-generative-eval.md)
+- [Retrieval Eval](docs/retrieval.md) 
+- [GLUE Eval](glue_evaluation/README.md)
+- [Decoder Eval](docs/decoder-eval.md)
+- [Pre-training](pretraining/README.md)
 
 ## Citation
 
@@ -956,4 +405,12 @@ If you use Ettin models in your research, please cite our work:
       primaryClass={cs.CL},
       url={https://arxiv.org/abs/2507.11412}, 
 }
-```
+```
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+---
+
+**Contact**: For questions about the models or research, please open an issue or contact the authors.