pro_bvv_en: 200M param frozen-embedding concept LM

📚 Paper (Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations) - 📚 Paper (Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate) - 💻 Code

Description

This is a conceptual English language model (200M parameters) trained from scratch with frozen, non-semantic token embeddings, demonstrating that transformer blocks can learn semantics even when the embedding layer contains no prior meaning.

Training details

• Trained on an English text corpus (~9B tokens) with 10% SFT data.
• Token embeddings are frozen (never trained) and initialized with non-semantic, visually-based vectors.
• All parameters except embeddings are trainable during pretraining.
• No expectation of SOTA, the goal is to demonstrate emergent learning capability.

Evaluation (main metrics)

Task	pro_bvv_en
MMLU	23.68% ± 0.17%
ARC-e	23.51% ± 0.71%
ARC-c	23.98% ± 1.74%
C-SENSE	19.54% ± 0.89%
SQUAD	9.61% ± 1.37%

---

⚠️ Limitations Research use only. Trained on a small subset. Quality, robustness, and reasoning are much lower than SOTA models. SFT was only lightly applied; not intended for real world use.

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@misc{bochkov2025emergentsemanticstokenembeddings,
      title={Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.04886},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.04886}, 
}

@misc{bochkov2025growingtransformersmodularcomposition,
      title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.07129},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.07129}, 
}

This work demonstrates that transformer blocks, not token embeddings, carry the semantic burden in LLMs — a step toward modular, fusable, multilingual LMs.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained('Bochkov/pro_bvv_en')
model = AutoModelForCausalLM.from_pretrained('Bochkov/pro_bvv_en', trust_remote_code=True).to('cuda')
inputs = torch.tensor([tokenizer.encode("Example input: ")], device='cuda')
outputs = model.generate(inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))