18 70 48

Mohammed Hamdy

mmhamdy

AI & ML interests

TechBio | AI4Sci | NLP | Reinforcement Learning

Recent Activity

upvoted an article 5 days ago

Continuous batching from first principles

reacted to Kseniase's post with ❤️ 14 days ago

12 Types of JEPA Since Yann LeCun together with Randall Balestriero released a new paper on JEPA (Joint-Embedding Predictive Architecture), laying out its theory and introducing an efficient practical version called LeJEPA, we figured you might need even more JEPA. Here are 7 recent JEPA variants plus 5 iconic ones: 1. LeJEPA → https://huggingface.co/papers/2511.08544 Explains a full theory for JEPAs, defining the “ideal” JEPA embedding as an isotropic Gaussian, and proposes the SIGReg objective to push JEPA toward this ideal, resulting in practical LeJEPA 2. JEPA-T → https://huggingface.co/papers/2510.00974 A text-to-image model that tokenizes images and captions with a joint predictive Transformer, enhances fusion with cross-attention and text embeddings before training loss, and generates images by iteratively denoising visual tokens conditioned on text 3. Text-JEPA → https://huggingface.co/papers/2507.20491 Converts natural language into first-order logic, with a Z3 solver handling reasoning, enabling efficient, explainable QA with far lower compute than large LLMs 4. N-JEPA (Noise-based JEPA) → https://huggingface.co/papers/2507.15216 Connects self-supervised learning with diffusion-style noise by using noise-based masking and multi-level schedules, especially improving visual classification 5. SparseJEPA → https://huggingface.co/papers/2504.16140 Adds sparse representation learning to make embeddings more interpretable and efficient. It groups latent variables by shared semantic structure using a sparsity penalty while preserving accuracy 6. TS-JEPA (Time Series JEPA) → https://huggingface.co/papers/2509.25449 Adapts JEPA to time-series by learning latent self-supervised representations and predicting future latents for robustness to noise and confounders Read further below ↓ It you like it, also subscribe to the Turing Post: https://www.turingpost.com/subscribe

liked a Space about 1 month ago

HuggingFaceH4/on-policy-distillation

View all activity

Organizations

Posts 9

Post

1713

What inspired the Transformer architecture in the "Attention Is All You Need" paper? And how were various ideas combined to create this groundbreaking model?

In this lengthy article, I explore the story and the origins of some of the ideas introduced in the paper. We'll explore everything from the fundamental attention mechanism that lies at its heart to the surprisingly simple explanation for its name, Transformer.

💡 Examples of ideas explored in the article:

✅ What was the inspiration for the attention mechanism?
✅ How did we go from attention to self-attention?
✅ Did the team have any other names in mind for the model?

and more...

I aim to tell the story of Transformers as I would have wanted to read it, and hopefully, one that appeals to others interested in the details of this fascinating idea. This narrative draws from video interviews, lectures, articles, tweets/Xs, and some digging into the literature. I have done my best to be accurate, but errors are possible. If you find inaccuracies or have any additions, please do reach out, and I will gladly make the necessary updates.

Read the article: https://huggingface.co/blog/mmhamdy/pandemonium-the-transformers-story

View all Posts