Update README.md
Browse files
README.md
CHANGED
|
@@ -41,6 +41,12 @@ It is available in the following sizes:
|
|
| 41 |
|
| 42 |
You can use these models directly with the `transformers` library. Since ModernBERT is a Masked Language Model (MLM), you can use the `fill-mask` pipeline or load it via `AutoModelForMaskedLM`.
|
| 43 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
Using `AutoModelForMaskedLM`:
|
| 45 |
|
| 46 |
```python
|
|
|
|
| 41 |
|
| 42 |
You can use these models directly with the `transformers` library. Since ModernBERT is a Masked Language Model (MLM), you can use the `fill-mask` pipeline or load it via `AutoModelForMaskedLM`.
|
| 43 |
|
| 44 |
+
**⚠️ We strongly suggest using ModernBERT with Flash Attention 2, as it is by far the best performing variant of the model, and is a 1:1 match of our research implementation. To do so, install Flash Attention as follows, then use the model as normal:**
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
pip install flash-attn
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
Using `AutoModelForMaskedLM`:
|
| 51 |
|
| 52 |
```python
|