ModernBERT and mmBERT in Laya

ModernBERT

ModernBERT is a bidirectional, encoder-only Transformer in the BERT style, made by Answer.AI, LightOn and collaborators and released in December 2024 (arXiv 2412.13663). From the ModernBERT-large model card:

• Two sizes: ModernBERT-base (22 layers, 149M parameters) and ModernBERT-large (28 layers, 395M).
• Trained on 2 trillion tokens of English and code.
• Native context of up to 8,192 tokens, using rotary position embeddings and alternating local and global attention.
• Apache 2.0 licence.

It is a masked language model. The card says to fine-tune it with standard BERT recipes for classification, retrieval or QA, which is what Laya does.

mmBERT

mmBERT is a multilingual encoder on the ModernBERT architecture from the Center for Language and Speech Processing at Johns Hopkins University. The paper, "mmBERT: A Modern Multilingual Encoder with Annealed Language Learning" (arXiv 2509.06888), was published in September 2025. From the mmBERT-base model card:

• mmBERT-small has 140M parameters and mmBERT-base 307M.
• Trained on more than 3T tokens across 1,800+ languages.
• Maximum sequence length of 8,192 tokens.
• MIT licence.

How Laya builds on them

From the Laya model card:

checkpoint	encoder	total params	default context
laya	ModernBERT-large	421M	512
laya-multilingual	mmBERT-base	322M	1,024 (up to 8,192)
laya-typed-decisions	ModernBERT-large	421M	1,024

The encoder is fully fine-tuned. On top sits a decision head trained from scratch: two transformer layers, an option-marker scorer and an act/escalate head. Every option is scored at its own [MASK] token and then softmaxed over that question's options. The answer space is therefore defined in the request, and new label sets need no retraining. Training uses what the card calls RLCD, reinforcement learning against strictly proper scoring rules, so that reporting honest probabilities earns the highest reward.

Why an encoder

A generative model writes its answer token by token, and you parse the text. An encoder reads the whole input once and returns scores. For Laya that means:

• One forward pass. All questions in a call are answered in the same pass. The card reports 39.5 ms for one question with laya and 32.8 ms with laya-multilingual on a Tesla T4.
• A probability for every option. You set the threshold in code instead of in a prompt.
• Only your labels. The model scores the options you wrote, so it cannot answer with a label you did not define. It can still pick the wrong one.

Limits that come from the encoder

• Short default context. laya reads 512 tokens per question, with 192 reserved for the options (head_max_len), which leaves about 320 for the state. The other two checkpoints default to 1,024 with 256 for options. Depending on the runtime, longer input is cut off or rejected: laya.cpp rejects it unless you pass --allow-truncation.
• Long documents. Only the multilingual checkpoint reads up to 8,192 tokens, with max_len=8192. The card reports 16 to 18 of 20 requests correct with up to about 4,000 tokens, and 8 to 17 of 20 beyond that.
• Languages. ModernBERT was trained on English and code. The card reports that the English checkpoint scores 0.000 on Khmer at 0.952 confidence, which is why the Router sends non-English text to the mmBERT checkpoint.
• Knowledge and reasoning. A 400M encoder knows far less than a decision model built on a multi-billion-parameter LLM. In akitaonrails' benchmark of Jev-like models, Laya scored 42.9% on the compact set against a chance level of about 31%. The 4B to 27B decision models in the same table scored between 74.4% and 99.4%, and Laya was among the fastest at 41 ms median.

Fine-tuning is cheap

The small encoder is also why fine-tuning is practical. The card's fine-tuning notebook runs on Kaggle's free 2x T4 GPUs, and the resulting laya-typed-decisions checkpoint scores 0.766 on the typed-decisions benchmark against 0.362 for the base English model. Other encoder decision models follow the same pattern: Supersonic Labs' Julia-1 is a 144M model on mmBERT-small. See Laya alternatives for more.

Related pages

• Laya in Python: install, the Router and the three checkpoints.
• Zero-shot classification with Laya: what the encoder can do without training.
• Laya benchmarks: independent accuracy and latency tests.