ModernBERT and mmBERT in Laya
Laya is not a generative LLM. Each checkpoint is an encoder model, ModernBERT-large for English or mmBERT-base for 100+ languages, with a small decision head on top. The encoder explains most of Laya's speed, its token limits and where it falls short.
427 projects
laya-autoresearch
Oqura-ai
A Python workspace for iteratively fine-tuning Laya typed-decision models against fixed evaluation contracts
laya_auto
diff-name-diffc
A live-web agent benchmark and fine-tuning workflow for Laya browser decisions
laya-poc
xanthus-pravin
A proof of concept fine-tunes Laya to detect burnout risk and related signals in team chats
laya-tetris
ArkAung
Experiments fine-tune Laya to play Tetris using text-based board features
laya-workiq-routing
hieunhums
A fine-tuned Laya model routes Microsoft 365 workplace questions to Work IQ sources
ollaya-decision-models-guide
Endogen
Explains how to run and query Kev and Laya decision models locally with Ollaya
laya-coding-decisions
emiliano-go
Trains Laya as a coding-agent judge to score requirement coverage and patch risk
laya-demo
vighnesh-chavan
Demonstrates Laya typed decisions in Doom and a custom pygame shooting game
laya-nethack
Leeleit
Adds Laya risk judgments to a NetHack bot and studies learning from human games
Kirk-laya
Benjamacs1
Runs an ONNX fine-tune of Laya locally to classify Minecraft chat lines
Laya-KG
Diva143V
Fine-tunes Laya for entity linking across biomedical and general knowledge graph domains
laya-fine-tuning
bnap00
Guides fine-tuning Laya on a 6 GB GPU with Pac-Man and Pigeonhole examples
sys1-bench
ugly-custard
Benchmarks Laya and Von decision encoders and provides a practical Laya usage guide
laya-finetune
zhangzhiyuan233
Fine-tunes the Laya model
laya-usecase
ompdas
Demonstrates Laya for ticket triage, policy-aware automation, and evaluation grading
laya-code-head
V0yagernb
Fine-tunes and evaluates Laya for code-review risk judgments
tbt-export-market-access-laya
py392170138
Prepares trade-policy data and runs Laya multilingual weak-supervision and fine-tuning experiments
laya-finetuned-banks-emails-QRC-tagging
bedirachit
Fine-tunes Laya to classify retail-bank emails as queries, requests, or complaints
megathon_arihant
arihant1602
Builds an on-device voice incident assistant with a fine-tuned Laya classifier and speech models
concise
timur27
Score sentence concision locally with a fine-tuned Laya model and a macOS helper
verdict
ashiksharonm
Fine-tunes Laya for multi-check LLM guardrails and evaluates it with open benchmarks
jey_pro_v2
buibui2026
Builds and evaluates a Laya fine-tuning baseline for legal RAG judging
robotics_jev
Kraga922
Fine-tunes a Laya-style model on Alfworld and Libero datasets for robotic decision-making
laya_dp
ranjith702
Fine-tunes Laya as a typed-decision classifier for DevPilot requests
ModernBERT
ModernBERT is a bidirectional, encoder-only Transformer in the BERT style, made by Answer.AI, LightOn and collaborators and released in December 2024 (arXiv 2412.13663). From the ModernBERT-large model card:
- Two sizes: ModernBERT-base (22 layers, 149M parameters) and ModernBERT-large (28 layers, 395M).
- Trained on 2 trillion tokens of English and code.
- Native context of up to 8,192 tokens, using rotary position embeddings and alternating local and global attention.
- Apache 2.0 licence.
It is a masked language model. The card says to fine-tune it with standard BERT recipes for classification, retrieval or QA, which is what Laya does.
mmBERT
mmBERT is a multilingual encoder on the ModernBERT architecture from the Center for Language and Speech Processing at Johns Hopkins University. The paper, "mmBERT: A Modern Multilingual Encoder with Annealed Language Learning" (arXiv 2509.06888), was published in September 2025. From the mmBERT-base model card:
- mmBERT-small has 140M parameters and mmBERT-base 307M.
- Trained on more than 3T tokens across 1,800+ languages.
- Maximum sequence length of 8,192 tokens.
- MIT licence.
How Laya builds on them
From the Laya model card:
| checkpoint | encoder | total params | default context |
|---|---|---|---|
laya |
ModernBERT-large | 421M | 512 |
laya-multilingual |
mmBERT-base | 322M | 1,024 (up to 8,192) |
laya-typed-decisions |
ModernBERT-large | 421M | 1,024 |
The encoder is fully fine-tuned. On top sits a decision head trained from scratch: two transformer layers, an option-marker scorer and an act/escalate head. Every option is scored at its own [MASK] token and then softmaxed over that question's options. The answer space is therefore defined in the request, and new label sets need no retraining. Training uses what the card calls RLCD, reinforcement learning against strictly proper scoring rules, so that reporting honest probabilities earns the highest reward.
Why an encoder
A generative model writes its answer token by token, and you parse the text. An encoder reads the whole input once and returns scores. For Laya that means:
- One forward pass. All questions in a call are answered in the same pass. The card reports 39.5 ms for one question with
layaand 32.8 ms withlaya-multilingualon a Tesla T4. - A probability for every option. You set the threshold in code instead of in a prompt.
- Only your labels. The model scores the options you wrote, so it cannot answer with a label you did not define. It can still pick the wrong one.
Limits that come from the encoder
- Short default context.
layareads 512 tokens per question, with 192 reserved for the options (head_max_len), which leaves about 320 for the state. The other two checkpoints default to 1,024 with 256 for options. Depending on the runtime, longer input is cut off or rejected: laya.cpp rejects it unless you pass--allow-truncation. - Long documents. Only the multilingual checkpoint reads up to 8,192 tokens, with
max_len=8192. The card reports 16 to 18 of 20 requests correct with up to about 4,000 tokens, and 8 to 17 of 20 beyond that. - Languages. ModernBERT was trained on English and code. The card reports that the English checkpoint scores 0.000 on Khmer at 0.952 confidence, which is why the
Routersends non-English text to the mmBERT checkpoint. - Knowledge and reasoning. A 400M encoder knows far less than a decision model built on a multi-billion-parameter LLM. In akitaonrails' benchmark of Jev-like models, Laya scored 42.9% on the compact set against a chance level of about 31%. The 4B to 27B decision models in the same table scored between 74.4% and 99.4%, and Laya was among the fastest at 41 ms median.
Fine-tuning is cheap
The small encoder is also why fine-tuning is practical. The card's fine-tuning notebook runs on Kaggle's free 2x T4 GPUs, and the resulting laya-typed-decisions checkpoint scores 0.766 on the typed-decisions benchmark against 0.362 for the base English model. Other encoder decision models follow the same pattern: Supersonic Labs' Julia-1 is a 144M model on mmBERT-small. See Laya alternatives for more.
Related pages
- Laya in Python: install, the
Routerand the three checkpoints. - Zero-shot classification with Laya: what the encoder can do without training.
- Laya benchmarks: independent accuracy and latency tests.
More ways to use Laya