Hosted Laya: $1 free, no card
laya.tools
Submit

ModernBERT and mmBERT in Laya

Laya is not a generative LLM. Each checkpoint is an encoder model, ModernBERT-large for English or mmBERT-base for 100+ languages, with a small decision head on top. The encoder explains most of Laya's speed, its token limits and where it falls short.

417 projects

pastewhat-ranker-v1

mizorewww

Distills a candidate-aware clipboard ranker into Laya multilingual for local inference

3·GitHub repo

doomLaya

azalio

Trains and compares Laya and Jev agents playing FreeDoom with reproducible results

3·GitHub repo

laya-cn

yangshun2005

Provides a Chinese translation of Laya documentation for its multilingual typed-decision engine

3·GitHub repo
W

laya-ara

Wouze

Fine-tunes multilingual Laya for Arabic intent classification, inference, and ranking

223·HF model
T

laya-cybersec

TextCortex

A multilingual Laya fine-tune detects prompt injection and data exfiltration, with an included ONNX build

113·HF model
T

laya-code

tindang

A Laya fine-tune that reranks code chunks by relevance to a task description

110·HF model
H

laya-tr-karar

hayriyigit

Fine-tunes Laya for Turkish typed decisions, including choice, yes/no, and score questions

50·HF model
H

laya-issue-triage

harikarthikmanyam

Fine-tunes Laya to classify GitHub issue types and assess whether more information is needed

44·HF model
D

vi-law-laya

duclvQ

Reranks Vietnamese legal passages with a Laya-based relevance model fine-tuned for query answering

40·HF model
A

laya-burmese-sib200

aungthuhein-dev

Fine-tunes Laya for calibrated seven-way Burmese topic classification on the SIB-200 dataset

39·HF model
L

laya-grounded

Luni

Fine-tunes Laya to improve grounding, contradiction handling, and calibration

2·HF model
W

laya-multilingual-FP8

Weidows

Provides an FP8 per-row quantization of the multilingual Laya checkpoint with multilingual evaluation results

2·HF model
F

laya-kannaka-evidence-gate

flaukowski

Fine-tunes Laya to assess whether memory excerpts contain evidence needed to answer a question

2·HF model
A

laya-turkish-mmlu

AhmetSemih

A multilingual Laya checkpoint fine-tuned on Turkish MMLU questions

2·HF model
I

laya-browser-v32b

ichenney

Fine-tunes a Laya-based decision model for browser-agent navigation and page-state judgments

2·HF model
I

laya-rlcd-training

InfinimindCreations

Trains Laya on labeled data for typed decisions and evaluates results against a held-out set

2·HF model

laya-ara

ASNB-Smart-Solutions

Fine-tunes Arabic Laya models for NLU classification and short-list RAG

2·GitHub repo

SnakeGame_Laya

Okbatti

Uses a fine-tuned Laya model to choose moves in a Snake game

2·GitHub repo

gutcheck

janmejai2002

Runs calibrated Jev- and Laya-style decisions on local NPUs or GPUs with agent tool leasing

2·GitHub repo

laya-mlx-zh

ZLHAOOO

Provides Chinese fine-tuned Laya weights, training data, and evaluations for Apple Silicon MLX

2·GitHub repo

laya

fivetop

Fine-tunes the Laya decision model on stock data for buy, sell, or hold signals

2·GitHub repo
A

atmem-laya-formation-v1

atmem

Provides synthetic scenarios and typed decisions for training and calibrating AtMem's Laya profile

1·HF dataset
M

LayaVoxprompt-DE-EN

MainzelMennchen

Provides German and English dictation examples for training a Laya prompt-intent classifier

1·HF dataset
T

laya-tr

TurkishCodeMan

Fine-tunes Laya for fast Turkish decision-making, candidate selection, and routing

69·HF model

ModernBERT

ModernBERT is a bidirectional, encoder-only Transformer in the BERT style, made by Answer.AI, LightOn and collaborators and released in December 2024 (arXiv 2412.13663). From the ModernBERT-large model card:

  • Two sizes: ModernBERT-base (22 layers, 149M parameters) and ModernBERT-large (28 layers, 395M).
  • Trained on 2 trillion tokens of English and code.
  • Native context of up to 8,192 tokens, using rotary position embeddings and alternating local and global attention.
  • Apache 2.0 licence.

It is a masked language model. The card says to fine-tune it with standard BERT recipes for classification, retrieval or QA, which is what Laya does.

mmBERT

mmBERT is a multilingual encoder on the ModernBERT architecture from the Center for Language and Speech Processing at Johns Hopkins University. The paper, "mmBERT: A Modern Multilingual Encoder with Annealed Language Learning" (arXiv 2509.06888), was published in September 2025. From the mmBERT-base model card:

  • mmBERT-small has 140M parameters and mmBERT-base 307M.
  • Trained on more than 3T tokens across 1,800+ languages.
  • Maximum sequence length of 8,192 tokens.
  • MIT licence.

How Laya builds on them

From the Laya model card:

checkpoint encoder total params default context
laya ModernBERT-large 421M 512
laya-multilingual mmBERT-base 322M 1,024 (up to 8,192)
laya-typed-decisions ModernBERT-large 421M 1,024

The encoder is fully fine-tuned. On top sits a decision head trained from scratch: two transformer layers, an option-marker scorer and an act/escalate head. Every option is scored at its own [MASK] token and then softmaxed over that question's options. The answer space is therefore defined in the request, and new label sets need no retraining. Training uses what the card calls RLCD, reinforcement learning against strictly proper scoring rules, so that reporting honest probabilities earns the highest reward.

Why an encoder

A generative model writes its answer token by token, and you parse the text. An encoder reads the whole input once and returns scores. For Laya that means:

  • One forward pass. All questions in a call are answered in the same pass. The card reports 39.5 ms for one question with laya and 32.8 ms with laya-multilingual on a Tesla T4.
  • A probability for every option. You set the threshold in code instead of in a prompt.
  • Only your labels. The model scores the options you wrote, so it cannot answer with a label you did not define. It can still pick the wrong one.

Limits that come from the encoder

  • Short default context. laya reads 512 tokens per question, with 192 reserved for the options (head_max_len), which leaves about 320 for the state. The other two checkpoints default to 1,024 with 256 for options. Depending on the runtime, longer input is cut off or rejected: laya.cpp rejects it unless you pass --allow-truncation.
  • Long documents. Only the multilingual checkpoint reads up to 8,192 tokens, with max_len=8192. The card reports 16 to 18 of 20 requests correct with up to about 4,000 tokens, and 8 to 17 of 20 beyond that.
  • Languages. ModernBERT was trained on English and code. The card reports that the English checkpoint scores 0.000 on Khmer at 0.952 confidence, which is why the Router sends non-English text to the mmBERT checkpoint.
  • Knowledge and reasoning. A 400M encoder knows far less than a decision model built on a multi-billion-parameter LLM. In akitaonrails' benchmark of Jev-like models, Laya scored 42.9% on the compact set against a chance level of about 31%. The 4B to 27B decision models in the same table scored between 74.4% and 99.4%, and Laya was among the fastest at 41 ms median.

Fine-tuning is cheap

The small encoder is also why fine-tuning is practical. The card's fine-tuning notebook runs on Kaggle's free 2x T4 GPUs, and the resulting laya-typed-decisions checkpoint scores 0.766 on the typed-decisions benchmark against 0.362 for the base English model. Other encoder decision models follow the same pattern: Supersonic Labs' Julia-1 is a 144M model on mmBERT-small. See Laya alternatives for more.

Related pages

More ways to use Laya

Run Laya on your MacLaya on MLXLaya on Core MLUse Laya from Node.jsLaya for PythonLaya on ONNX RuntimeLaya on OpenVINOLaya for RustLaya in C++ and GGUFRun Laya in the browserLaya for Go, Java, Elixir, Ruby and ZigContent moderation with LayaText classification with LayaRouting with LayaLaya for agentsLaya in coding toolsLaya for search and RAGGames played by LayaLaya benchmarksSelf-host a Laya API serverZero-shot classification with LayaLaya alternativesHosted Laya API