Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
laya-ov-int8
rupeshs
An int8 OpenVINO export of Laya for CPU inference
jev-laya-benchmark
harrymunro
Benchmarks local MLX Laya against TypeSafe's hosted Jev on synthetic decision tasks
sysone-bench
instax-dutta
Compares Laya and Jev on identical inputs across multiple decision benchmarks
runtime-tutorials
Runtime-weekly
Provides runnable Python guides for Laya classification and a comparison of Laya with other systems
decision-systems-bakeoff
actuallyrizzn
Runs a reproducible benchmark comparing Flybrain, Jev, and Laya on four public tests
flappy-laya-openvino-cpu
rupeshs
Runs Laya inference with OpenVINO to control a Flappy Bird game on CPU
laya-studio
Hantlowt
Specializes frozen Laya representations for new decision tasks and exports portable artifacts
traffic-solver-laya-
alipyth
Simulates city traffic controlled by Laya and includes a model playground and tutorial
vegaml
NandhaKishorM
Produces calibrated typed decisions from frozen language models with conformal sets and abstention
chunklaya
myxamediyar
Chunks long documents and uses Laya for typed-decision lookup across inputs up to one million tokens
zero-shot-ie-bench
umstek
Compares Laya and other zero-shot systems across information-extraction and classification tasks
structured-decision-bench
zhengbangbo
Benchmarks structured decisions from Qwen3, Jev, and Laya CoreML
doomLaya
azalio
Trains and compares Laya and Jev agents playing FreeDoom with reproducible results
Laya-Showcase
zamax14
Demonstrates and benchmarks multilingual Laya alongside Jev and GPT models
laya-ternary-lite
xixi3548942758-design
Quantizes Laya to ternary weights for smaller, lower-memory inference
laya-thalamus
lcbkmm
Middleware uses Laya for agent tool routing with confidence-based LLM fallback and evaluation
open-jev-laya-bench
pngwn
Shared benchmark of Laya 421M vs System One 4B and Bespoke Nimble 9B on accuracy and calibration
laya-multilingual-FP8
Weidows
Provides an FP8 per-row quantization of the multilingual Laya checkpoint with multilingual evaluation results
laya-turkish-mmlu
AhmetSemih
A multilingual Laya checkpoint fine-tuned on Turkish MMLU questions
laya-multilingual-onnx
Emerald7664
CPU-optimized ONNX checkpoints for multilingual Laya decision inference
laya-ko
2nugu
Fine-tunes Laya for Korean and evaluates it on classification, relation extraction, inference, and similarity tasks
laya-typed-decisions
Ankit1106
Fine-tunes Laya on typed-decision examples and reports held-out evaluation metrics
sokudan
hiroki-abe-58
Builds a Japanese typed-decision model and benchmarks it against Laya
LayaSnakeGame
AmirTahaMim
A Pygame Snake game compares human play, a greedy baseline, and local Laya decisions
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya