Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
laya-test
QI366
Tests Laya models for intent recognition across different business scenarios
laya-assisted-screening
kamranb02
Screens underground hydrogen storage literature with Laya and evaluates its classifications against human labels
laya-1244-weight-patch
code2saurabh
Patches Laya checkpoint weights to address a positive-sentiment classification failure
S1CogBench
dengzhe-hou
Benchmarks System-1 decision models, including Laya, on 24 cognitive experiments
laya-edge-bench
parang12
Compares Laya model variants for decision quality and resource use on simulated IoT edge workloads
weight-edit-lab
demirelo
Studies sequential hand-written weight edits for neural networks, including a pilot on the Laya decision model
glassbox
code2saurabh
Inspects, evaluates, edits and ships small decision models, including Laya checkpoints
laya-rag-judge-ko
realizer-1966
Runs browser-based RAG claim verification and evaluation with the Laya typed-decision model
pLaya
McThizzin
A Go terminal client for running, comparing, and benchmarking decisions through a Laya server
laya-browser-grounding
Who-Code
A study and demo using Laya-MLX to ground browser automation actions on page elements
CompoundLab
realizer-1966
Runs browser-based RAG claim verification, evaluation, and routing with the Laya typed-decision model
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya