Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
laya_finetuning
mkthoma
Fine-tunes Laya to normalise messy place records into a 10-class taxonomy, with phase reports
laya-tictactoe
amdisrar
Tests Laya's tic-tac-toe decisions against a Minimax reference
laya-snake
zhenda-hub
Runs Laya in Snake and compares its play with four other strategies
laya-classifier-finetuning
sallout
Fine-tunes Laya for custom text classifiers and optionally exports models to Core ML
jev-vs-laya-dino
RodolfoBonis
A Chrome Dino game demo compares Jev and Laya on gameplay, latency, cost, and performance
jev-laya-case-study
Takuakira
A controlled case study compares Jev and Laya decision responses as repair costs change
finetuning-laya-for-browser-agents
kp-algomaster
Code, datasets, and evaluations fine-tune Laya for structured browser automation decisions
frontdoor
debojitroy
Screens messages with Laya inference, specialization experiments, and reproducible evaluations
CatalogMesh
debojitroy
Maps supplier catalog products to marketplace categories using Laya inference
Decision-Models-Are-Not-Calculators
AnthusAI
A preregistered study compares Jev, Kev, and Laya on fair-die predictions
decision-model-eval
relevan-dev
An evaluation compares Laya, Jev, and Claude on search-index configuration decisions
doomgemma
dexmac221
Compares Laya with fine-tuned Gemma models in FreeDoom accuracy, calibration, latency, and gameplay tests
laya-email-classifier
mpegram3rd
Explores Laya configurations for classifying email
laya-e2-multitype
13974737898-lgtm
Studies Laya decision quality through reproducible benchmarks, comparisons, and fine-tuning experiments
maclaya
jannisleifeld
Serves local Laya models on Apple Silicon through a Jev-compatible API with a dashboard and playground
laya-experiments
RafaelJaime
Spanish notebooks evaluate Laya decision behavior, diagnostics, message routing, and product recommendation
laya-ludo
mnk786
A browser-based four-player Ludo game uses Laya and Jev to choose legal moves
laya-jev-homeautomation
CoryHawkless
A home-automation command router and benchmark harness exercises Laya
rag-laya-poc
Mr-PU
Compares standard RAG with a pipeline that uses Laya to rerank retrieved passages
laya-football-evaluation
steve-dev-55
Preregisters a multi-agent protocol for evaluating typed decision engines as real-time football forecasters
layev
CompleteDotTech
A research implementation of typed decisions, calibration, and reward training inspired by Kev and Laya
zh-decision-bench
CodyQin
Benchmarks calibration and robustness of System One decision models on Chinese-language tasks
system-one-benchmark
yanng981
Compares accuracy, calibration and latency across System One models on multilingual datasets
laya-android
edwardmonteiro
An Android space dogfight uses Laya’s typed decisions to control an enemy fighter
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya