Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
laya-mlx
mizorewww
Runs Laya typed-decision models locally on Apple Silicon using MLX
laya-coreml
mizorewww
Runs Laya typed-decision models locally on Apple Silicon using Core ML and the Neural Engine
openJev-verdict-2.0
Heman10x-NGU
Develops and benchmarks a non-autoregressive typed-decision model against Jev and Laya
laya-playground
wdobry
A local website combines Laya model demos, games, a benchmark, and an agent skill
laya-vs-jev
virajbhartiya
A side-by-side T-Rex game compares local Laya and hosted Jev decision models
laya-mps
afshinm
Runs Laya typed-decision inference locally on Apple Silicon using Metal Performance Shaders
eikos
caiovicentino
Builds calibrated typed-decision models for finance and trading and evaluates them against Laya
JevAny
SimpleJev
Training and evaluation infrastructure for System 1 decision models, with Laya as a benchmark baseline
lev
jlt-commons
Implements a typed decision engine using Laya checkpoints and GGUF chat models
edgejev
yzfly
Exports, quantizes, and serves Laya and other Jev-style models for offline CPU inference
MacJev-322M-4K-Laya
chaoliangUNSW
Fine-tunes Laya for long-context local Mac agents and compares it with the base model
laya-jev-lab
yibie
Compares Jev and Laya decision models and evaluates a local-first inference cascade
opendecider
manjunathshiva
Provides calibrated System 1 decision models and benchmarks them against Laya and other models
laya_router
glukicov
A Python model router compares local Laya decisions with a GPT-5 nano routing model
jevbench
dhruvmehra
A reproducible benchmark compares JEV, Laya, and other classifiers across datasets and metrics
laya-as-judge
rbrus
Uses Laya-MLX to evaluate models and agents with typed-decision judgments
kime
tamnd
A Rust inference engine and HTTP server runs Laya checkpoints across CPU, CUDA, and Apple GPUs
MacJev-322M-4K-Laya-GGUF
chaoliangUNSW
Provides GGUF encoder files and a runtime for running MacJev with llama.cpp
MacJev-322M-4K-Laya-MLX
chaoliangUNSW
Runs the MacJev decision model natively on Apple silicon with an MLX runtime
laya-plays-smb3
cv
Demonstrates Laya controlling Super Mario Bros. 3 with recorded, replay-verified decisions
reflexbench
brida-ai
Benchmarks Laya and other typed-decision engines across quality, calibration, robustness, and latency
laya-jev-benchmark
Luni
Benchmarks Laya against Jev and other models on phishing detection and calibration
laya-onnx
MstyAI
Runs Laya decision models locally with ONNX Runtime through a Go library and CLI
jev-tests
schacon
Compares Laya, Jev, Kev, and Claude in three macOS typed-decision demos
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya