Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
228 projects
doom-war
rythmn1111
Compares local MLX Laya and hosted Jev as decision-makers in a Doom deathmatch
laya-micro
osamabinIaggin
Prunes and quantizes Laya for low-memory single-board computer deployment
laya-invaders
inhabitants
Uses Laya to choose targets in a Space Invaders game and compares it with a rule baseline
jev-laya-benchmark
EnesDemir143
Benchmarks TypeSafe Jev and Laya-MLX on structured issue classification
laya-learning-demo
Daryl9441
A runnable tutorial teaches Laya decision primitives, routing, fine-tuning, and comparison with Jev
layaAsRagJudge
vishalmysore
Verifies RAG claims in the browser using Laya decisions and evaluates accuracy against labeled claims
laya-cuda
adamjen
Benchmarks Laya typed-decision checkpoints with a CUDA backend on NVIDIA GPUs
laya-exp
changtimwu
Experiments comparing an MLX Laya port with Jev and a TypeSafe-compatible Laya client
laya-lab
Alevsk
Runs Laya locally with reproducible scenarios that explore its capabilities on Apple Silicon
laya-inference-engine
aaditagrawal
Provides a GPU inference engine and performance modes for Laya on NVIDIA Blackwell
verdict
Manavarya09
Provides calibrated zero-shot and fine-tuned typed decision models as an alternative to Jev
kime
tamnd
A Rust inference engine and HTTP server runs Laya checkpoints across CPU, CUDA, and Apple GPUs
laya-vs-dijkstra
antonellof
Compares Laya MLX pathfinding decisions with Dijkstra on seeded weighted mazes
laya-mlx-zh
ZLHAOOO
Provides Chinese fine-tuned Laya weights, training data, and evaluations for Apple Silicon MLX
extra-laya-bench-ft-data
cjdd3b
Provides additional data for Laya benchmark fine-tuning
laya-formatting-fragility
pranaysuyash
Measures how formatting changes affect typed-decision model outputs
next-jev-laya-test
JonesLin
Provides multilingual evaluation data for testing Jev and Laya across text-classification tasks
laya-onnx-bench
gdelatournelle
Provides a reproducible evaluation set for Laya's ONNX decision backend
laya-rs
codesoda
A planned Rust runtime for local Laya decisions with Metal acceleration and a Jev-compatible HTTP API
Laya-vs-Jev
abcdmku
A side-by-side test bench compares the local Laya model with the hosted Jev API
laya-blackjack
adambloebaum
Trains and evaluates a Laya-based blackjack policy with a simulator and probability dashboard
laya-car-snake
KarimHFahmy
Uses Laya to choose safe moves in a car-and-trailer grid game
laya-warehouse-safety
senthilr-nv
Simulates warehouse robot decisions using Laya, with a browser replay and deterministic collision-safety layer
laya-rlcd-benchmark
mourad-ghafiri
Evaluates the multilingual Laya model on adversarial decision-making scenarios
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya