Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
laya-experiments
Mac-007
Runs experiments and empirical evaluations of Laya for text-based decision making
laya-trader
zakanysa
Tests Laya-driven trading strategies on an on-chain order book and evaluates their real costs
laya-benchmark
LokenSI
Benchmarks local decision models and Jev against Laya on English and Norwegian tasks
laya_mobile
SerAbin1
A Flutter app for testing Laya model performance on mobile
laya-fast-finetune
sathwikbs
Fine-tunes Laya with a packed training layout to reduce repeated state encoding
Jev-OpenSource-Alternative-Laya
Nagwa-Ahmed
Evaluates local Laya checkpoints on zero-shot text classification datasets and exports Excel reports
agent-failure-triage
PawnsApprentice
An experiment compares Laya with TF-IDF and an LLM judge for triaging failed agent runs
compare-IA-decision
laurentvv
A benchmark harness compares local System One decision models, including Laya, using measured results
Jev-OpenSource-Alternative-DEMO-Laya
Nagwa-Ahmed
Evaluates local Laya classification across Hugging Face datasets and exports Excel reports
aubin
Emrevrg
Provides calibrated decision models, computer-use tools, and a Laya/Jev-compatible API
layaFineTuning
codingpipeline
Fine-tunes Laya to route customer return requests and evaluates its accuracy and safety
the-new-turing-test
glowleaf
A Magic: The Gathering game harness tests Laya as a player
laya-banking77
hari-hashing
Evaluates Laya decision-model classification and routing on the Banking77 intent dataset
laya-bias-audit
maheshwari2000
An audit measures option-order, key, and wording biases in Laya decision-model checkpoints
HeXA-laya-demo
queso-gato1355
A local dashboard demonstrates Laya decisions for an autonomous delivery robot scenario
hybrid-decision-agent
musaugurlu
Tests a Laya-first agent architecture that routes requests to Gemini Flash and benchmarks its performance
Laya_Jev_opensource_guide
gotree94
Explains how to self-host Laya and Jev, including setup, fine-tuning, and benchmarks
laya-vs-jev-toxicity
jmgb27
Compares Jev and fine-tuned Laya on toxicity classification using Civil Comments
distill-public
abhi-sama
Fine-tunes local Laya decision models from synthetic data and evaluates and exports calibrated models
laya-demo
bbaotran
Snake game played by Laya on CPU, with Jev and other decision backends for comparison
jev-laya-lab
KarlPeixoto
Explores Laya and JEV decision-making through Python experiments and evaluations
laya-local-benchmark
saksham-personal
Benchmarks and packages offline CPU Laya decision workflows for Windows VDI environments
Sytem-1-model-testing--jev--Laya-
dinod001
Benchmarks System 1 decision models against autoregressive LLMs for latency, cost, and accuracy
system1-prompt-lab
msmidhunms
Evaluates and compares Laya and other System 1 classifiers with datasets and prompt optimization
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya