Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
S1Q
CYMCharming
Implements and evaluates low-bit quantization for typed decision models including Laya
jev-smart-home-lab
SafalFrom2050
A smart-home simulation that compares Jev API and local Laya decisions
laya-blackjack
hugging-apps
An interactive blackjack demo using a Laya fine-tune to predict action preferences and game outcomes
jev-laya
diptihumraskar
A zero-shot topic classification benchmark comparing Jev, Laya and CLM
layastudio
Lu-cong-1
Provides a local visual workbench for testing, evaluating, and serving Laya decision models
laya-calibration
BargLabs
Publishes a preregistered calibration experiment evaluating Laya as a judge of agent completion reports
laya-qbert
mateusbsmg
Fine-tunes Laya to play Q*bert and reports its performance against a teacher and baseline
LAYA-Decision-Head-
satviksangamkar11
Studies a local decision head and benchmarks it against Laya as a baseline
laya-qcs6490
nameissakthi25
Converts and benchmarks Laya on a Qualcomm QCS6490 NPU using a reproducible quantization recipe
laya-btc-decision-console
avtandilkasparov-debug
Measures Laya decision latency and compares paper-trading strategies in a local dashboard
laya-on-rk3588-jetson
suharvest
Converts and benchmarks Laya for deployment on Rockchip RK3588 and NVIDIA Jetson Orin
graph-loop-train
jtatman
Runs an adaptive training loop to improve fine-tuned Laya models
jev-laya-lab
luoshuai990529
Compares Jev and locally hosted Laya on decision accuracy, answer distributions, and task completion time
LayaETF
AM1975MA
Benchmarks Laya formulations for pairwise ETF decisions using a leakage-safe research harness
laya-reproduction
jimni3155
Reproduces selected Laya benchmark experiments, including typed-decision checks and option-name robustness
laya-retex
TojiiSamaa
Reports measured Laya experiments, fine-tuning, and applications in local search and tool routing
laya-cycle-dashboard
gchat137
Displays replayable simulation results from Laya decision-model game runs
laya-incident-triage
curtiscovington
Evaluates Laya against rule-based and trained baselines for incident triage and routing
jeff-laya-routing-study
tiw
Compares Jeff and Laya for sub-agent routing using a stacking model and confidence cascade
yt_laya_groq_comparison
BLUERAY94
Benchmarks Laya locally and optionally compares its checkpoints with Groq
decision-map-bench
move38studios
Benchmarks Jev, Clef, and Laya decision models on geography questions across a world grid
laya-check
e9tgldr
Benchmarks Laya-MN against JEV on Mongolian language understanding
laya-pii-bench
dfridkin
Benchmarks the open-weight Laya model as a local PII classifier on synthetic clinical-trial documents
Laya-sol
sushan5140
Prepares and validates an evaluation of GPT-6.1 Sol against Laya's typed-decision benchmark
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya