Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
laya-evals
Gjusev
Evaluates Laya decision models and audits confidence calibration for automated decisions
laya-evals-gate
Gjusev
Compares Laya evaluation metrics against baselines and fails CI when accuracy or calibration regresses
typed-decision-slay-the-spire
MindrLabs
Compares Convai Laya and Nimble models playing Slay the Spire, including fine-tuned runs
google-adk-laya-router
derrickqin
A Google ADK example routes requests among Gemini tiers using Laya and evaluates routing results
laya-jev-benchmark
eugeneproai
Compares local Laya and hosted Jev across decision scenarios with benchmark telemetry
ferrum
SampleBias
Plans a Hermit unikernel whose resource policies are selected by an external Laya process
laya-vs-llms-benchmark
stuaninauts
Compares Laya with machine-learning and language models on financial complaint triage
laya-android
Byun11
Fine-tunes Laya into a typed-decision policy for Android GUI action selection
goal-classification-bench
fuljorge
Benchmarks Laya and Strands Decider for customer-goal classification in Brazilian Portuguese
miniLaya
yesterdayfuture
Fine-tunes a BERT-based typed-decision model and compares its outputs with Jev and Laya
decision-model-benchmarks
di37
Compares four decision models, including Laya, across classification, reranking, routing, and tool selection
business-classifier-benchmark
EduardoMoraga
Benchmarks classical classifiers against Jev and Laya for employee attrition prediction
llm-decision-api
OpenAurora
Provides a Qwen-powered semantic decision API and compares its results with Laya and Jev
vega-laya-chess
harishkotra
Plays and benchmarks chess matches between local Vega and Laya decision models
laya-formatting-fragility
pranaysuyash
Measures how formatting changes affect typed-decision model outputs
laya-swedish-gate-v4-data
jonasaise
Provides Swedish preference pairs for training and evaluating the Laya preference gate
decision-benchmark-jev-laya-julia
morrenhale
Provides benchmark data comparing decision models Jev, Laya, and Julia across classification tasks
laya-multidomain-decisions
devinblaze
A 665-example benchmark dataset of structured decisions across four domains
laya-session-guard-risk-ranked-metrics
hxrikp
Publishes metrics from a rejected Laya prompt-injection risk-ranking experiment
riidolaya-statehint-laya-reference-v0.1-development
JooYoon
Reports development measurements using a frozen pretrained Laya checkpoint
arc-usdc-laya-bench
goktugoguz
Publishes training rows, audited labels and benchmark results for Laya on Arc USDC transfer questions
stellar-laya-bench
goktugoguz
Publishes benchmark results and held-out evaluations for Laya on Stellar operations
next-jev-laya-test
JonesLin
Provides multilingual evaluation data for testing Jev and Laya across text-classification tasks
laya-browser-suite-c
cklxx
Provides held-out real-site tasks for evaluating a browser agent
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya