Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
LayaAI
Abii9443
Evaluates chatbot responses using Laya decision models and supports fine-tuning
laya-vs-jev
jupiutrera
Compares Laya and Jev confidence calibration on typed decisions using a pixel-art visualization
RightDesk-Laya
Supankan
Evaluates support-request routing workflows with a local Laya inference adapter and baseline comparisons
laya-korean-eval
sungreong
Measures Korean classification performance, latency, memory, and fine-tuning effects for Laya Multilingual
decision-model-benchmarks
asopitech-publishing
Benchmarks decision models including Laya and Jev on Japanese routing and support classification tasks
harmful-request-classification
zacharyabolton
Experiments compare TF-IDF, DeBERTa, and fine-tuned Laya for harmful-request classification
rag-benchmark
erendikmenn
A local Turkish RAG benchmark measures the contribution of Laya reranking
system-one-demo
dinh144
A research demo compares Laya, Kev, and small language models for agent-memory decisions
tetromino
tomayac
A browser tetromino game uses the Decisions API with selectable Laya and open-jev backends
system1-rerank-bench
dchristopoulos
A benchmark compares Laya and other decision models with rerankers on BEIR retrieval datasets
jev-websearch-eval
anweat
An offline web-search evidence evaluation compares hosted Jev with local Laya and a rule baseline
Reliability-of-Fine_Tuned-Typed-Decision-Models-in-Multiple_Choice-Tasks
PXShou329
Research code studies typed-decision fine-tuning, calibration, robustness, and transfer for Laya and other models
system-one-bench
alilibx
Benchmarks typed-decision models, including Laya, across labelled datasets for accuracy, calibration, latency and cost
laya-demo
foxnukem
Runs a Tower Bloxx browser agent that uses Laya to decide when to drop blocks
laya-governance-gate
yashkumar167272
Evaluates Laya decisions for schema-change governance and calibration
Laya-Dilemma-Lab
SherwinDesouza
Simulates cooperation decisions among 100 agents using the Laya model
laya-vs-jev
JinhoKim46
Compares Laya and Jev on decision quality, speed, and cost for support-ticket routing
laya-vs-reranker-benchmark
gowthxm07
Benchmarks Laya relevance filtering against cross-encoder reranking in RAG workflows
openai-decisions-vs-jev-vs-laya
mateusfonsek
Benchmarks OpenAI Decisions, Jev, and Laya on Portuguese-language decision tasks
mlx-quantization-lora-eval
D-Harshith
Benchmarks MLX quantization and LoRA alongside Laya encoder classifiers on GitHub issue classification
2048GameDecision
Devikrishna545
Compares Jev and Laya playing 2048 and fine-tunes Laya for improved accuracy
laya-autoresearch
Oqura-ai
A Python workspace for iteratively fine-tuning Laya typed-decision models against fixed evaluation contracts
laya_auto
diff-name-diffc
A live-web agent benchmark and fine-tuning workflow for Laya browser decisions
laya-mlx
Thar-Paing123
A native MLX runtime for running Laya typed-decision models on Apple silicon
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya