Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
228 projects
laya-vs-jev
darrenli6
Compares Laya and Jev in snake-racing and fighting-game arenas
jev-vs-laya
janagarajsn
A Flask app benchmarks Jev and local Laya on synthetic customer-support tickets
laya-jev-eval
yuvrajrox
Evaluates Laya against Jev on email intent and explores text generation with Laya’s backbone
laya-open-eval
lgy1027
Packages local Laya inference and evaluation with an optional HTTP adapter
laya-vs-llms
Ujjwal3115
Benchmarks Laya against hosted language models on developer commit triage and CI safety decisions
laya-zh-eval
lzero07
Evaluates Laya checkpoints on Chinese skill-routing requests
laya-support-ticket-triage
IshaanLabs
A local Laya system triages customer-support tickets and benchmarks results in an interactive dashboard
laya-or-jev-pong
bananadonn
Compares local Laya and hosted Jev decision APIs by having them control paddles in Pong
llm-jev-laya-bench
PerryLink
Measures the cost, latency, and failure boundaries of judgment layers
laya-gtx1650-benchmark
PredictiveManish
Measures Laya inference latency on a laptop with a GTX 1650 GPU
laya-vs-llm-benchmark
Shray15
Benchmarks Laya against a local LLM for selecting MCP tools
pol2-laya-studio
shentonyan
A local interactive playground visualizes and evaluates typed decisions from Laya models
ai-update-radar-lab
starhunt
Tests Laya-based project relevance decisions and compares results with saved Jev evaluations
Jev
Caho1
A research repository documents Laya fine-tuning, evaluation, local inference and browser automation experiments
trust-router
NotoriousPOG
Compares rules, BERT, and Laya for shadow-mode AI security routing
jev-frontend-lab
dante01yoon
Six frontend demos compare Jev and local Laya decision pipelines with optional LLM refinement
featherbench2
ed-is-ai
Benchmarks language models and typed-decision models on custom tasks and agentic decision sequences
yks-bench
UgurcanAkkok
Benchmarks local Laya checkpoints and hosted Jev on Turkish university entrance exam questions
PacmanLocalJev
raboija
Runs local Laya decisions in a Windows Pac-Man-inspired game with live probabilities and benchmarks
jev-demo
nadeem4
A demo site compares Jev and Laya playing games and includes benchmarks and agent experiments
faceid-bench
mazenDDr
A face-unlock benchmark measures detection and recognition pipelines and tests Laya for yes/no decisions
sys1bench
rssr25
A benchmark measures calibration, robustness, and other properties of typed decision models including Laya
typed-decision-bench
4nt0ineB
Compares Jev, OpenJev, Laya, and other models on zero-shot classification tasks in English and French
crisis-triage
mazenDDr
Fine-tunes and evaluates Laya for multilingual disaster-message triage with uncertainty routing
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya