Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
layarerank
anguyenbus
Reranks documents in batches with Laya decision models and evaluates fine-tuned checkpoints
laya-eval
imaddde867
An evaluation compares local Laya with jevmlx and hosted Jev on public cases
laya_test
liyan24
A Chinese-language test suite evaluates Laya and Jev with a 120-question set and dual-backend comparison
laya-escalate
Sidharth-06
A Python package audits Laya confidence calibration and provides repairs and escalation policies
Laya-lab
DamianPrist
Compares local Laya with DeepSeek and tests Laya's choice-position effects
laya-lambda-cdk
KICU1129
Deploys Laya inference on AWS Lambda and reports Japanese classification and latency evaluations
jev-laya-trader
PrateekReddy116
Compares Jev and Laya decision engines in a paper-trading sandbox
jev-laya-benchmarks
stemirkhan
Benchmarks Jev and Laya against specialized rerankers on BEIR datasets
laya-snag-finetune
Adityaakr
Fine-tunes and evaluates Laya as a pull-request requirement verification engine
jev-laya-trader
lucasmartins-ai
A paper-trading system uses Jev and a local Laya model to judge trading signals
poc-laya-model
CtrlShiftCoder
A Spanish-language evaluation toolkit benchmarks Laya and provides local and HTTP backends
ai-agents-and-laya-system-meld
ManasviAtGitHub
Uses Laya for perception and speaker selection in a multi-agent MELD conversation system
playwright-cli-laya-bakeoff
NerdishShah
Evaluates local Laya in a Playwright CLI workflow for adding a product to a Shopify cart
laya_ai_model_demo
lalithdabilpuram01
Tests Laya on labelled support emails and measures inference latency across local devices
testing_arena_jev_vs_laya
Shadowsweep
Compares Jev and Laya in an automated security attack-detection and defense system
elevator-decision-arena
CircleCI-Research
Benchmarks Jev, Laya, and classic algorithms for simulated elevator dispatch decisions
photo-decision-studio
nsdkoo
Compares local Laya and DeepSeek on photo classification using image descriptions
near_agi
yassinebkr
Evaluates and post-trains Laya as a local transition predictor for agent planning
jev-labs
valdepeace
A TypeScript harness compares Jev, Kev and Laya typed-decision engines and publishes experiment results
chess-war
jhonsu01
A pixel-art chess match pits local Laya against Jev using identical typed questions
triage-incident
mumit
A synthetic network-incident benchmark evaluates Laya alongside Jev, Kev and other models
system1bench
CYMCharming
A reproducible benchmark evaluates Laya and other models on typed decisions, safety and robustness
system-one-on-snapdragon
esterhuizen
Setup scripts and benchmarks run open typed-decision models, including Laya, on Snapdragon hardware
local-jev-bench
tak-bro
Benchmarks local Jev-style decision models, including Laya, on Apple Silicon
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya