Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
flylaya
leewbl
A fruit-fly behavior study combines connectome-driven motion with a Laya intent layer
Laya_pokemon
KevinYu969
Connects the Laya decision model to PyBoy for local Pokémon control experiments
laya-lab
karth2512
Benchmarks Laya classification and fine-tuning methods against classical baselines and the Jev API
playwright-system1-triage
kint4
Classifies Playwright failures and benchmarks calibrated CI decisions using local Laya and the Jev API
decision-models-vs-small-llms
pournasserian
Compares Laya and other decision models with small LLMs across six agent decision tasks
laya-burn
lileeei
Runs and fine-tunes the Laya decision model natively in Rust using the Burn tensor framework
laya-vs-local-llm
vyshakhgnair
Compares Laya with a local language model on tool routing and document classification
System1Othello
PanikParty
Benchmarks System One models, including Laya, on Othello decisions against search engines
dispatch-system1-router
rafi-seniang
Compares Jev, Laya, and other decision methods in a configurable text-triage router
nirnay
nandukmelath
Provides a CPU decision model and benchmarks its performance against Laya
ReflexFoundry
nazeeh111
Trains and evaluates local Laya decision heads for coding-agent workflows
TSP-Laya
TanishkBansode
Fine-tunes Laya for the traveling salesperson problem and benchmarks its generated tours
System-One-Lab---Comparing-Laya-and-Decider-0.8B
AagneyN
Benchmarks and fine-tunes Laya for support-ticket classification alongside Decider 0.8B
laya-cuda
Banshal-Yadav
Ports Laya typed-decision inference to PyTorch CUDA with GPU and CPU acceleration
s1kit
Eliot5566
Tests, calibrates, and routes System One decision models, including Laya, on custom data
system-one-models-research
jshaam7
Presents research notes and evaluation designs comparing Laya, Jev, and conventional baselines
d1-consolidation-copilot
VincenzoVant
Maps multilingual company accounts and matches intercompany entries, comparing Laya with conventional models
review-intelligence-ptbr
renatospessotto
Fine-tunes and evaluates Laya for Portuguese review aspect classification
@opentask/taskin-difficulty-estimator
sidartaveloso
Estimates task difficulty with Jev and Laya, then benchmarks their scores against human ratings
laya-audit
Javihaus
A benchmark measures Laya's confidence calibration and compares its errors across datasets
Laya_testing
aryanlokhande1
A local Python proof of concept tests Laya customer decisioning with synthetic and historical data
jev-vs-laya
KPMGE
Compares Laya base and fine-tuned models with hosted Jev for support-message triage
pkv-laya-pilot
saski
Compares local Laya relevance ranking with lexical retrieval for a Markdown knowledge vault
laya-typed-decisions-fast
whllowp
Runs, benchmarks, and fine-tunes Laya typed-decision checkpoints on CPU
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya