Laya benchmarks
The official Laya README publishes detailed numbers, and a growing set of independent repositories now test those claims. The overall picture: Laya is much faster when run locally and free to self-host, while Jev is more accurate out of the box on most shared tests.
539 projects
LayaTrain
ceren15
Fine-tunes and benchmarks Laya for Turkish tool calling with the 118.studio tool catalog
Jev-Laya-Demo
johanlrdahl
Demonstrates Laya ticket triage against local Ollama models with typed decisions and quality reports
cctv-decision-model
jxb1st
Compares Laya-Vision and Qwen on typed anomaly decisions from CCTV frames
landing-car
fl0wo
Compares typed-decision models by using their flight-control decisions to land a simulated car
layamd
mauimauer
Serves and fine-tunes Laya decision models on AMD Strix Halo using a custom HIP engine
laya-vs-jev
codingpipeline
Compares Laya, Jev, an LLM classifier, and a TF-IDF baseline on return-request routing
arcade-decision-agent
fuleinist
Uses local Laya decisions to play browser arcade games, with Jev and OpenAI models supervising
DotaBotAI
Mohsen-Arshad
Benchmarks a synthetic Dota-style lane-control agent with optional Laya strategic advice
laya-vega-benchmark
shadowmkj
Benchmarks Laya and Vega playing Chrome Dino under real-time latency constraints
Quantlabs_G-L
rmrodriguezgit
Simulates Polymarket BTC up/down trading using GANs, Q-learning, and Laya MLX audits
laya-playground
davideconsonni
Provides a local playground for testing Laya's candidate-selection decisions and confidence
laya-africa-evals
nakisisageorge
Evaluates Laya with African-language triage datasets and a prompt-injection robustness harness
local-ai-system-one-lab
felipegenef
Teaches and tests local open-weight System One models, including Laya, through lessons and experiments
laya-local
ira-at-work
Benchmarks local Laya for coding-agent command gating and context-compaction decisions
urdu-ecommerce-sentiment-laya
raahlm
Documents Roman Urdu review sentiment research, including a fine-tuned LAYA checkpoint and evaluation
system-one-bench
DarkStark9000
Measures Laya inference latency, execution phases, GPU kernels and answer parity across backends
laya-2048
ctfpanda17
Fine-tunes Laya with imitation learning and DAgger to play 2048
laya-vs-vega
ChiragSehra
Benchmarks Laya against vegaml on Apache Jira issues
laya-vs-slm-demo
ETTE154
Compares local Laya and Llama models on multiple-choice tasks with inference metrics
jev-alternative
devrecated
Fine-tunes Laya on labelled data, calibrates probabilities, evaluates results, and provides Jev-compatible serving
laya-test
limjunda
Evaluates Laya decision-model performance on insurance cases and AI orchestrator routing
sample-laya-system-one-ecs-gpu
aws-samples
Deploys Laya on Amazon ECS GPU and compares its decisions with Amazon Nova
systemone-benchlab
lavanya1402
Benchmarks Jev and Laya as gatekeepers for AI agent actions on synthetic policy cases
retro-judge
Maverick7
A benchmark compares Laya, Double DQN, imitation learning, and a scripted agent in Space Invaders
Official numbers
The core README and its BENCHMARKS.md report latency on a Tesla T4 (32.8 ms for one question with laya-multilingual, 39.5 ms with laya) and accuracy across 51 languages. Its Jev figures are third-party published, not measured by the Laya authors.
Independent head-to-heads
- sysone-bench: byte-identical inputs across 751 states. Jev leads on triage, guardrails, moderation, Banking77 and multilingual intent. Laya leads on AG News and MNLI.
- jev-laya-benchmark: 1,470 synthetic items. Jev 92.9% vs Laya 65.3% of judgments correct. For a single question, Laya was 42 ms locally on an M3 Pro and Jev was 136 ms. Jev was faster above 3 to 4 questions per request.
- laya-jev-lab: 40 Chinese support tickets. Jev 78% at 588 ms, Laya 57% at 7.6 ms on an M4 Max. A cascade that escalates below 0.60 confidence matched Jev's accuracy at 1.8x its speed.
- JevBench: a composite board. In v1.4.2, Laya (English, CPU) ranks 41st with a score of 30.25 and Jev 1.13.0 ranks 2nd with 63.29.
- laya-mcp eval: key accuracy 32.5% for Laya vs 98.8% for Jev, with median latency of 26 ms vs 138 ms.
CPU latency
laya-cpu-benchmark measured laya-multilingual in steady state on an Intel i9-14900HX. One question took 50 ms for a 71-token state, 307 ms for 261 tokens, and 1,315 ms for 927 tokens. Batching gave little gain on CPU. Its conclusion: a realistic CPU call is 0.3 to 2 seconds, not 33 ms.
Run your own
git clone https://github.com/glukicov/laya_router && cd laya_router
uv sync --all-extras
uv run laya-router eval run --backend laya --device mps
Or use jevbench (dhruvmehra) to compare Jev, Laya, LLMs and BERT models on public datasets.
Reading these numbers
Most of these suites are small (40 to 1,500 items), synthetic or in one domain, and their authors say so. Latency depends on hardware, state length and question count. The core README's own advice: treat Laya as a fast base to specialise, and fine-tune for accuracy.
More ways to use Laya