Laya on ONNX Runtime
ONNX is how most non-Python Laya runtimes work. An exported graph plus the tokenizer and calibration config is enough to answer typed questions with ONNX Runtime, and several projects publish ready-made exports so you can skip the PyTorch conversion step.
211 projects
laya_demo
Cokefish9527
Deploys Laya ONNX weights locally on CPU and serves decisions through an HTTP API
laya-node
roryyu
Provides a Node.js SDK for local Laya typed decisions using Transformers.js and ONNX
laya-cpp
samiul000
Implements native C++ Laya inference with ONNX Runtime, benchmarking, and INT8 quantization
vibe-check
ghobs91
Detects ragebait and spam in real time using Laya ONNX
laya-multilingual-qnn
asopitech
Runs Laya's multilingual decision model on Qualcomm Hexagon NPU through ONNX Runtime QNN
jev-systemone-local
katya4oyu
Plans a local Jev-compatible System One server with MLX, Core ML, and ONNX Laya backends
ruby_llm-providers-laya
codenamev
A RubyLLM provider uses local ONNX Laya checkpoints to answer judge questions
ruby_decision_model-providers-laya
codenamev
Adds a local ONNX-backed Laya provider to the Ruby decision-model gem
usejev
ali-master
Serves Laya through native ONNX inference on Bun with a TypeSafe-compatible API and bilingual playground
laya-universal
abusuraihsakhri
Provides cross-platform inference utilities for Laya checkpoints using ONNX Runtime and MLX
laya-cuda-bench
bhushankinge
Benchmarks Laya inference throughput, latency and cost across NVIDIA GPUs and serving backends
xcanv-models
attachemd
Provides opt-in ONNX Laya decision-model packs quantized for local inference
jev-course-demo
clchrf
Evaluates university AI course plans in the browser using quantized Laya ONNX inference
laya-ui
bharath-ui1027
Provides a multilingual Laya decision engine with UI components and optional serving integrations
omp-laya
agusbyna
An omp wrapper adds Laya decisions and prompt pre-screening to agent workflows
@johnhenry/aimatey-native-laya
johnhenry
Runs ConvAI's Laya typed-decision model on-device through a TypeScript ONNX Runtime port
@zatsepin/jigor
zatsepin
Offers a CLI and HTTP gateway for local ONNX Laya and von models and remote Jev decisions
NLaya
ljfio
Ports Laya typed-decision models to .NET with TorchSharp and ONNX Runtime
laya
archevel
Runs Laya typed-question inference in the browser using ONNX Runtime Web and WebGPU
laya-docker
hrodic
Packages the Laya decision model as an offline Docker service using ONNX Runtime
pixelkit-laya
PixelKit-Labs
Provides local typed-decision APIs and ONNX inference adapters for TypeScript and React Native
laya-lead-scoring
reislucaz
Scores CRM leads against ideal-customer criteria using calibrated Laya probabilities
mahabodi
jollypk
Combines topology memory and ONNX Runtime-based Laya decisions in a System-1 engine for AI agents
laya
HENILCHOPRA
Provides a multilingual, non-autoregressive model for typed decisions and optional serving integrations
Use a pre-converted export
inferenceprince/laya-onnx is an fp16 export of the English checkpoint. From its model card:
pip install onnxruntime huggingface_hub tokenizers numpy
hf download inferenceprince/laya-onnx --local-dir laya-onnx
The card includes a full Python example that builds the prompt, runs the graph and applies the fitted temperature. It reports load-to-first-answer of 3 to 5 s against 25 to 35 s for PyTorch on an Intel i5-14400F. The official package also has a laya[onnx] extra with an ONNXAgent class in laya.onnx_agent.
INT8 on CPU
edgejev converts Laya to ONNX, quantizes it to INT8 and serves it:
uv tool install "edgejev[build]"
edgejev build --backend laya --out ./jev-int8
Its README reports 15.6 ms per question for the INT8 multilingual model on a 4 vCPU Xeon (Cascade Lake) and 32.1 ms in fp32, with the model shrinking from 1290 MB to 324 MB. The same table shows AG News accuracy dropping from 92.8% to 91.2%.
ONNX in other languages
- Node.js: @receptron/laya, using receptron/laya-onnx.
- Java: Laya4j.
- Go: laya-onnx (MstyAI).
- Ruby: ruby-laya and laya (EduardoGHdez).
- Windows GPUs through DirectML: laya-windows.
- Local server: ollaya runs ONNX Runtime on CPU and CUDA.
- Browser: layaForWeb with ONNX Runtime Web.
Caveats
The graph returns raw logits. Divide by the fitted temperature from rl_agent_config.json before the softmax, or the probabilities mean nothing. Quantized builds drift from PyTorch: the layaForWeb README reports 97.9% top-answer agreement for its int8 build, and it notes that its confidence values are approximate because calibration was fitted on the full-precision model.
More ways to use Laya