Laya on ONNX Runtime
ONNX is how most non-Python Laya runtimes work. An exported graph plus the tokenizer and calibration config is enough to answer typed questions with ONNX Runtime, and several projects publish ready-made exports so you can skip the PyTorch conversion step.
211 projects
laya
NandhaKishorM
Runs multilingual typed-decision models and provides routing, serving, and integration options
laya
receptron
Runs Laya typed-decision models from Node.js and TypeScript through ONNX Runtime
layaForWeb
vishalmysore
Runs a quantized ONNX version of the Laya decision model entirely in the browser
laya-onnx
Mattepiu
Exports a fine-tuned Laya decision model to ONNX for Python and Node.js inference
ruby-laya
codenamev
Provides a Ruby port of Laya with typed decisions, checkpoint routing, and ONNX Runtime inference
edgejev
yzfly
Exports, quantizes, and serves Laya and other Jev-style models for offline CPU inference
open-jev-laya
killkli
Runs Laya multilingual ONNX inference in browser-based decision demos
laya-rs
redwolf2019
Provides a Rust HTTP inference runtime and server for multilingual Laya models
WechatVibe
tswawa
Analyzes WeChat conversations for intent, emotion, and participant profiles using Laya models
Laya4j
githubMJ
A Java 17+ SDK runs Laya typed-decision models through ONNX Runtime
laya-multilingual-ONNX
onnx-community
Provides an ONNX conversion of Laya multilingual for use with Transformers.js
laya-onnx
techtheist
Provides quantized ONNX exports of English and multilingual Laya checkpoints
laya-onnx
receptron
Exports Laya to ONNX for use with Node.js or ONNX Runtime
laya-onnx
MstyAI
Runs Laya decision models locally with ONNX Runtime through a Go library and CLI
laya-multilingual-onnx
mizchi
Exports multilingual Laya to ONNX for native and browser WebGPU inference
laya-multilingual-onnx-int4
androidli
Provides 4-bit weight-only ONNX quantizations of Laya’s multilingual decision model
laya-onnx
gqgs
Exports the Laya model to quantized ONNX for browser inference
kevin
gauravsaini
Runs a browser automation agent that uses Laya-ONNX to rank candidate actions
laya-web
r4ai
Runs Laya typed-decision models in browsers and Node.js using ONNX Runtime Web
laya-windows
Zuhair-01
Ports Laya typed-decision inference to Windows using ONNX Runtime and DirectML
jevtpp
wiatrM
A C++20 library for typed model-backed decisions with optional ONNX Runtime and native Laya backends
shed
IAmUnbounded
Provides remote access to desktop coding-agent sessions with local Laya-based routing
duckdb-jev-spatial
mraad
Adds DuckDB Jev-style SQL functions backed by a local Laya model for spatial examples
laya-onnx
inferenceprince
Exports Laya to ONNX for inference with ONNX Runtime on CPUs, GPUs, and browsers
Use a pre-converted export
inferenceprince/laya-onnx is an fp16 export of the English checkpoint. From its model card:
pip install onnxruntime huggingface_hub tokenizers numpy
hf download inferenceprince/laya-onnx --local-dir laya-onnx
The card includes a full Python example that builds the prompt, runs the graph and applies the fitted temperature. It reports load-to-first-answer of 3 to 5 s against 25 to 35 s for PyTorch on an Intel i5-14400F. The official package also has a laya[onnx] extra with an ONNXAgent class in laya.onnx_agent.
INT8 on CPU
edgejev converts Laya to ONNX, quantizes it to INT8 and serves it:
uv tool install "edgejev[build]"
edgejev build --backend laya --out ./jev-int8
Its README reports 15.6 ms per question for the INT8 multilingual model on a 4 vCPU Xeon (Cascade Lake) and 32.1 ms in fp32, with the model shrinking from 1290 MB to 324 MB. The same table shows AG News accuracy dropping from 92.8% to 91.2%.
ONNX in other languages
- Node.js: @receptron/laya, using receptron/laya-onnx.
- Java: Laya4j.
- Go: laya-onnx (MstyAI).
- Ruby: ruby-laya and laya (EduardoGHdez).
- Windows GPUs through DirectML: laya-windows.
- Local server: ollaya runs ONNX Runtime on CPU and CUDA.
- Browser: layaForWeb with ONNX Runtime Web.
Caveats
The graph returns raw logits. Divide by the fitted temperature from rl_agent_config.json before the softmax, or the probabilities mean nothing. Quantized builds drift from PyTorch: the layaForWeb README reports 97.9% top-answer agreement for its int8 build, and it notes that its confidence values are approximate because calibration was fitted on the full-precision model.
More ways to use Laya