Laya on ONNX Runtime
ONNX is how most non-Python Laya runtimes work. An exported graph plus the tokenizer and calibration config is enough to answer typed questions with ONNX Runtime, and several projects publish ready-made exports so you can skip the PyTorch conversion step.
211 projects
laya-js
abhimanyupandian
Packages Laya for local inference in browsers and Node.js using ONNX Runtime
laya-mbti
shibadogcap
Estimates MBTI traits from X archives using Laya in a client-side web app
workwe-laya
WorkWeonline
Provides a multilingual System 1 decision engine with typed outputs and optional serving and integration extras
laya-node-server
localhost-anon
A fully offline Node.js server and playground runs Laya through ONNX Runtime
laya-mirror-541
Shijiuwei
Mirrors the Laya project and documents high-availability and network topology specifications
Laya-Android-Coin
yvyvyvbing
Packages Laya as an offline Android decision app using quantized ONNX and a WebAssembly runtime
@wexare/laya-web
sarozarena
Runs the Laya typed-decision model in browsers using ONNX Runtime Web
LayaPL
pleveneur
Runs locally converted Laya ONNX models from Node.js using ONNX Runtime
laya-multilingual-onnx
Youssful
Provides an ONNX Laya model and tokenizer for SupremeChatModeration
laya-ts-example
poad
A local web playground runs Laya through a Node.js API server using ONNX Runtime
laya-onnx
D-Ajay-Kumar
Converts Laya weights to ONNX and benchmarks the converted model
voice-assistant-laya
Joshua265
A prototype Wayland voice assistant uses an ONNX Laya model to decide desktop actions
laya_demo_test
j75689
A Snake demo compares Python and Rust Laya runners with ONNX Runtime and Core ML backends
portable-coder-models
sndstone
Distributes licensed ONNX model packs for the Portable Coding Tool installer
laya-multilingual-webapp
realizer-1966
Runs the multilingual Laya decision engine as a browser PWA using ONNX Runtime Web
laya-onnx-android
rookery-labs
Runs Laya as an Android System-1 decision engine with ONNX Mobile and a Rust JNI tokenizer
openjevx
muthuishere
Runs an ONNX Jev-compatible decision model fine-tuned from Laya using RLCD
laya-scam-detector
chaceli
Fine-tunes and serves Laya as an ONNX scam-phrase detector with a FastAPI playground
laya-trt-serve
nmcglo
Exports Laya to TensorRT and serves it through a Jev-compatible API
laya-intent-router
vrajnotviraj
Provides an ONNX intent router distilled from Laya, with training and inference code
laya-local-serve
yyong-brs
Runs Laya locally with Python, TypeScript, and integrations for serving and decision workflows
laya-on-aws-lambda
gunnargrosch
Deploys Laya as a Jev-compatible decision API on AWS Lambda using TypeScript or Python
system-one-on-snapdragon
esterhuizen
Setup scripts and benchmarks run open typed-decision models, including Laya, on Snapdragon hardware
@banh-dev/laya
banh-kevin
Native Laya ONNX inference package for the Banh workflow CLI
Use a pre-converted export
inferenceprince/laya-onnx is an fp16 export of the English checkpoint. From its model card:
pip install onnxruntime huggingface_hub tokenizers numpy
hf download inferenceprince/laya-onnx --local-dir laya-onnx
The card includes a full Python example that builds the prompt, runs the graph and applies the fitted temperature. It reports load-to-first-answer of 3 to 5 s against 25 to 35 s for PyTorch on an Intel i5-14400F. The official package also has a laya[onnx] extra with an ONNXAgent class in laya.onnx_agent.
INT8 on CPU
edgejev converts Laya to ONNX, quantizes it to INT8 and serves it:
uv tool install "edgejev[build]"
edgejev build --backend laya --out ./jev-int8
Its README reports 15.6 ms per question for the INT8 multilingual model on a 4 vCPU Xeon (Cascade Lake) and 32.1 ms in fp32, with the model shrinking from 1290 MB to 324 MB. The same table shows AG News accuracy dropping from 92.8% to 91.2%.
ONNX in other languages
- Node.js: @receptron/laya, using receptron/laya-onnx.
- Java: Laya4j.
- Go: laya-onnx (MstyAI).
- Ruby: ruby-laya and laya (EduardoGHdez).
- Windows GPUs through DirectML: laya-windows.
- Local server: ollaya runs ONNX Runtime on CPU and CUDA.
- Browser: layaForWeb with ONNX Runtime Web.
Caveats
The graph returns raw logits. Divide by the fitted temperature from rl_agent_config.json before the softmax, or the probabilities mean nothing. Quantized builds drift from PyTorch: the layaForWeb README reports 97.9% top-answer agreement for its int8 build, and it notes that its confidence values are approximate because calibration was fitted on the full-precision model.
More ways to use Laya