Laya on OpenVINO
laya-openvino is an unofficial fork of Laya that adds an OpenVINO backend for CPU inference. It exports the English checkpoint to OpenVINO IR and serves it through `OVAgent`, which returns the same payload as the PyTorch `Agent.predict`.
11 projects
laya-openvino
rupeshs
Adds an OpenVINO inference backend and serving support for Laya checkpoints
laya-ov-int8
rupeshs
An int8 OpenVINO export of Laya for CPU inference
flappy-laya-openvino-cpu
rupeshs
Runs Laya inference with OpenVINO to control a Flappy Bird game on CPU
laya-typed-decisions-ov-int8
rupeshs
Provides an int8 OpenVINO conversion of Laya fine-tuned for typed-decision workflows
laya-ov-rs
rupeshs
Provides Rust inference for Laya decision models exported to OpenVINO
laya-npu
rahmanef63
Runs Laya typed-decision models on Intel hardware and other configured endpoints
laya-snapdragon
piffie
Runs Laya typed decisions on Snapdragon X NPUs through ONNX Runtime and Qualcomm QNN
laya-web-app
meossistant
Provides an offline web runtime for Laya with OpenVINO and PyTorch backends
laya-first-try
diemrt
Finds and ranks local files with Laya, with CPU and OpenVINO inference options
laya-directml-reshape-fix
xKOKOKOx
Reproduces and verifies an ONNX Runtime DirectML Reshape workaround on a Laya multilingual export
LAYA-NPU
hellonearthis
Serves Laya-based intent classification and decision routing with Intel NPU acceleration
Install
laya-openvino is not on PyPI. Install it from GitHub (Python 3.10+):
pip install git+https://github.com/rupeshs/laya-openvino.git
The import name is still laya, so it replaces the upstream laya package if both are in the same environment.
Use the INT8 model
import laya
from huggingface_hub import snapshot_download
snapshot_download("rupeshs/laya-ov-int8", local_dir="laya-ov-int8")
agent = laya.OVAgent("laya-ov-int8") # device="CPU" by default
answers = agent.predict(state, questions)["answers"]
rupeshs/laya-ov-int8 is 405 MB. To export and quantize yourself:
pip install nncf
python -m laya.ov export convaiinnovations/laya --out laya-ov
python research/scripts/quantize_ov.py laya-ov laya-ov-int8 --mode int8
Reported numbers
On a 12th-gen Intel CPU, the README reports one question at 136 ms with PyTorch fp32, 90 ms with the fp16 IR and 40 ms with INT8. INT8 reproduces 98.9% of the PyTorch top answers. INT4 is 243 MB but no faster than INT8, and it shifts probabilities by up to 0.69, which breaks threshold-based gating. The author recommends INT8.
A GPU is not always faster: on an i7-12700, the integrated UHD 770 took 241 ms per request against 48 ms on the CPU in the Gradio demo.
Projects
- flappy-laya-openvino-cpu: Flappy Bird played by one
noulquestion per frame. Its README reports a mean 28 ms per decision with the INT8 IR. - The fork's own Gradio demo (
examples/app.py) has presets for sentiment, triage, phishing, LLM guard, moderation and routing.
Caveats
Only the English checkpoint is supported. The multilingual and typed-decisions checkpoints have not been exported or tested. The English model's accuracy drops sharply on other languages while its confidence stays high, so send it English input only. The project is not affiliated with Convai Innovations.
More ways to use Laya