Laya on OpenVINO

Install

laya-openvino is not on PyPI. Install it from GitHub (Python 3.10+):

pip install git+https://github.com/rupeshs/laya-openvino.git

The import name is still laya, so it replaces the upstream laya package if both are in the same environment.

Use the INT8 model

import laya
from huggingface_hub import snapshot_download

snapshot_download("rupeshs/laya-ov-int8", local_dir="laya-ov-int8")

agent = laya.OVAgent("laya-ov-int8")   # device="CPU" by default
answers = agent.predict(state, questions)["answers"]

rupeshs/laya-ov-int8 is 405 MB. To export and quantize yourself:

pip install nncf
python -m laya.ov export convaiinnovations/laya --out laya-ov
python research/scripts/quantize_ov.py laya-ov laya-ov-int8 --mode int8

Reported numbers

On a 12th-gen Intel CPU, the README reports one question at 136 ms with PyTorch fp32, 90 ms with the fp16 IR and 40 ms with INT8. INT8 reproduces 98.9% of the PyTorch top answers. INT4 is 243 MB but no faster than INT8, and it shifts probabilities by up to 0.69, which breaks threshold-based gating. The author recommends INT8.

A GPU is not always faster: on an i7-12700, the integrated UHD 770 took 241 ms per request against 48 ms on the CPU in the Gradio demo.

Projects

• flappy-laya-openvino-cpu: Flappy Bird played by one noul question per frame. Its README reports a mean 28 ms per decision with the INT8 IR.
• The fork's own Gradio demo (examples/app.py) has presets for sentiment, triage, phishing, LLM guard, moderation and routing.

Caveats

Only the English checkpoint is supported. The multilingual and typed-decisions checkpoints have not been exported or tested. The English model's accuracy drops sharply on other languages while its confidence stays high, so send it English input only. The project is not affiliated with Convai Innovations.