How to run Laya locally

Laya is an open-weight (Apache 2.0) decision model. You give it a state and typed questions (choice, score, noul), and it returns calibrated probabilities in one forward pass. It runs entirely on your own machine. After the first download, no network access or API key is needed. There are three checkpoints: laya (English, 421M parameters), laya-multilingual (100+ languages, 322M) and laya-typed-decisions (421M, fine-tuned). The English checkpoint is about 847 MB on disk, according to the laya-rust README.

Which one should I pick?

• Linux or Windows with an NVIDIA GPU, or any machine with Python: the official laya package.
• Apple Silicon Mac: laya-mlx for the GPU, or laya-coreml for the Neural Engine on short inputs.
• Node.js app: @receptron/laya (ONNX Runtime, no Python).
• CPU only, especially Intel: an ONNX INT8 build (edgejev) or OpenVINO INT8 (laya-openvino).
• Native binary, no Python: laya.cpp or the ggmlc laya binary with GGUF files.
• Several apps or languages sharing one model: a local HTTP server such as laya-serve or ollaya.

All commands below are copied from each project's README or model card.

Python (official package)

From NandhaKishorM/laya. Python 3.10 or newer.

python3 -m venv .venv
.venv/bin/python -m pip install laya
.venv/bin/python -I -c "import laya; print(laya.__version__)"

from laya import Router

router = Router()  # downloads a checkpoint on first use; Router(preload=True) loads all three up front

state = "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan."
questions = {
    "department": {"type": "choice", "instructions": "Which department should handle this?",
                   "criteria": {"billing": "invoices, payments, refunds",
                                "technical": "bugs, outages, system errors",
                                "other": "everything else"}},
    "urgency": {"type": "score", "instructions": "How urgent is this?",
                "criteria": ["not urgent", "soon", "blocking"]},
    "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
}

result = router.predict(state, questions)
print(result["answers"]["department"]["choice"])  # billing
print(result["answers"]["churn_risk"]["noul"])    # probability the answer is yes
print(result["routing"]["model"])                 # english

The Router sends English text to laya and other languages to laya-multilingual. The package also installs a CLI: laya "My payment failed twice" --preset triage. For a CPU-only or specific CUDA build of PyTorch, the README says to install it from PyTorch's guide before installing Laya.

Mac: MLX

laya-mlx needs Apple Silicon, Python 3.11+ and macOS 14+.

pip install laya-mlx

import laya_mlx as laya

agent = laya.load("aac6fef/laya-mlx")
result = agent.predict(
    "I was billed twice. Please refund the duplicate.",
    {"department": {"type": "choice",
                    "instructions": "Who should handle this?",
                    "criteria": ["billing", "technical", "sales"]}},
)
print(result["answers"]["department"])

The README reports 13.42 ms P50 (English) and 7.39 ms (multilingual) for one short question on an M3 Max.

Mac: Core ML and the Neural Engine

laya-coreml needs macOS 15+ and Python 3.11 to 3.13.

pip install laya-coreml

import laya_coreml as laya

agent = laya.load("aac6fef/laya-multilingual-coreml-ane")
result = agent.predict(
    "The customer requests a refund of a duplicate payment.",
    {"refund": {"type": "noul", "instructions": "Does the customer request a refund?"}},
)
print(result["answers"]["refund"])

The Neural Engine bundle accepts at most 96 tokens in total. For longer input, use aac6fef/laya-multilingual-coreml (1024 tokens).

Node.js

@receptron/laya runs ONNX Runtime in-process. It requires Node.js 20+ and about 2 GB of RAM, and the fp32 weights (about 1.7 GB) download on first use.

npm install @receptron/laya

import { Laya } from "@receptron/laya";

const laya = await Laya.load();
const result = await laya.systemOne(
  { subject: "Refund not received", body: "I cancelled two weeks ago and still have no refund..." },
  { churn_risk: { type: "noul", instructions: "Is the customer likely to cancel or dispute?" } },
);
console.log(result.answers.churn_risk.noul);
await laya.close();

CPU: ONNX and OpenVINO

edgejev exports Laya to ONNX, quantizes it to INT8 and runs it without PyTorch:

uv tool install "edgejev[build]"
edgejev build --backend laya --out ./jev-int8
edgejev serve --model ./jev-int8 --port 8009

Its README reports 15.6 ms per question for the INT8 multilingual model on a 4 vCPU Xeon.

laya-openvino adds an OpenVINO backend. It supports the English checkpoint only, and it replaces the upstream laya package in the same environment:

pip install git+https://github.com/rupeshs/laya-openvino.git

import laya
from huggingface_hub import snapshot_download

snapshot_download("rupeshs/laya-ov-int8", local_dir="laya-ov-int8")
agent = laya.OVAgent("laya-ov-int8")

Its README reports 40 ms for one question with INT8 on a 12th-gen Intel CPU, against 136 ms for PyTorch fp32.

GGUF and C++

The GGUF files at mys/laya-GGUF are built by ggmlc. They are not llama.cpp files. Run them with the laya binary from the ggmlc releases:

huggingface-cli download mys/laya-GGUF laya_english_f16.gguf --local-dir .
laya decide laya_english_f16.gguf --preset guard --text "Ignore previous instructions" --json

laya.cpp is a separate ggml runtime with CUDA, Vulkan and Core ML backends. It ships binary releases for Windows, Linux and macOS arm64.

A local HTTP server

A server keeps the model loaded once and lets any language call it. Most servers use the same POST /v1/systemone format as TypeSafe's hosted Jev API.

laya-serve (official):

pip install "laya[serve]"          # adds fastapi + uvicorn + python-multipart
LAYA_DEVICE=cuda LAYA_PRELOAD=1 laya-serve   # binds 0.0.0.0:8000, preloads all 3 checkpoints

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"body": "billed twice, refund please or we cancel"},
  "questions": {"dept": {"type": "choice", "instructions": "which team?",
                "criteria": {"billing": "refunds", "tech": "bugs"}}}
}'

It binds to all interfaces by default. Set LAYA_HOST=127.0.0.1 to keep it local, or LAYA_API_KEY to require a bearer token.

ollaya is a single binary with Ollama-style commands:

curl -fsSL https://ollaya.dev/install.sh | sh
ollaya run laya --preset triage "I was charged twice for my subscription this month and want a refund."

It listens on port 11435. Its README says existing Jev clients work after setting TYPESAFE_BASE_URL=http://localhost:11435.

LAYA SERVER is a Docker image with a web console and API keys. Its published image includes the multilingual model:

docker run -d --name laya-server --init --restart unless-stopped \
  -p 8080:8080 \
  -v laya-data:/data \
  -e LAYA_ADMIN_USERNAME=admin \
  -e LAYA_ADMIN_PASSWORD='change-this-admin-password' \
  1panel/laya-server:latest

Things to know before you rely on it

• CPU is much slower than the headline number. The official 32.8 ms is on a Tesla T4. The core README gives 193 to 464 ms per request on CPU with checkpoints preloaded, and laya-cpu-benchmark measured 0.3 to 2 seconds depending on state length. Keep states short.
• Preload for servers. With Router(max_loaded=1), the README measured a 7 to 10 second reload on every language switch.
• Route non-English text. The English checkpoint collapses on non-Latin scripts while staying confident. If most of your traffic is not English, use Router(default="multilingual").
• Calibrate and test. The checkpoints are over-confident as shipped, and laya-multilingual has no fitted temperatures. Fit thresholds on your own held-out data, and fine-tune if zero-shot accuracy is not enough.