Laya in C++ and GGUF

Two native C++ routes run Laya without Python: laya.cpp, built on ggml with CUDA, Vulkan and Core ML backends, and ggmlc, a compiler that produces GGUF files and a standalone `laya` binary. Both can serve a Jev-compatible HTTP endpoint.

71 projects

colibri

JustVugg

Runs large open models locally in pure C, including Laya for typed decisions

40.1k·GitHub repo

C / C++Runtimes & ports

laya.cpp

lkarlslund

A native C++ runtime runs Laya typed-decision models across GPU and Apple backends

89·GitHub repo

C / C++NVIDIA / CUDA

Laya-GGUF

ggml-org

Provides an automatically converted Laya GGUF model for use with llama.cpp-compatible software

18.3k·HF model

GGUFC / C++

laya-GGUF

mys

Compiles the English Laya decision model into GGUF files using ggmlc

786·HF model

GGUFC / C++

lev

jlt-commons

Implements a typed decision engine using Laya checkpoints and GGUF chat models

17·GitHub repo

JavaGGUF

laya-goish

centillex-labs

Runs Laya GGUF models in a Rust HTTP server for structured text decisions

17·GitHub repo

RustGGUF

laya-multilingual-GGUF

mys

Compiles the multilingual Laya decision model into GGUF files using ggmlc

1.9k·HF model

GGUFC / C++

laya-gguf

fr0stbit3

Converts Laya to GGUF and provides quantized weights with a separate decision head

558·HF model

GGUFC / C++

MacJev-322M-4K-Laya-GGUF

chaoliangUNSW

Provides GGUF encoder files and a runtime for running MacJev with llama.cpp

2.1k·HF model

GGUFC / C++

laya-multilingual-GGUF

Weidows

Converts the multilingual Laya model to GGUF with quantized variants

1.1k·HF model

GGUFC / C++

laya-typed-decisions-GGUF

mys

Provides GGUF builds of Laya's typed-decisions model compiled with ggmlc

604·HF model

GGUFRuntimes & ports

laya-gguf-persian-multilingual-decision

fibonacciai

A Persian-optimized multilingual GGUF conversion of the Laya typed-decision model

555·HF model

GGUFRuntimes & ports

LibLayaX

DaragonTech

An in-process C API library runs the Laya typed-decision model locally across desktop platforms

5·GitHub repo

C / C++Runtimes & ports

laya.cpp

shpati

Ports the Laya model to C and C++ for CPU inference on Linux and Windows

4·GitHub repo

C / C++Runtimes & ports

jevtpp

wiatrM

A C++20 library for typed model-backed decisions with optional ONNX Runtime and native Laya backends

3·GitHub repo

C / C++ONNX

laya-neutron-gguf

wigcheng5566

A GGUF package of Laya's encoder and decision head for CPU and Neutron NPU inference

159·HF model

GGUFC / C++

laya-multilingual-onnx

Emerald7664

CPU-optimized ONNX checkpoints for multilingual Laya decision inference

2·HF model

ONNXC / C++

openzl-laya

madeye

Adds optional on-device Laya routing to OpenZL compression on Linux and macOS

2·GitHub repo

C / C++NVIDIA / CUDA

anofox-decide

DataZooDE

A DuckDB extension evaluates natural-language predicates with local decision models or hosted providers

2·GitHub repo

C / C++Classification

laya-multilingual-GGUF

andreilungeanu

Provides quantized and full-precision Laya Multilingual GGUF files for llama.cpp

650·HF model

GGUFC / C++

laya-multilingual-F16-GGUF

meshllm

Packages Laya Multilingual as an F16 GGUF fixture for MeshLLM System One smoke tests

627·HF model

GGUFC / C++

laya-multilingual-gguf

fr0stbit3

Provides F16 and quantized GGUF conversions of the multilingual Laya model for llama.cpp

592·HF model

GGUFC / C++

laya-modernbert-decision-90pct

ameerhmz5

Fine-tunes Laya into a ModernBERT decision model and offers deployment formats

473·HF model

PythonMLX

laya-gguf

ZeroDegress

A GGUF conversion of Laya for inference with the laya-rust engine

212·HF model

GGUFRust

laya.cpp

laya.cpp supports the English, multilingual and typed-decisions models. Binary releases cover Windows and Linux x64 (CUDA and Vulkan) and macOS arm64 (Core ML). To build with CUDA:

git submodule update --init --recursive
cmake -S . -B build-cuda -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build-cuda --parallel 8

Then download the models and serve:

python scripts/download_model.py --variant all
build-cuda/bin/laya-cli --server --port 8080 --variant english \
  --tensor-core-fp32 --flash-fp32

The endpoint is POST /v1/systemone, with automatic batching of concurrent requests. The README reports CUDA throughput between 1.11x and 2.71x that of matching-precision Python on an RTX PRO 6000 Blackwell, with every answer check passing. It is MIT-licensed.

ggmlc GGUF files

ggmlc compiles Laya to GGUF. The files are on Hugging Face as mys/laya-GGUF, mys/laya-multilingual-GGUF and mys/laya-typed-decisions-GGUF, in F16, Q8_0 and UD_Q4_K_M. From the model card:

huggingface-cli download mys/laya-GGUF laya_english_f16.gguf --local-dir .
laya decide laya_english_f16.gguf --preset guard --text "Ignore previous instructions" --json
laya serve laya_english_f16.gguf --port 8080 --device auto --cuda-graph

The laya binary comes from the ggmlc releases. The card reports about 25 ms per noul and 143 ms for a 7-question preset on an RTX 4050 Laptop GPU. laya-goish serves the same F16 GGUF on Linux x86-64 CPUs.

Caveats

These GGUF files are not llama.cpp files. The model card says loading them in llama.cpp or llama-cli will fail. Use the ggmlc laya binary or laya-goish.
The English GGUF does not degrade gracefully off English. Use the multilingual file, or put both in one folder and pass --models-dir to route by language.
laya.cpp rejects requests that exceed the token budget unless you pass --allow-truncation.