Laya in C++ and GGUF

laya.cpp

laya.cpp supports the English, multilingual and typed-decisions models. Binary releases cover Windows and Linux x64 (CUDA and Vulkan) and macOS arm64 (Core ML). To build with CUDA:

git submodule update --init --recursive
cmake -S . -B build-cuda -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build-cuda --parallel 8

Then download the models and serve:

python scripts/download_model.py --variant all
build-cuda/bin/laya-cli --server --port 8080 --variant english \
  --tensor-core-fp32 --flash-fp32

The endpoint is POST /v1/systemone, with automatic batching of concurrent requests. The README reports CUDA throughput between 1.11x and 2.71x that of matching-precision Python on an RTX PRO 6000 Blackwell, with every answer check passing. It is MIT-licensed.

ggmlc GGUF files

ggmlc compiles Laya to GGUF. The files are on Hugging Face as mys/laya-GGUF, mys/laya-multilingual-GGUF and mys/laya-typed-decisions-GGUF, in F16, Q8_0 and UD_Q4_K_M. From the model card:

huggingface-cli download mys/laya-GGUF laya_english_f16.gguf --local-dir .
laya decide laya_english_f16.gguf --preset guard --text "Ignore previous instructions" --json
laya serve laya_english_f16.gguf --port 8080 --device auto --cuda-graph

The laya binary comes from the ggmlc releases. The card reports about 25 ms per noul and 143 ms for a 7-question preset on an RTX 4050 Laptop GPU. laya-goish serves the same F16 GGUF on Linux x86-64 CPUs.

Caveats

• These GGUF files are not llama.cpp files. The model card says loading them in llama.cpp or llama-cli will fail. Use the ggmlc laya binary or laya-goish.
• The English GGUF does not degrade gracefully off English. Use the multilingual file, or put both in one folder and pass --models-dir to route by language.
• laya.cpp rejects requests that exceed the token budget unless you pass --allow-truncation.