Local runtime · Embedded library
ONNX Runtime GenAI
Cross-platform generative inference library that implements the token generation loop for ONNX models, including pre- and post-processing, KV-cache management, sampling, constrained decoding, and adapters, consumed through Python, C, C++, C#, and Java APIs.
- Maintainer
- Microsoft
- Accelerators
- CPU · NVIDIA CUDA · DirectML · NPU
- Model formats
- ONNX
- Source model
- Open source
- Licenses
- MIT License
- Boundary
- Represents the generate() API layer published as onnxruntime-genai, not ONNX Runtime itself, which is the general-purpose ONNX inference engine and execution-provider framework underneath it. Products that embed this library, including Foundry Local, Windows ML, and the Visual Studio Code AI Toolkit, are separate boundaries. The library has no server, no network endpoint, and no graphical interface; it is exercised through host application code, which places it on the runtime side of the ADR 015 runtime-versus-assistant test as an embedded library rather than an assistant.
Open in the directory → Open official documentation ↗ Repository ↗
Editorial ratings appear in the directory beside the profile they belong to and are never compared across collections.