{ "version": "1.0", "verified_at": "2026-09-18", "runtimes": [ { "id": "exo", "name": "exo", "maintainer": "exo labs", "runtime_type": "desktop_runner", "repo": "exo-explore/exo", "url": "https://github.com/exo-explore/exo", "description": "Device-mesh inference runtime that connects a user's own computers into a cluster, splitting models across devices by pipeline or tensor sharding and serving them through OpenAI-, Claude-, Ollama-, and Responses-compatible APIs behind a web dashboard.", "runtime_boundary": "Represents the exo-explore/exo software maintained by exo labs. It is not MLX or MLX distributed, the Apple inference backend and communication layer it builds on and names in its README, and not the Hugging Face hub models it downloads, which are MLX conversions such as the mlx-community checkpoints in its examples. It is not Ollama, though it serves an Ollama-compatible API path, and not a managed cloud service. Because the product exists to run models on the user's own hardware and its dashboard chat is how that capability is exercised, it classifies as a runtime rather than an assistant under ADR 015. The record is distinct from the published Exo Harness project, an unrelated agent runtime that shares only the name.", "accelerators": [ "cpu", "metal", "multi_gpu" ], "model_formats": [ "mlx" ], "serving_modes": [ "single_stream", "distributed_serving" ], "api_styles": [ "openai_compatible", "anthropic_compatible", "service_native" ], "deployment_surfaces": [ "desktop_app", "local_cli" ], "model_management": "Models are downloaded from the Hugging Face hub, either MLX conversions such as mlx-community/Llama-3.2-1B-Instruct-4bit or user-supplied custom model IDs, into EXO_DEFAULT_MODELS_DIR -- ~/.local/share/exo/models on Linux and ~/.exo/models on macOS -- with additional writable directories through EXO_MODELS_DIRS and read-only directories such as NFS mounts through EXO_MODELS_READ_ONLY_DIRS, where models cannot be deleted. EXO_OFFLINE=true confines serving to already-downloaded models. Custom models are added through the /models/add endpoint, stored as custom model cards under ~/.local/share/exo/custom_model_cards/, and gated by a trust_remote_code setting that defaults to false; the API reference documents listing models with a downloaded-status filter, searching the Hugging Face hub, and deleting custom models. Model instances are created, awaited, and deleted through the /instance endpoints.", "hardware_requirements": "Tier-1 tested and maintained support is Apple Silicon macOS only: Mac Studio M3 Ultra, Mac Mini M4 Pro, and MacBook Pro M5 and M4 Max, executing on the GPU through MLX. On Linux the README states exo currently runs on CPU only, with GPU support under development, and PLATFORMS.md lists Linux CUDA, Linux CPU, Linux Vulkan, and Windows support as planned rather than supported. The macOS app requires macOS Tahoe 26.2 or later, and RDMA over Thunderbolt 5 requires macOS 26.2 plus Thunderbolt 5 hardware with every device in the cluster cabled to every other device. Nodes discover each other over libp2p, and a coordinator-only node started with --no-worker needs no accelerator of its own.", "operational_controls": "Devices discover each other automatically, and a custom libp2p namespace through EXO_LIBP2P_NAMESPACE isolates clusters on the same network. Placement is topology-aware, with a /instance/previews endpoint that previews valid placements and reports the memory delta per node, and both pipeline and tensor sharding are documented. --no-worker runs coordinator-only nodes and --legacy-daemon enables SysV-style background operation; on Linux, configuration, data, and cache paths follow the XDG base directory specification and logs rotate automatically under ~/.cache/exo/exo_log/. Distributed tracing is toggled by EXO_TRACING_ENABLED, and the bundled exo-bench tool measures prefill and generation speed across placement configurations. The API and dashboard listen on localhost:52415 with no documented authentication, and custom Hugging Face models requiring trust_remote_code must be explicitly enabled.", "strengths": [ "The clearest documented multi-device inference cluster for consumer hardware: automatic device discovery, topology-aware pipeline and tensor sharding across hosts, and day-0 RDMA over Thunderbolt 5 support that the README describes as a 99% reduction in inter-device latency.", "Four API-compatible surfaces -- OpenAI chat completions, OpenAI responses, Claude messages, and Ollama -- let existing clients and tools including OpenWebUI talk to the cluster without adaptation.", "Model and storage control is explicit rather than implicit: default, additional writable, and read-only model directories, an offline mode, gated trust_remote_code for custom models, and cluster isolation through a libp2p namespace." ], "tradeoffs": [ "Accelerator coverage is narrow and platform-split: Apple Silicon Metal is the only supported GPU path, Linux runs on CPU, and CUDA, Vulkan, and Windows are roadmap items rather than reviewed support.", "Model format support is confined to MLX checkpoints, so GGUF, Safetensors, and other weight formats cannot be loaded directly.", "Serving is interactive and distributed rather than batched: continuous batching, parallel-request handling, and KV-cache management are not documented, and the localhost API performs no authentication.", "The macOS app demands a recent OS and installs a network profile and system-level components, while RDMA additionally requires a fully cabled Thunderbolt 5 topology with exactly matching OS versions across devices." ], "licenses": [ "Apache-2.0" ], "source_model": "open_source", "license_note": "The repository publishes a single Apache-2.0 license file at its root carrying the standard Apache License 2.0 text, and the README displays the same Apache-2.0 badge. No NOTICE file or additional component license files are published in the tree; MLX and other inference dependencies are fetched as packages rather than vendored, so no separate component terms alter the project's own license.", "license_evidence": [ { "license_id": "Apache-2.0", "scope": "Repository-wide license file covering the exo runtime, dashboard, and bindings", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/exo-explore/exo/blob/main/LICENSE", "blob_sha": "4db5f186e8470aeba22ef522826366364344114b", "immutable_url": "https://api.github.com/repos/exo-explore/exo/git/blobs/4db5f186e8470aeba22ef522826366364344114b" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 6.0, "model_format_support": 4.0, "serving_concurrency": 5.5, "api_interoperability": 8.0, "deployment_operations": 6.0, "model_lifecycle_management": 6.5, "observability_control": 6.0, "documentation_transparency": 7.5, "overall": 6.05 }, "evidence": [ { "kind": "web", "label": "README: features, configuration, and API usage", "url": "https://github.com/exo-explore/exo/blob/main/README.md", "verified_at": "2026-09-18" }, { "kind": "web", "label": "API reference for instance, model, and inference endpoints", "url": "https://github.com/exo-explore/exo/blob/main/docs/api.md", "verified_at": "2026-09-18" }, { "kind": "web", "label": "Platform support tiers and roadmap", "url": "https://github.com/exo-explore/exo/blob/main/PLATFORMS.md", "verified_at": "2026-09-18" }, { "kind": "web", "label": "Repository license file", "url": "https://github.com/exo-explore/exo/blob/main/LICENSE", "verified_at": "2026-09-18" } ], "verified_at": "2026-09-18", "stars": 47766, "stars_verified_at": "2026-10-07" }, { "id": "geniex", "name": "GenieX", "maintainer": "Qualcomm", "runtime_type": "desktop_runner", "repo": "qualcomm/GenieX", "url": "https://geniex.aihub.qualcomm.com/", "description": "On-device inference runtime for Qualcomm Snapdragon that pulls, caches, and runs language and vision-language models on the Hexagon NPU, Adreno GPU, or CPU, dispatching either to llama.cpp with Qualcomm's Hexagon backend or to the Qualcomm AI Engine Direct runtime, through a CLI, an OpenAI-compatible local server, and Python, Android, and C SDKs.", "runtime_boundary": "Represents the GenieX software runtime, not the Snapdragon silicon it targets and not Qualcomm AI Hub, the hosted model-compilation and device service from which pre-compiled bundles are fetched. The project was formerly published as NexaAI/nexa-sdk; that repository now redirects to qualcomm/GenieX, and the reviewed product name and owner are GenieX and Qualcomm, described in its own documentation as the community version of Qualcomm GENIE. It is not llama.cpp, which it vendors as a submodule and dispatches to, and not Qualcomm AI Engine Direct, the proprietary Qualcomm runtime reached through a separate plugin repository. Because the product exists to run models on the user's own device and its chat session and local server are how that capability is exercised, it classifies as a runtime rather than an assistant under ADR 015.", "accelerators": [ "cpu", "npu" ], "model_formats": [ "gguf" ], "serving_modes": [ "single_stream", "speculative_decoding" ], "api_styles": [ "openai_compatible", "service_native" ], "deployment_surfaces": [ "local_cli", "library", "container", "mobile_device" ], "model_management": "geniex pull downloads a model into a local cache from Hugging Face, Qualcomm AI Hub, Docker Hub, or a local directory or AI Hub zip archive, prompting for a GGUF precision where several exist and recording an inferred model type. geniex model list enumerates AI Hub models with a build for the detected chipset, geniex model set-type rewrites the cached manifest to switch between the language-only and multimodal load path, and geniex config get or set chipset controls chipset detection. Gated Hugging Face repositories are reached through GENIEX_HFTOKEN, HF_TOKEN, or the cached Hugging Face token file in that priority order. The local server does not auto-download; a model must be pulled first. No removal command is documented.", "hardware_requirements": "Execution is confined to Qualcomm Snapdragon; the documentation states there is no x86 or non-Snapdragon ARM build. The validated set is Snapdragon X Elite and X2 Elite on Windows ARM64, Snapdragon 8 Elite and 8 Elite Gen 5 on Android, and Dragonwing IQ-9075 and IQ-8275 on Linux ARM64, with chipset auto-detection on all three and an explicit chipset required for Qualcomm AI Hub pulls from the Android SDK. The llama.cpp runtime reaches the Hexagon NPU, the Adreno GPU through OpenCL, and the CPU, with a hybrid per-tensor Hexagon-plus-CPU scheduler and optional spreading across several HTP cores; the Qualcomm AI Engine Direct runtime is NPU-only and coerces other compute-unit requests with a warning. Python must be an ARM64 build.", "operational_controls": "A compute unit is selected per run as npu, gpu, cpu, hybrid, or an explicit HTP device list, and sampler and model flags including context length are set on the same command. Logging has six levels set by flag or the GENIEX_LOG environment variable. Speculative decoding is enabled with a draft model and draft-token bounds, and reports an accepted-draft ratio in the profiling block. The bundled server exposes a Swagger UI at its root and accepts any non-empty API key without checking it. Qualcomm AI Engine Direct bundles have precision, context length, and KV-cache size baked in and reject attempts to change them at runtime.", "strengths": [ "Two runtimes behind one interface give both breadth and depth on the same device: any community GGUF from Hugging Face runs on NPU, GPU, or CPU through llama.cpp, while Qualcomm AI Hub bundles run pre-compiled and pinned to the Hexagon NPU.", "Model lifecycle is a complete product surface, covering pull from four sources, local cache, precision selection, chipset-aware listing, manifest editing, and gated-repository authentication.", "The documentation is unusually specific about limits, naming the validated chipset list, the compute-unit alias mapping, the baked-in constraints of pre-compiled bundles, and the exact error returned when audio is sent to an unsupported runtime." ], "tradeoffs": [ "Hardware coverage is deliberately confined to Qualcomm Snapdragon, so the runtime cannot serve any other fleet, and the Qualcomm AI Engine Direct path additionally requires a bundle compiled for the specific chipset.", "Serving capability is interactive rather than concurrent: no batching, parallel-request handling, KV-cache management, or distributed serving is documented, and the server performs no authentication.", "Observability is limited to log levels, a benchmarking tool, and per-run throughput output; there is no metrics or health endpoint for an operator.", "The project is published as a developer preview, and the NPU path depends on proprietary Qualcomm runtime libraries rather than being open source end to end." ], "licenses": [ "BSD-3-Clause", "LicenseRef-Proprietary" ], "source_model": "mixed_source", "license_note": "The repository publishes a BSD-3-Clause license file covering GenieX itself, and its NOTICE file lists bundled Go, C/C++, Python, and Android dependencies under MIT, Apache-2.0, and BSD terms, none of which restrict or alter the project's own license; bindings/python/LICENSE is a symlink to the same root file. The Qualcomm AI Engine Direct execution path, however, is not open source: the geniex-qairt submodule resolves to qualcomm/geniex-qairt-plugin, whose own third-party notices state that the QAIRT headers under qnn-api/ and the prebuilt Windows, Android, and Linux runtime libraries are proprietary Qualcomm Technologies code governed by the Qualcomm AI Runtime SDK end-user licence agreement and explicitly not relicensed under BSD-3-Clause. The README also states that use of the project is subject to Qualcomm's terms of use. An open-source product built around a proprietary accelerator runtime is a mixed_source boundary. The proprietary component is vendored as a git submodule, so its notices live in a separate repository and are recorded as dated web terms rather than a pinned blob in this record's own tree.", "license_evidence": [ { "license_id": "BSD-3-Clause", "scope": "Repository-wide license file for the GenieX SDK, CLI, server, and bindings", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/qualcomm/GenieX/blob/main/LICENSE", "blob_sha": "74eadadcbcbd7754420fde09f261326df8cf2697", "immutable_url": "https://api.github.com/repos/qualcomm/GenieX/git/blobs/74eadadcbcbd7754420fde09f261326df8cf2697" }, { "license_id": "LicenseRef-Proprietary", "scope": "Qualcomm AI Runtime SDK headers and prebuilt runtime libraries that carry the NPU execution path, vendored through the geniex-qairt-plugin submodule and governed by the Qualcomm AI Runtime SDK end-user licence agreement. The submodule's own third-party notices state these files are not relicensed under BSD-3-Clause. The evidence sits outside this record's repository, so it is recorded as dated web terms rather than a pinned blob.", "kind": "web_terms", "url": "https://github.com/qualcomm/geniex-qairt-plugin/blob/main/THIRD_PARTY_NOTICES.md", "verified_at": "2026-08-30" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 4.0, "model_format_support": 6.0, "serving_concurrency": 2.5, "api_interoperability": 6.5, "deployment_operations": 5.5, "model_lifecycle_management": 7.0, "observability_control": 4.0, "documentation_transparency": 7.5, "overall": 5.17 }, "evidence": [ { "kind": "web", "label": "Repository overview, interfaces, and runtimes", "url": "https://github.com/qualcomm/GenieX/blob/main/README.md", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Product overview and architecture", "url": "https://geniex.aihub.qualcomm.com/en/get-started/what-is-geniex", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Supported chipsets, compute units, and runtime constraints", "url": "https://geniex.aihub.qualcomm.com/en/get-started/platforms", "verified_at": "2026-08-30" }, { "kind": "web", "label": "CLI command and flag reference", "url": "https://geniex.aihub.qualcomm.com/en/run/cli/reference", "verified_at": "2026-08-30" }, { "kind": "web", "label": "OpenAI-compatible local server endpoints", "url": "https://geniex.aihub.qualcomm.com/en/run/cli/local-server", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Model sources, formats, and precisions", "url": "https://geniex.aihub.qualcomm.com/en/models/supported", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Speculative decoding with a multi-token-prediction draft model", "url": "https://geniex.aihub.qualcomm.com/en/tutorials/speculative-decoding-mtp", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Repository license file", "url": "https://github.com/qualcomm/GenieX/blob/main/LICENSE", "verified_at": "2026-08-30" }, { "kind": "web", "label": "QAIRT plugin third-party notices", "url": "https://github.com/qualcomm/geniex-qairt-plugin/blob/main/THIRD_PARTY_NOTICES.md", "verified_at": "2026-08-30" } ], "verified_at": "2026-08-30", "stars": 8419, "stars_verified_at": "2026-10-07" }, { "id": "llama-cpp", "name": "llama.cpp", "maintainer": "ggml-org", "runtime_type": "embedded_library", "repo": "ggml-org/llama.cpp", "url": "https://github.com/ggml-org/llama.cpp", "description": "Dependency-free C/C++ inference implementation for language and vision-language models, consumed as a library, command-line tool, and bundled OpenAI-compatible server across a wide range of hardware backends.", "runtime_boundary": "Represents the llama.cpp project and the tools it ships. Downstream products that embed it, including Ollama, LM Studio, LocalAI, and Jan, are separate boundaries and are not represented by this record.", "accelerators": [ "cpu", "cuda", "rocm", "metal", "vulkan", "sycl", "multi_gpu" ], "model_formats": [ "gguf" ], "serving_modes": [ "single_stream", "continuous_batching", "parallel_requests", "distributed_serving", "speculative_decoding" ], "api_styles": [ "openai_compatible", "service_native" ], "deployment_surfaces": [ "library", "local_cli", "container" ], "model_management": "Models load from local GGUF files or are fetched directly from Hugging Face through a repository flag. The project provides quantization and conversion tooling rather than a managed local model store.", "hardware_requirements": "The project documents a backend matrix spanning CPU vector extensions on x86, ARM, and RISC-V alongside CUDA, HIP, Metal, Vulkan, SYCL, and further vendor backends. CPU and GPU hybrid inference partially accelerates models larger than available accelerator memory.", "operational_controls": "The bundled server exposes configurable parallel slots, continuous batching, multi-GPU split modes, a Prometheus-compatible metrics endpoint, and a slot monitoring endpoint, each settable by flag or environment variable.", "strengths": [ "Backend coverage is the broadest in the collection, spanning CPU vector extensions, CUDA, HIP, Metal, Vulkan, SYCL, and additional vendor backends.", "The bundled server documents continuous batching, configurable parallel slots, multi-GPU split modes, and Prometheus-compatible metrics.", "Distributed execution across machines is available through the project's RPC backend." ], "tradeoffs": [ "Execution is limited to the project's own GGUF format, so weights published in other formats require a conversion step.", "There is no managed local model store; model retention, naming, and cleanup are the operator's responsibility.", "Capability varies by build configuration, and several documented backends are marked in progress." ], "licenses": [ "MIT" ], "source_model": "open_source", "license_note": "The repository publishes a single MIT license file covering the inference implementation and bundled tools.", "license_evidence": [ { "license_id": "MIT", "scope": "Repository-wide license file for llama.cpp and its bundled tools", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/ggml-org/llama.cpp/blob/master/LICENSE", "blob_sha": "e7dca554bcb802f98408383a864404e3aa4eacca", "immutable_url": "https://api.github.com/repos/ggml-org/llama.cpp/git/blobs/e7dca554bcb802f98408383a864404e3aa4eacca" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 9.5, "model_format_support": 6.5, "serving_concurrency": 8.0, "api_interoperability": 8.0, "deployment_operations": 7.5, "model_lifecycle_management": 6.0, "observability_control": 7.5, "documentation_transparency": 8.0, "overall": 7.7 }, "evidence": [ { "kind": "web", "label": "Project overview and backend matrix", "url": "https://github.com/ggml-org/llama.cpp/blob/master/README.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Server flags, batching, and metrics", "url": "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Build and backend guide", "url": "https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md", "verified_at": "2026-08-29" } ], "verified_at": "2026-08-29", "stars": 130527, "stars_verified_at": "2026-10-07" }, { "id": "lm-studio", "name": "LM Studio", "maintainer": "Element Labs", "runtime_type": "desktop_runner", "repo": null, "url": "https://lmstudio.ai/docs/app", "description": "Desktop application and local server for discovering, downloading, and running language models on a personal machine, with a command-line interface and headless mode for unattended hosts.", "runtime_boundary": "Represents the local model runner and its server. Its chat window is how local execution is exercised rather than a broad conversational workspace with durable projects and connected information, so the product is curated as a runtime rather than an assistant.", "accelerators": [ "cpu", "cuda", "rocm", "metal", "vulkan" ], "model_formats": [ "gguf", "mlx" ], "serving_modes": [ "single_stream", "parallel_requests" ], "api_styles": [ "openai_compatible", "service_native" ], "deployment_surfaces": [ "desktop_app", "local_cli" ], "model_management": "Models are discovered and downloaded from Hugging Face inside the application and managed through its interface or the lms command-line tool, which also controls the daemon and server.", "hardware_requirements": "Documented requirements cover Apple silicon on macOS 14 or newer, x64 systems with AVX2 and ARM Snapdragon X Elite on Windows, and x64 or ARM64 on Ubuntu 20.04 or newer. Intel-based Macs are not supported and at least 4 GB of dedicated video memory is recommended on Windows.", "operational_controls": "Inference runtimes are selected automatically per model and can be managed and updated from a dedicated runtimes surface. The server can be started and stopped from the application or the command-line tool.", "strengths": [ "Ships llama.cpp engine variants for CPU, CUDA, Vulkan, ROCm, and Metal alongside an Apple MLX engine, selecting an engine and offload parameters automatically.", "Model discovery, download, and management are integrated into one interface with a command-line equivalent.", "Provides a headless mode for servers and continuous-integration hosts in addition to the desktop application." ], "tradeoffs": [ "The application core is proprietary; only the command-line tool and client SDKs are published under an open license.", "Execution is limited to GGUF and MLX weights, and MLX is available only on Apple silicon.", "Operational visibility is confined to in-application server logs, with no documented metrics endpoint." ], "licenses": [ "LicenseRef-Proprietary", "MIT" ], "source_model": "mixed_source", "license_note": "The application and its inference core are governed by proprietary terms that grant a non-transferable license for personal and internal business use. The separately published command-line tool and client SDKs are MIT licensed, which makes the product an open wrapper over a closed operational core.", "license_evidence": [ { "license_id": "LicenseRef-Proprietary", "scope": "LM Studio application, inference core, and documentation governed by the published terms of service", "kind": "web_terms", "url": "https://lmstudio.ai/terms", "verified_at": "2026-08-29" }, { "license_id": "MIT", "scope": "LM Studio command-line tool and client SDKs published in separate repositories", "kind": "web_terms", "url": "https://github.com/lmstudio-ai/lms/blob/main/LICENSE", "verified_at": "2026-08-29" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 9.0, "model_format_support": 7.0, "serving_concurrency": 5.5, "api_interoperability": 8.0, "deployment_operations": 6.5, "model_lifecycle_management": 8.0, "observability_control": 4.5, "documentation_transparency": 7.5, "overall": 7.05 }, "evidence": [ { "kind": "web", "label": "Application documentation", "url": "https://lmstudio.ai/docs/app", "verified_at": "2026-08-29" }, { "kind": "web", "label": "System requirements", "url": "https://lmstudio.ai/docs/app/system-requirements", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Runtime management command reference", "url": "https://lmstudio.ai/docs/cli/runtime/runtime", "verified_at": "2026-08-29" } ], "verified_at": "2026-08-29", "stars": null, "stars_verified_at": null }, { "id": "localai", "name": "LocalAI", "maintainer": "Ettore Di Giacinto and contributors", "runtime_type": "compatibility_gateway", "repo": "mudler/LocalAI", "url": "https://localai.io/docs/basics/getting_started/", "description": "Self-hosted server presenting compatible text, embedding, image, and audio APIs over interchangeable local inference backends, with a model gallery and a web interface.", "runtime_boundary": "Represents the self-hosted gateway and the backends it manages, not the individual inference engines it can load and not any hosted service.", "accelerators": [ "cpu", "cuda", "rocm", "vulkan", "sycl" ], "model_formats": [ "gguf", "safetensors" ], "serving_modes": [ "single_stream", "parallel_requests", "distributed_serving" ], "api_styles": [ "openai_compatible", "anthropic_compatible", "service_native" ], "deployment_surfaces": [ "local_cli", "container", "orchestrated_cluster" ], "model_management": "Models are installed from a built-in gallery through the web interface or the command-line tool, and backends are downloaded automatically to match detected hardware.", "hardware_requirements": "Dedicated container images cover CPU-only operation alongside NVIDIA, AMD, Intel, and Vulkan acceleration. The project documents automatic detection of available GPU capability and states that a GPU is not required.", "operational_controls": "A built-in web interface manages installations and configuration. A distributed mode registers GPU worker nodes with a front-end coordinator for scaling beyond one host.", "strengths": [ "Presents OpenAI-compatible, Anthropic Messages, and Open Responses APIs over one self-hosted endpoint, spanning text, embeddings, images, and audio.", "A model gallery plus automatic backend selection removes most manual setup for mixed hardware.", "Documents container, orchestrated, and distributed multi-node deployment paths." ], "tradeoffs": [ "Capability depends on which backend serves a given model, so behaviour is not uniform across the catalogue.", "As a gateway it inherits the limits of the engines it loads rather than optimising execution itself.", "Published operational visibility is centred on the web interface rather than a documented metrics surface." ], "licenses": [ "MIT" ], "source_model": "open_source", "license_note": "The repository publishes a single MIT license file covering the gateway.", "license_evidence": [ { "license_id": "MIT", "scope": "Repository-wide license file for the LocalAI gateway", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/mudler/LocalAI/blob/master/LICENSE", "blob_sha": "65ebf26018f7d8eefefe1741b81009daed03ff93", "immutable_url": "https://api.github.com/repos/mudler/LocalAI/git/blobs/65ebf26018f7d8eefefe1741b81009daed03ff93" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 8.5, "model_format_support": 7.5, "serving_concurrency": 7.0, "api_interoperability": 9.0, "deployment_operations": 8.5, "model_lifecycle_management": 8.0, "observability_control": 5.5, "documentation_transparency": 7.0, "overall": 7.72 }, "evidence": [ { "kind": "web", "label": "Getting started and deployment guide", "url": "https://localai.io/docs/basics/getting_started/", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Repository overview", "url": "https://github.com/mudler/LocalAI/blob/master/README.md", "verified_at": "2026-08-29" } ], "verified_at": "2026-08-29", "stars": 49415, "stars_verified_at": "2026-10-07" }, { "id": "mlc-llm", "name": "MLC LLM", "maintainer": "MLC AI community", "runtime_type": "embedded_library", "repo": "mlc-ai/mlc-llm", "url": "https://llm.mlc.ai/docs/", "description": "Machine-learning compiler and deployment engine that converts and compiles language models into platform-native binaries, then executes them on MLCEngine through Python, a command line, a REST server, and Swift and Kotlin SDKs across desktop GPUs and phones.", "runtime_boundary": "Represents the MLC LLM package, its conversion and compilation tooling, and the MLCEngine runtime, distributed as the Python package, CLI, REST server, and iOS/Android SDKs built from this repository. It is not Apache TVM, the general-purpose tensor compiler and TensorIR stack it is built on and installs as a separate dependency; TVM supplies the code generation, while this record covers the model definitions, quantization schemes, engine, and serving surfaces layered above it. It is also not WebLLM: mlc-ai/web-llm is a separately maintained, versioned, and licensed package with its own record that runs MLCEngine inside a browser through WebGPU; both consume the same compiled MLC model format, but only mlc-ai/mlc-llm's own release is scored here. The iOS and Android chat applications built from this repository are demonstrations of the SDKs, not an assistant workspace, so the project stays on the runtime side of the ADR 015 test.", "accelerators": [ "cuda", "rocm", "metal", "vulkan", "opencl", "multi_gpu" ], "model_formats": [ "mlc", "safetensors", "pytorch" ], "serving_modes": [ "single_stream", "parallel_requests", "distributed_serving", "speculative_decoding" ], "api_styles": [ "openai_compatible", "service_native" ], "deployment_surfaces": [ "local_cli", "library", "mobile_device" ], "model_management": "A model reference of the form HF://org/repo resolves against the Hugging Face Hub, downloading pre-quantized weights into a local cache and compiling a model library on first use, after which both are reused. Custom models go through mlc_llm convert_weight to quantize a Hugging Face checkpoint into the project's own sharded parameter format, mlc_llm gen_config to emit mlc-chat-config.json and process tokenizers, and mlc_llm compile to produce the platform binary; the result can be uploaded back to the Hub. Packaging tooling bundles libraries and weights into iOS and Android application builds. There is no pinning, listing, or removal command for cached artifacts.", "hardware_requirements": "The project's platform table documents AMD, NVIDIA, and Intel GPUs on Linux and Windows through Vulkan, with ROCm and CUDA as vendor backends; Apple and Intel GPUs on macOS through Metal; and Apple A-series GPUs on iOS and iPadOS through Metal, plus Adreno and Mali GPUs on Android through OpenCL. The server accepts a device string of cuda, metal, vulkan, rocm, opencl, or auto. There is no documented CPU-only execution path, and compiling for a new target requires the TVM compiler installed alongside the package.", "operational_controls": "The server takes one of three preset engine modes — interactive, local, and server — which set maximum concurrent sequences, total sequence length, and prefill chunk size, each overridable individually along with tensor-parallel shards, GPU memory utilization, prefix-cache recycling, context or sliding-window size, and attention-sink size. Speculative decoding is selected per launch as small-draft, Eagle, or Medusa with an additional draft model. Per-request event logging is enabled with a tracing flag, and CORS origins, methods, headers, and credentials are configurable. There is no authentication, metrics endpoint, or health endpoint.", "strengths": [ "One compiler and one engine target an unusually wide set of execution surfaces — desktop GPUs on three operating systems plus iOS and Android phones — with the same OpenAI-compatible interface on each.", "Compilation is a first-class, documented workflow: weight conversion, quantization configuration, model-library compilation, packaging for mobile builds, and defining a new model architecture each have their own reference page.", "The server exposes tensor-parallel sharding, prefix caching, configurable batch sizing through engine modes, and three families of speculative decoding without requiring a separate orchestrator." ], "tradeoffs": [ "Nothing runs without conversion and compilation: weights must be transformed into the project's own quantized shard format and paired with a compiled model library for the exact target, so a stock checkpoint cannot be pointed at and served directly.", "Operational visibility is minimal, with per-request tracing and a CLI statistics command but no metrics endpoint, health endpoint, authentication, container image, or orchestration guidance.", "The supported model architecture list is narrower than mainstream server engines, and the published documentation is still versioned 0.1.0 with pages of differing freshness, so capability must be confirmed per model and per release." ], "licenses": [ "Apache-2.0" ], "source_model": "open_source", "license_note": "The repository publishes a single verbatim Apache-2.0 license file covering the compiler, engine, and tooling. The accompanying NOTICE file carries only the project name and a contributor copyright line and imposes no additional terms. The Apache TVM dependency is installed separately and is not vendored into this repository.", "license_evidence": [ { "license_id": "Apache-2.0", "scope": "Repository-wide license file for the MLC LLM compiler, MLCEngine runtime, and tooling", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/mlc-ai/mlc-llm/blob/main/LICENSE", "blob_sha": "261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64", "immutable_url": "https://api.github.com/repos/mlc-ai/mlc-llm/git/blobs/261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 7.5, "model_format_support": 4.5, "serving_concurrency": 6.0, "api_interoperability": 6.5, "deployment_operations": 5.0, "model_lifecycle_management": 6.5, "observability_control": 3.5, "documentation_transparency": 5.5, "overall": 5.71 }, "evidence": [ { "kind": "web", "label": "Project overview and platform support table", "url": "https://github.com/mlc-ai/mlc-llm/blob/main/README.md", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Introduction covering the download, compile, and run workflow", "url": "https://llm.mlc.ai/docs/get_started/introduction.html", "verified_at": "2026-08-30" }, { "kind": "web", "label": "REST server flags, engine modes, and API endpoints", "url": "https://llm.mlc.ai/docs/deploy/rest.html", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Weight conversion and quantization workflow", "url": "https://llm.mlc.ai/docs/compilation/convert_weights.html", "verified_at": "2026-08-30" }, { "kind": "web", "label": "iOS Swift SDK deployment", "url": "https://llm.mlc.ai/docs/deploy/ios.html", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Android SDK deployment", "url": "https://llm.mlc.ai/docs/deploy/android.html", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Python package installation options", "url": "https://llm.mlc.ai/docs/install/mlc_llm.html", "verified_at": "2026-08-30" }, { "kind": "web", "label": "Repository license file", "url": "https://github.com/mlc-ai/mlc-llm/blob/main/LICENSE", "verified_at": "2026-08-30" } ], "verified_at": "2026-09-03", "stars": 23210, "stars_verified_at": "2026-10-07" }, { "id": "mlx-lm", "name": "MLX LM", "maintainer": "Apple machine learning research", "runtime_type": "embedded_library", "repo": "ml-explore/mlx-lm", "url": "https://github.com/ml-explore/mlx-lm", "description": "Python package for generating text with and fine-tuning language models on Apple silicon through the MLX array framework, usable as a library, command-line tool, and small local server.", "runtime_boundary": "Represents the language-model package rather than the underlying MLX array framework, and covers Apple silicon execution only.", "accelerators": [ "metal" ], "model_formats": [ "mlx", "safetensors" ], "serving_modes": [ "single_stream", "distributed_serving" ], "api_styles": [ "openai_compatible", "service_native" ], "deployment_surfaces": [ "library", "local_cli" ], "model_management": "Models are fetched from the Hugging Face Hub by identifier, and the package can quantize models and upload the result back to the Hub. Prompt caches persist to disk for reuse across generations.", "hardware_requirements": "Execution targets Apple silicon through MLX. No other accelerator or operating system is supported.", "operational_controls": "Generation and sampling parameters are set per command or per call. Distributed execution is available through the MLX distributed interface.", "strengths": [ "First-party Apple silicon execution path with quantization, upload, and low-rank or full fine-tuning in one package.", "Hugging Face Hub integration reaches a large published catalogue of MLX-compatible models with a single command.", "Usable as a Python module, a set of command-line tools, and a local server without additional components." ], "tradeoffs": [ "Apple silicon is the only supported target, so the package cannot serve mixed or non-Apple fleets.", "The bundled server is explicitly documented as not recommended for production because it implements only basic security checks.", "Serving capability is minimal, with no documented continuous batching, metrics, or administrative surface." ], "licenses": [ "MIT" ], "source_model": "open_source", "license_note": "The repository publishes a single MIT license file covering the package.", "license_evidence": [ { "license_id": "MIT", "scope": "Repository-wide license file for the MLX LM package", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/ml-explore/mlx-lm/blob/main/LICENSE", "blob_sha": "98ff47b9ef9d4ac9f1a4bde3db13dd27456c5ea1", "immutable_url": "https://api.github.com/repos/ml-explore/mlx-lm/git/blobs/98ff47b9ef9d4ac9f1a4bde3db13dd27456c5ea1" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 3.0, "model_format_support": 5.0, "serving_concurrency": 4.0, "api_interoperability": 4.5, "deployment_operations": 4.5, "model_lifecycle_management": 7.0, "observability_control": 2.5, "documentation_transparency": 6.0, "overall": 4.43 }, "evidence": [ { "kind": "web", "label": "Package overview", "url": "https://github.com/ml-explore/mlx-lm/blob/main/README.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Server documentation and limitations", "url": "https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md", "verified_at": "2026-08-29" } ], "verified_at": "2026-08-29", "stars": 7235, "stars_verified_at": "2026-10-07" }, { "id": "ollama", "name": "Ollama", "maintainer": "Ollama", "runtime_type": "desktop_runner", "repo": "ollama/ollama", "url": "https://docs.ollama.com/", "description": "Local model runner that packages model download, storage, and serving behind one CLI, desktop application, and REST API on the user's own hardware.", "runtime_boundary": "Represents the self-operated Ollama runtime. Ollama Cloud, the hosted tier operated by the same organization, is a separate managed inference service record. The model library Ollama distributes is not part of this record.", "accelerators": [ "cpu", "cuda", "rocm", "metal", "multi_gpu" ], "model_formats": [ "gguf", "safetensors" ], "serving_modes": [ "single_stream", "parallel_requests" ], "api_styles": [ "service_native", "openai_compatible", "anthropic_compatible" ], "deployment_surfaces": [ "desktop_app", "local_cli", "container" ], "model_management": "Models are pulled, pushed, created from a Modelfile, copied, listed, and deleted through the CLI and REST API. Safetensors weights and adapters can be imported into the local store.", "hardware_requirements": "Documented NVIDIA support requires compute capability 5.0 or newer with driver 550 or newer; AMD support runs through ROCm; Apple silicon runs through Metal. Models that do not fit one accelerator are spread across available GPUs, and available memory governs how many models load concurrently.", "operational_controls": "Server environment variables govern loaded-model count, parallel requests per model, and queue depth. CUDA_VISIBLE_DEVICES restricts GPU selection. Running models and versions are inspectable through the API.", "strengths": [ "Model lifecycle management is unusually complete for a local runner, covering pull, push, create, copy, import, and delete through both CLI and API.", "Documents OpenAI-compatible and Anthropic-compatible endpoints alongside its native API, plus embeddings, tool calling, structured outputs, and vision.", "Installers, container images, and hardware support are documented per platform with explicit driver and compute-capability floors." ], "tradeoffs": [ "Concurrency is limited to parallel request slots and multiple loaded models; the documentation describes no continuous batching or distributed serving.", "Documented accelerator coverage spans NVIDIA, AMD, and Apple silicon; Vulkan and Intel paths are not part of the published hardware support matrix.", "Operational visibility is limited to running-model and version endpoints with no documented metrics endpoint." ], "licenses": [ "MIT" ], "source_model": "open_source", "license_note": "The repository publishes a single MIT license file covering the runtime source.", "license_evidence": [ { "license_id": "MIT", "scope": "Repository-wide license file for the Ollama runtime", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/ollama/ollama/blob/main/LICENSE", "blob_sha": "8e3dc978a7ca8c53f56bbedc5b558116140fc02e", "immutable_url": "https://api.github.com/repos/ollama/ollama/git/blobs/8e3dc978a7ca8c53f56bbedc5b558116140fc02e" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 8.0, "model_format_support": 6.5, "serving_concurrency": 6.0, "api_interoperability": 9.0, "deployment_operations": 7.0, "model_lifecycle_management": 9.0, "observability_control": 5.0, "documentation_transparency": 8.5, "overall": 7.31 }, "evidence": [ { "kind": "web", "label": "Hardware support", "url": "https://docs.ollama.com/gpu", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Concurrency and multi-GPU behaviour", "url": "https://docs.ollama.com/faq", "verified_at": "2026-08-29" }, { "kind": "web", "label": "OpenAI compatibility", "url": "https://docs.ollama.com/api/openai-compatibility", "verified_at": "2026-08-29" }, { "kind": "web", "label": "API introduction", "url": "https://docs.ollama.com/api/introduction", "verified_at": "2026-08-29" } ], "verified_at": "2026-08-29", "stars": 182412, "stars_verified_at": "2026-10-07" }, { "id": "onnxruntime-genai", "name": "ONNX Runtime GenAI", "maintainer": "Microsoft", "runtime_type": "embedded_library", "repo": "microsoft/onnxruntime-genai", "url": "https://onnxruntime.ai/docs/genai/", "description": "Cross-platform generative inference library that implements the token generation loop for ONNX models, including pre- and post-processing, KV-cache management, sampling, constrained decoding, and adapters, consumed through Python, C, C++, C#, and Java APIs.", "runtime_boundary": "Represents the generate() API layer published as onnxruntime-genai, not ONNX Runtime itself, which is the general-purpose ONNX inference engine and execution-provider framework underneath it. Products that embed this library, including Foundry Local, Windows ML, and the Visual Studio Code AI Toolkit, are separate boundaries. The library has no server, no network endpoint, and no graphical interface; it is exercised through host application code, which places it on the runtime side of the ADR 015 runtime-versus-assistant test as an embedded library rather than an assistant.", "accelerators": [ "cpu", "cuda", "directml", "npu" ], "model_formats": [ "onnx" ], "serving_modes": [ "single_stream", "continuous_batching", "speculative_decoding" ], "api_styles": [ "service_native" ], "deployment_surfaces": [ "library" ], "model_management": "Models are ONNX directories containing a genai_config.json alongside graph and tokenizer assets, loaded from a local path; a model-package layout can bundle several hardware variants of one model and select at load time. The repository ships a model builder that converts and quantizes Hugging Face PyTorch, quantized PyTorch, and GGUF sources into the required ONNX layout, and the project documents downloading prebuilt ONNX models through the Hugging Face CLI or Foundry Local. There is no runtime model store, registry, or removal command; LoRA adapters use a documented .onnx_adapter flatbuffers file and multi-adapter switching.", "hardware_requirements": "The project's support matrix documents Linux, Windows, macOS, and Android on x86, x64, and arm64, with CPU, CUDA, DirectML, NvTensorRtRtx, OpenVINO, QNN, and WebGPU acceleration; AMD GPU support is listed as roadmap. Qualcomm Snapdragon NPU execution requires the separate onnxruntime-qnn execution-provider plugin and a model packaged with compiled QNN graph artifacts, and Python, C, C++, and C# are supported while Java requires a source build.", "operational_controls": "Search, sampling, and KV-cache behaviour are configured through genai_config.json and per-generator search options; provider options such as the Qualcomm HTP performance mode are set on the model configuration. Runtime options can terminate and restart a generation session and toggle ONNX Runtime profiling per token during generation.", "strengths": [ "One generation loop targets CPU, CUDA, DirectML, and Qualcomm NPU hardware across Windows, Linux, macOS, and Android from Python, C, C++, C#, and Java bindings.", "Constrained decoding through LLGuidance provides Lark grammar, JSON schema, and regex outputs for tool calling, alongside multi-LoRA adapters with a published adapter file specification.", "The repository documents a paged-attention engine with continuous batching, and both draft-model and n-gram speculative decoding, with explicit statements of their limits." ], "tradeoffs": [ "Execution is limited to ONNX graphs, so any other weight format must first be converted with the model builder or Olive, and per-execution-provider artifacts such as QNN context binaries are built separately.", "There is no server, endpoint, authentication, metrics, or health surface; only in-process profiling and logging, so operational visibility depends entirely on the host application.", "The continuous-batching Engine API is documented as a transitional host-facing surface that is not thread-safe, and the published support matrix still lists speculative decoding as roadmap while the repository documents it as implemented, so capability varies by version and build." ], "licenses": [ "MIT" ], "source_model": "open_source", "license_note": "The repository publishes a single MIT license file covering the generate() API implementation. A ThirdPartyNotices.txt file lists the licenses of bundled dependencies and does not restrict or alter the project's own MIT terms.", "license_evidence": [ { "license_id": "MIT", "scope": "Repository-wide license file for the ONNX Runtime GenAI library", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/microsoft/onnxruntime-genai/blob/main/LICENSE", "blob_sha": "9e841e7a26e4eb057b24511e7b92d42b257a80e5", "immutable_url": "https://api.github.com/repos/microsoft/onnxruntime-genai/git/blobs/9e841e7a26e4eb057b24511e7b92d42b257a80e5" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 7.0, "model_format_support": 5.5, "serving_concurrency": 5.5, "api_interoperability": 4.0, "deployment_operations": 5.0, "model_lifecycle_management": 6.0, "observability_control": 4.0, "documentation_transparency": 6.5, "overall": 5.46 }, "evidence": [ { "kind": "web", "label": "Project overview and support matrix", "url": "https://github.com/microsoft/onnxruntime-genai/blob/main/README.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Generate API documentation", "url": "https://onnxruntime.ai/docs/genai/", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Package installation options", "url": "https://onnxruntime.ai/docs/genai/howto/install.html", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Qualcomm NPU execution provider guide", "url": "https://github.com/microsoft/onnxruntime-genai/blob/main/docs/qnn.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Paged attention engine and continuous batching", "url": "https://github.com/microsoft/onnxruntime-genai/blob/main/docs/paged_attention_engine.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Speculative decoding methods and limits", "url": "https://github.com/microsoft/onnxruntime-genai/blob/main/docs/SpeculativeDecoding.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Model download options", "url": "https://github.com/microsoft/onnxruntime-genai/blob/main/docs/DownloadModels.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Model builder conversion and quantization", "url": "https://github.com/microsoft/onnxruntime-genai/blob/main/src/python/py/models/README.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Adapter file specification", "url": "https://onnxruntime.ai/docs/genai/reference/adapter.html", "verified_at": "2026-08-29" } ], "verified_at": "2026-08-29", "stars": 1133, "stars_verified_at": "2026-10-07" }, { "id": "openvino-model-server", "name": "OpenVINO Model Server", "maintainer": "Intel", "runtime_type": "server_engine", "repo": "openvinotoolkit/model_server", "url": "https://github.com/openvinotoolkit/model_server", "description": "C++ inference server that serves generative and classic deep-learning models compiled by OpenVINO onto Intel CPU, GPU, and NPU targets, exposing OpenAI-compatible, KServe, and TensorFlow Serving endpoints over REST and gRPC.", "runtime_boundary": "Represents the model server, not the OpenVINO toolkit it is built on: the toolkit supplies the runtime plugins, model conversion, and optimization tooling and is a separate boundary, as are the OpenVINO GenAI library and the models published in Intel's Hugging Face collections. The server is headless and operated as a container, a binary package, a Windows service, or an in-process C API host; it ships no graphical or conversational interface, which keeps it on the runtime side of the ADR 015 runtime-versus-assistant test.", "accelerators": [ "cpu", "npu", "multi_gpu" ], "model_formats": [ "onnx", "gguf" ], "serving_modes": [ "continuous_batching", "parallel_requests", "speculative_decoding" ], "api_styles": [ "openai_compatible", "service_native" ], "deployment_surfaces": [ "local_cli", "container", "orchestrated_cluster", "library" ], "model_management": "Models live in a versioned model repository on local storage, S3, Google Cloud Storage, Azure Blob, or the Hugging Face Hub. A pull mode downloads models in OpenVINO IR or GGUF form without starting the service, and a conversion path exports and quantizes PyTorch sources through optimum-cli. Version policies choose which numbered versions are served, configuration changes and new or removed models are applied at runtime without restarting, and a model cache speeds repeated loads.", "hardware_requirements": "Execution targets Intel hardware: x86 CPUs including Xeon, Intel integrated and discrete GPUs through the OpenVINO GPU plugin built on oneDNN and OpenCL, and NPU devices, on Linux and Windows as containers or binary packages. GPU and NPU use requires host drivers and, for containers, the accelerator-enabled image and device passthrough. NPU execution uses the stateful pipeline and static shapes only, so dynamic models must be reset with a fixed batch size or shape, while CPU and GPU use the continuous-batching pipeline.", "operational_controls": "The server takes device selection per model through target_device, including HETERO and AUTO virtual devices with latency, throughput, and cumulative-throughput hints. LLM graphs configure KV-cache size, maximum active sequences, dynamic split fuse, prefix caching, and cache eviction. Generative endpoints can require a bearer API key from a file or environment variable, and request limits cover execution streams, gRPC threads, REST workers, message size, and memory quota.", "strengths": [ "One server presents OpenAI-compatible chat, completions, responses, embeddings, image generation, and audio endpoints, a Cohere-compatible rerank API, and KServe and TensorFlow Serving APIs over both REST and gRPC.", "Generative serving uses continuous batching and paged attention with prefix caching, cache eviction, dynamic split fuse, and draft-model speculative decoding, and classic models run through a DAG scheduler and MediaPipe graphs.", "Operations are mature: Prometheus-compatible metrics with a documented metric list, versioned model repositories on cloud storage, runtime configuration reload, Docker, bare-metal, Windows service, Kubernetes and OpenShift deployment, and a C API for in-process embedding." ], "tradeoffs": [ "Hardware coverage is Intel-only; there is no documented CUDA, ROCm, or Metal execution path, so the server does not serve mixed-vendor accelerator fleets.", "Generative models are normally served from OpenVINO IR exported with optimum-cli, GGUF support is limited to text generation on a published model list, and NPU serving requires static shapes.", "The documentation states that gRPC and REST endpoints provide no access restrictions or traffic encryption by default, with an API key available only for the generative /v3 endpoints, so production exposure depends on a reverse proxy; scaling beyond one host is by replicas rather than multi-host model parallelism." ], "licenses": [ "Apache-2.0" ], "source_model": "open_source", "license_note": "The repository publishes a single Apache-2.0 license file covering the server, reproduced unchanged in the release package directory. A thirdparty-licenses directory carries the licenses of the components redistributed in the binary and container packages and does not restrict the server's own terms.", "license_evidence": [ { "license_id": "Apache-2.0", "scope": "Repository-wide license file for the OpenVINO Model Server", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/openvinotoolkit/model_server/blob/main/LICENSE", "blob_sha": "261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64", "immutable_url": "https://api.github.com/repos/openvinotoolkit/model_server/git/blobs/261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 5.0, "model_format_support": 7.0, "serving_concurrency": 8.0, "api_interoperability": 8.5, "deployment_operations": 8.5, "model_lifecycle_management": 7.5, "observability_control": 8.5, "documentation_transparency": 8.5, "overall": 7.54 }, "evidence": [ { "kind": "web", "label": "Project overview, features, and packages", "url": "https://github.com/openvinotoolkit/model_server/blob/main/README.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Accelerator and virtual device guide", "url": "https://github.com/openvinotoolkit/model_server/blob/main/docs/accelerators.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Metrics endpoint and metric families", "url": "https://github.com/openvinotoolkit/model_server/blob/main/docs/metrics.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Security considerations and endpoint limits", "url": "https://github.com/openvinotoolkit/model_server/blob/main/docs/security_considerations.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Pull mode for Hugging Face models", "url": "https://github.com/openvinotoolkit/model_server/blob/main/docs/pull_hf_models.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "LLM graph configuration reference", "url": "https://github.com/openvinotoolkit/model_server/blob/main/docs/llm/reference.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Server parameters including draft models and API key", "url": "https://github.com/openvinotoolkit/model_server/blob/main/docs/parameters.md", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Feature list", "url": "https://github.com/openvinotoolkit/model_server/blob/main/docs/features.md", "verified_at": "2026-08-29" } ], "verified_at": "2026-08-29", "stars": 945, "stars_verified_at": "2026-10-07" }, { "id": "sglang", "name": "SGLang", "maintainer": "SGLang project", "runtime_type": "server_engine", "repo": "sgl-project/sglang", "url": "https://docs.sglang.io/", "description": "Serving engine for language and multimodal models built around radix-tree prefix caching, continuous batching, and multi-accelerator parallelism on operator-controlled hardware.", "runtime_boundary": "Represents the self-hosted serving engine and its runtime, not the models it serves and not any managed platform that deploys it for a customer.", "accelerators": [ "cpu", "cuda", "rocm", "multi_gpu" ], "model_formats": [ "safetensors", "pytorch", "fp8", "awq", "gptq" ], "serving_modes": [ "continuous_batching", "parallel_requests", "distributed_serving", "speculative_decoding" ], "api_styles": [ "openai_compatible", "service_native" ], "deployment_surfaces": [ "library", "local_cli", "container", "orchestrated_cluster" ], "model_management": "Models load from Hugging Face repositories or local paths at launch, with adapter support. The engine does not provide a managed local model store.", "hardware_requirements": "Documented execution covers NVIDIA and AMD accelerators, Intel Xeon processors, Google TPU, and Ascend NPU targets, with published configuration recipes for specific data-centre accelerator families.", "operational_controls": "Parallelism strategy, prefix caching, and speculative decoding are configured at launch, and container images support orchestrated deployment.", "strengths": [ "Radix-tree prefix caching, continuous batching, and tensor, pipeline, and data parallelism are documented as core scheduling capabilities.", "Accelerator coverage extends beyond NVIDIA and AMD to Intel Xeon, Google TPU, and Ascend NPU targets.", "Native multimodal handling covers text, image, video, and audio alongside structured output and tool calling." ], "tradeoffs": [ "Targets accelerator-backed server deployment with no desktop packaging or graphical interface.", "Documentation is oriented toward data-centre accelerators, so consumer-hardware guidance is thinner than in desktop-first runtimes.", "Model selection is fixed at launch, and the engine provides no local model lifecycle management." ], "licenses": [ "Apache-2.0" ], "source_model": "open_source", "license_note": "The repository publishes a single Apache-2.0 license file covering the serving engine.", "license_evidence": [ { "license_id": "Apache-2.0", "scope": "Repository-wide license file for the SGLang engine", "kind": "git_blob", "path": "LICENSE", "url": "https://github.com/sgl-project/sglang/blob/main/LICENSE", "blob_sha": "9c422689c8f5c317c7c65153b1209349ec57007e", "immutable_url": "https://api.github.com/repos/sgl-project/sglang/git/blobs/9c422689c8f5c317c7c65153b1209349ec57007e" } ], "score_profile": "local_runtime", "score": { "hardware_accelerator_coverage": 8.0, "model_format_support": 8.0, "serving_concurrency": 9.0, "api_interoperability": 8.0, "deployment_operations": 8.0, "model_lifecycle_management": 6.5, "observability_control": 7.0, "documentation_transparency": 7.5, "overall": 7.86 }, "evidence": [ { "kind": "web", "label": "Project documentation overview", "url": "https://docs.sglang.io/", "verified_at": "2026-08-29" }, { "kind": "web", "label": "Repository overview", "url": "https://github.com/sgl-project/sglang/blob/main/README.md", "verified_at": "2026-08-29" } ], "verified_at": "2026-08-29", "stars": 36826, "stars_verified_at": "2026-10-07" }, { "id": "tensorflow-serving", "name": "TensorFlow Serving", "maintainer": "Google", "runtime_type": "server_engine", "repo": "tensorflow/serving", "url": "https://www.tensorflow.org/tfx/guide/serving", "description": "Production model server that loads versioned SavedModel servables from a watched model directory and answers predict, classify, regress, and metadata calls over a gRPC-first interface with a mirrored REST surface.", "runtime_boundary": "Represents the tensorflow_model_server binary and the serving libraries in the tensorflow/serving repository, not the TensorFlow framework in tensorflow/tensorflow. The framework builds and trains models and supplies the session runtime this server links against; the server's own documentation states that it deals with the inference aspect of machine learning, taking models after training and managing their lifetimes, which is the separate product this record reviews. TensorFlow Lite and LiteRT are a different execution stack for mobile and embedded targets and are out of scope here; the server carries only an experimental --prefer_tflite_model flag that picks up a model.tflite file inside a SavedModel directory, which the flag text marks as removable at any time. Managed Google Cloud offerings that host this server for customers, including Vertex AI prebuilt TensorFlow prediction containers, hold the serving contract themselves and are inference-service boundaries rather than part of this record. The server is headless, operated as a container, an APT-installed binary, or a Kubernetes deployment, and ships no graphical or conversational interface, which keeps it on the runtime side of the ADR 015 runtime-versus-assistant test.", "accelerators": [ "cpu", "cuda" ], "model_formats": [ "saved_model" ], "serving_modes": [ "single_stream", "parallel_requests" ], "api_styles": [ "service_native" ], "deployment_surfaces": [ "local_cli", "container", "orchestrated_cluster", "library" ], "model_management": "Models are staged by the operator as numbered version directories under a model_base_path; there is no fetch, pull, registry, or removal command. The server polls the filesystem on --file_system_poll_wait_seconds and loads new versions automatically. A ModelServerConfig file names multiple models and sets a model_version_policy of latest, specific versions, or all, and version_labels attach names such as stable and canary that clients address through the request ModelSpec or a /v1/models//labels/