LLM Inference in Production by BentoML; "core concepts and performance metrics, to optimization techniques and operation best practices"
truespar/paddock: Native Rust inference server for open models on NVIDIA and Apple Silicon.
github.com
OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio
OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio