LLM Inference in Production by BentoML; "core concepts and performance metrics, to optimization techniques and operation best practices"
Related
Introduction | LLM Inference in Production
bentoml.com · · Page
truespar/paddock: Native Rust inference server for open models on NVIDIA and Apple Silicon.
github.com · · Page
OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio
Grok
Note ·
Connect Grok to Hermes Agent
Software construction kits
Note ·
Software Construction Kits.. inspired by Folklore.org: Calculator Construction Set. See also Malleable software in the…
Extract Structured Information
Note ·
Extracting structured information from unstructured text using LLMs is a good tool to have in your llm toolkit…
AI on a budget
Note ·
What if you do not want to use AI SaaS products for any reason (monetary - subscriptions add up quick, privacy - do not…