Havuz

Network-faithful simulation of LLM serving and training deployments

An open-source inference engine for AI functions in SQL

AI/LLM on Kubernetes 基础设施知识库：涵盖 GPU、大数据、调度、分布式训练、LLM 推理、RAG、Agent 与生产运维。| AI/LLM on Kubernetes Infrastructure Knowledge Base: Covering GPUs,

Proxima lets existing GPUs serve 4x more concurrent requests

Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant k

Hikari's curated English measured notes on local LLM systems (serving, specdec, training, steering).

Thinking-native compiler for LEET Language Understanding item sets — killer-item design grammar, verification separation, deterministic gate

Plug-and-play, dependency-free web command center for DGX Spark fleets (and any GPU boxes): live per-node GPU/CPU/mem/temps/power, optional

Open-source local-first computer-use agent for Linux, Windows, and macOS, powered by local VLMs or compatible APIs.

Local-first AI coding agent runtime — your models, fail-closed safety, evidence-gated self-improvement

Zero-leak online detection of LLM decoding corruption and Free Web Search for your AI agents! Catch repetition loops, language drift and gar

Can my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

The Continuous Verification and Optimization layer for self-hosted LLMs

Live per-node GPU, host, and vLLM stats for DGX Spark clusters, in your terminal

Honeycomb Lab — hex map + OpenAI gateway control plane for a home AI fleet

Qwen3.6 vLLM Toolkit — Launcher + Templates, Optimized for 2×24 GB. vLLM launcher with an optimized hybrid chat template drawing from the be

Open-source control panel for the real llama.cpp: see where a model lands in your VRAM before loading, every llama-server flag per model, on

The open-core AI workbench — notebooks, agents, RAG, voice, and images across any model: OpenAI, Anthropic, Google, xAI, or local via Ollama

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-awa

Tenstorrent backend plugin for vLLM.

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

Turn documents into structured JSON with local-first document AI. Run 100% locally by default, with API, CLI, and Web UI.

Self-hosted vLLM made simple. Git-friendly YAML for models and profiles, full lifecycle from an intuitive CLI and TUI.

Your AI intranet: network the computers you already own for inference and training.

croit AIplane — one plane for all your AI. Self-hosted AI infrastructure connecting applications and users with models, agents, tools and en

Local-first desktop assistant for the whole job hunt — scrape listings, match with a local LLM, generate applications in your own voice, and

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cl

An autonomous agentic pipeline that finds, proves, and patches real C memory-safety vulnerabilities end-to-end using a single 7B open-source

nvtop for vLLM — an interactive terminal dashboard for vLLM serving performance (concurrency, throughput, cache & KV memory, latency, spec-d

QwenCleo-ASR is the currently State-Of-The-Art (SOTA) open-source model for Egyptian Arabic & code-switching speech recognition Built on Qwe

Hands-on GPU/HPC infrastructure operations: K8s GPU scheduling, HAMi sharing, Slurm, observability & vLLM inference. Learn it free on a lapt

Run the LLM Swarm Router on machines to distribute Local Ai to the Swarm - More Machines - MORE SPEED

Open-source local AI server configs, GFX906 runtime maintenance, reproducible benchmarks, and QC methods for affordable AI research infrastr

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Cal

Dual-engine (llama.cpp + vLLM) LLM benchmarking pipeline for GGUF & safetensors on NVIDIA GPUs — speed, quality, live dashboard, publishable

让 MinerU 在 AMD 显卡上以 vLLM + hybrid-auto-engine 满血运行！ 本项目是对官方文档中 AMD/ROCm 生态支持的有效补充。针对 MinerU 3.x 提供了一套完整的 ROCm 7.x + PyTorch 2.11.0 + vLLM 源码

An open-source, coding agent harness designed specifically for local LLMs. Designed as everything a user needs to jumpstart their own harnes

LLM Infra 工程实战 — 面向应用层工程师的 LLM 基础设施

AMD Radeon 9700 AI PRO RDNA multi GPU Tools for llm/vLLM inference, tuning, overclocking, undervolting and LLM models benchmarking

Production-ready, bleeding-edge, upstream-first vLLM Docker image for the NVIDIA DGX Spark (GB10 / sm_121a), verified against real models on

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.

Practical local LLM recipes and benchmarks for RTX 5060 Ti setups

Diagnose vLLM inference servers

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

Open-source, self-hostable social listening.

Self-hosted Chess.com alternative: Game Review for your Chess.com & Lichess games, play vs Stockfish or a friend, lessons + opening trainer,

Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/a