Havuz

GLM-5.3-Flash EXL3 (320B MoE) on 2x NVIDIA DGX Spark — production serving kit, 1M context, 97%+ multi-session prefix caching, DFlash2 spec d

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Specul

Apple Silicon mlx with Zero Dependency Java

Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4

llmBench is a high-depth benchmarking tool designed to measure the raw performance of local LLM runtimes (Ollama, llama.cpp) while providing

Enchant Version of 9Router. 304+ Providers (API-key, OAuth, free-tier, and 39 web-cookie providers), 6 combo strategies (fallback, round-rob

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

Agentic Runtime

Bounded, inspectable LLM inference pipelines from declared YAML — runs offline against Ollama or any OpenAI-compatible local server, emittin

Can my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducib

BuildArena: Physics-grounded Besiege Building for LLMs

Ollama-compatible LLM server on a tuned llama.cpp fork. Beats vllm by 40% on single stream decode, and by 20% on multi stream decode, and be

Reproduce Lium's LocalMaxxing leaderboard runs (7 boards, 20 Verified) on a rented Lium GPU in one script — vLLM/llama.cpp recipes, measured

AI inference library for mobile devices

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

Google Antigravity / Cloud Code Assist inference provider plugin for Hermes Agent

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

Local AI for your whole house: chat, images and voice on your own GPU. Computes each model's flags, fits them to your VRAM, and hot-swaps th

llambda.lisp inferencer for LLMs in GGUF format

Runtime security for AI agents. In-process, zero dependencies, Apache 2.0.

Human as Agent（人即智能体）· Human-as-LLM 人工代理网关：把工程师变成模型，OpenAI 兼容 /v1 接入 Agent 调度池，涉密/需人工任务路由给真实工程师。为 AI Agent 提效、为平台/运维收集工单。Human-as-LLM Gate

Local AI engine in Rust: run GGUF and ONNX models for chat, vision, speech and embeddings on your own GPU, RAM and CPU, with automatic place

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Distributed, peer-to-peer, decentralized network for LLM inference — peers seed & leech completions, metered by BitTorrent-protocol-style up

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, re

The agent that evolves with you 𖤍

📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

Proxima lets existing GPUs serve 4x more concurrent requests

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

SWE-Serve: an agentic benchmark of 53 production inference-engineering tasks derived from merged SGLang pull requests, run with Harbor.

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic 8 GB RAM path. Native C, no GP

Run a coding model at 300+ tok/s on your Mac. Native Rust + Metal inference engine for LiquidAI LFM2.5-24B-A2B (MLX 4-bit) on Apple Silicon.

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Neura lets you run long, tool heavy coding sessions without blowing up token costs by compressing old context into references and only resto

Open-source, local-first AI team workspace. Several AI models work together in one conversation, each with its own provider, model and tool

Diagnose vLLM inference servers

Retrieval-augmented generation: training-free document-assessment routing for cost-efficient multi-agent RAG.

What I run on my inference server: Qwen3.8-Flash-Next on two RTX 3090s, and how I got here

AI Infra 推理系统学习资料：从深度学习算子到 vLLM 服务端部署

🔥 Run large open language models on your phone. A C++20 GGUF inference engine that streams models larger than RAM from flash, paces itself b

Benchmark llama.cpp on your own GPU/CPU boxes and compare against everyone else's — sweep runner, an sha256-keyed index of ~4M GGUF files, a

Run stock llama.cpp with the stock Unsloth Qwen3.6-35B-A3B 4-bit on a 16 GB Mac.

DeepSeek-V4.1-Flash-1017 on one 256 GB M5 Ultra: 1M context, 70.7-104.9 tok/s decode with DSpark drafts, 1,739-1,867 tok/s cold prompts (8k-