Havuz

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducib

BuildArena: Physics-grounded Besiege Building for LLMs

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embe

Ollama-compatible LLM server on a tuned llama.cpp fork. Beats vllm by 40% on single stream decode, and by 20% on multi stream decode, and be

Reproduce Lium's LocalMaxxing leaderboard runs (7 boards, 20 Verified) on a rented Lium GPU in one script — vLLM/llama.cpp recipes, measured

AI inference library for mobile devices

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

Google Antigravity / Cloud Code Assist inference provider plugin for Hermes Agent

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

Local AI for your whole house: chat, images and voice on your own GPU. Computes each model's flags, fits them to your VRAM, and hot-swaps th

llambda.lisp inferencer for LLMs in GGUF format

Runtime security for AI agents. In-process, zero dependencies, Apache 2.0.

Human as Agent（人即智能体）· Human-as-LLM 人工代理网关：把工程师变成模型，OpenAI 兼容 /v1 接入 Agent 调度池，涉密/需人工任务路由给真实工程师。为 AI Agent 提效、为平台/运维收集工单。Human-as-LLM Gate

Local AI engine in Rust: run GGUF and ONNX models for chat, vision, speech and embeddings on your own GPU, RAM and CPU, with automatic place

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Distributed, peer-to-peer, decentralized network for LLM inference — peers seed & leech completions, metered by BitTorrent-protocol-style up

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, re

The agent that evolves with you 𖤍

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

Proxima lets existing GPUs serve 4x more concurrent requests

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

SWE-Serve: an agentic benchmark of 53 production inference-engineering tasks derived from merged SGLang pull requests, run with Harbor.

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic 8 GB RAM path. Native C, no GP

A static performance model for LLM inference: given (model, GPU, interconnect, workload), instantly answer VRAM feasibility, latency(TTFT, T

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Neura lets you run long, tool heavy coding sessions without blowing up token costs by compressing old context into references and only resto

Open-source, local-first AI team workspace. Several AI models work together in one conversation, each with its own provider, model and tool

Diagnose vLLM inference servers

What I run on my inference server: Qwen3.8-Flash-Next on two RTX 3090s, and how I got here

AI Infra 推理系统学习资料：从深度学习算子到 vLLM 服务端部署

Benchmark llama.cpp on your own GPU/CPU boxes and compare against everyone else's — sweep runner, an sha256-keyed index of ~4M GGUF files, a

The Continuous Verification and Optimization layer for self-hosted LLMs

single-executable / library which combines llama.cpp, whisper.cpp, and stable-diffusion.cpp

MoE expert compression toolkit — prune, merge, and forge specialist models for local inference.

High-performance CUDA inference engine for Qwen models, written from scratch in C++/CUDA

Typed decisions from unstructured state: one forward pass, zero generated text. Local, deterministic, Jev-compatible. Not affiliated with ty

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cl

Serve DeepSeek and Qwen, run an on-demand model library, and fine-tune with Unsloth QLoRA on 2x NVIDIA DGX Spark. TP=2 vLLM lanes behind one