Havuz

Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-com

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

JEV-inspired parallel decisions for CUDA LLMs. One context, many decisions. vLLM API, game-agent examples, and reproducible benchmarks.

CPU-only, pure-Rust implementation of LiquidAI's LFM2.5-8B-A1B LLM

Fast Gemma 4 inference in pure Java

Code samples for building AI applications on AWS with Amazon Bedrock, including Converse API, tool use, RAG with Knowledge Bases, guardrails

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

Production-ready, bleeding-edge, upstream-first vLLM Docker image for the NVIDIA DGX Spark (GB10 / sm_121a), verified against real models on

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

Frontier-class LLM inference on a laptop CPU — gpt-oss:20b at ~110 tok/s on Apple M4 Max, 7.5x llama.cpp, no GPU. From-scratch NEON/SME kern

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

CPU-optimized LLM inference engine (C)

DFlash vLLM for DGX Spark — Plug & Play Block-Diffusion Speculative Decoding

Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind

Network-faithful simulation of LLM serving and training deployments

a LLM inference engine to run on consumer hardware

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

First open-source TurboQuant KV cache compression for LLM inference. Drop-in for HuggingFace. pip install turboquant.

Run AI models on Apple Silicon and Raspberry Pi

Community benchmark database for running LLMs on Apple Silicon Macs

SGLang Docker image for AMD Strix Halo (gfx1151 / Radeon 8060S)

Serve your models fast on your own infra in 3 steps.

BOAR - Adaptive Local Intelligence

Local LLM inference engine written from scratch in Rust — hand-written AVX-512 assembly kernels, Metal & Vulkan compute shaders. Supports Qw

one-year LLM serving trace from Chutes

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embe

AI Gateway

Tu propio "ChatGPT" con todos los modelos de NaN (qwen, deepseek, gemma, mimo, whisper, kokoro), self-hosted en Docker. Pones tu API key y l

Recomputation-free, context-independent KV caching for large language models

ML model ports (ASR · TTS · LLM · VLM · vision · audio codecs) on the rlx multi-backend compiler + runtime — CPU/Metal/MLX/CUDA/wgpu

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducib

Atlas Inference is a dependency free lightweight inference engine meant to run LLMs on YOUR compute seamlessly. Supports GB10 and Strix Halo

Pure-Rust MoE inference engine for Qwen3.5/3.6 GGUF. CPU+GPU hybrid experts, NVRTC CUDA kernels, OpenAI-compatible server. 2x llama.cpp pref

WHIRL (Windows HIP Inference for RDNA LLMs): native Windows C++/HIP LLM inference engine for the AMD Ryzen AI Max+ 395 (Radeon 8060S) and Ra

qMLX: Custom inference engine for Qwen 3.5 122B on Apple Silicon, extending MLX with hybrid attention support, SSD-backed KV cache, and RYS

Compute-for-memory research lab for running larger LLMs on memory-constrained consumer GPUs. Official website exvram.pro

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

Dynamic per-token early exit for LLM inference. Skip layers tokens don't need

Qwen3.8-Flash-Next (180B MoE) on one 12-16 GB NVIDIA GPU, Linux or Windows. Own engine, own compression.

The ultimate memory backend for OpenClaw and Claude Code. Persistent conversation memory with LLM fact extraction, foundation-model rerankin

Project UAI is a universal AI agent that runs directly inside Roblox. It brings a Claude Code–style agent loop to the Roblox client, allowin

Qwen3.8-Flash-Next (125B MoE) up to 1M context on one or two RTX cards (tuned on 3080 10gb). A measured, tuned and benchmarked fork of Str

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

A web-based memory usage and performance calculator for Huggingface GGUF models

Fast WebGPU runtime for 1-bit (binary-weight) LLMs in the browser. Bit-exact, zero runtime dependencies.

neural networks in pure C

Local inference you can prove. A single-binary GGUF engine in C: serves, scores, and trains LoRA through the quantized weights it deploys, b

A free textbook on building an LLM inference stack from scratch in Rust.

A MoE inference engine you can read in an afternoon.