Havuz

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimiz

Inference-native Tokenmaxxing Agent Harness for Loop Engineering

Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark,

Run any local LLM engine, auto-tuned to your GPU — polished web UI + OpenAI/Anthropic-compatible API. Point Claude Code at your own machine

Your AI intranet: network the computers you already own for inference and training.

The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLL

Camelid: a Rust-native local inference backend with evidence-gated model compatibility.

Evidence-aware on-prem LLM inference sizing and TCO calculator

Provider-pluggable orchestration runtime for multi-model AI inference. ( Sakana Fugu style )

Frontier: A Discrete-Event Simulator for Modern LLM Serving

Zero-leak online detection of LLM decoding corruption and Free Web Search for your AI agents! Catch repetition loops, language drift and gar

CPU-first LLM inference server on llama.cpp. Runs useful models on free-tier ARM boxes; rewriting the input made it ~6x faster and more accu

Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGU

Tree-based speculative decoding for Apple Silicon (MLX). ~10-15% faster than DFlash on code, ~1.5x over autoregressive. First MLX port with

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-com

Network-faithful simulation of LLM serving and training deployments

Open-source self-hosted home AI inference platform for AMD Strix Halo — multi-backend slots, OpenAI-compatible gateway, Vue 3 + FastAPI + sy

Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind

CPU-only, pure-Rust implementation of LiquidAI's LFM2.5-8B-A1B LLM

Frontier-class LLM inference on a laptop CPU — gpt-oss:20b at ~110 tok/s on Apple M4 Max, 7.5x llama.cpp, no GPU. From-scratch NEON/SME kern

CPU-optimized LLM inference engine (C)

Production-ready, bleeding-edge, upstream-first vLLM Docker image for the NVIDIA DGX Spark (GB10 / sm_121a), verified against real models on

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embe

qMLX: Custom inference engine for Qwen 3.5 122B on Apple Silicon, extending MLX with hybrid attention support, SSD-backed KV cache, and RYS

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

Run AI models on Apple Silicon and Raspberry Pi

DFlash vLLM for DGX Spark — Plug & Play Block-Diffusion Speculative Decoding

SGLang Docker image for AMD Strix Halo (gfx1151 / Radeon 8060S)

ML model ports (ASR · TTS · LLM · VLM · vision · audio codecs) on the rlx multi-backend compiler + runtime — CPU/Metal/MLX/CUDA/wgpu

Community benchmark database for running LLMs on Apple Silicon Macs

Recomputation-free, context-independent KV caching for large language models