Havuz

Run AI models on Apple Silicon and Raspberry Pi

Community benchmark database for running LLMs on Apple Silicon Macs

BOAR - Adaptive Local Intelligence

Local LLM inference engine written from scratch in Rust — hand-written AVX-512 assembly kernels, Metal & Vulkan compute shaders. Supports Qw

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducib

Atlas Inference is a dependency free lightweight inference engine meant to run LLMs on YOUR compute seamlessly. Supports GB10 and Strix Halo

qMLX: Custom inference engine for Qwen 3.5 122B on Apple Silicon, extending MLX with hybrid attention support, SSD-backed KV cache, and RYS

Compute-for-memory research lab for running larger LLMs on memory-constrained consumer GPUs. Official website exvram.pro

Dynamic per-token early exit for LLM inference. Skip layers tokens don't need

Qwen3.8-Flash-Next (180B MoE) on one 12-16 GB NVIDIA GPU, Linux or Windows. Own engine, own compression.

The ultimate memory backend for OpenClaw and Claude Code. Persistent conversation memory with LLM fact extraction, foundation-model rerankin

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

A web-based memory usage and performance calculator for Huggingface GGUF models

Fast WebGPU runtime for 1-bit (binary-weight) LLMs in the browser. Bit-exact, zero runtime dependencies.

neural networks in pure C

Local inference you can prove. A single-binary GGUF engine in C: serves, scores, and trains LoRA through the quantized weights it deploys, b

Memory-aware GGUF inference runtime with OpenAI-compatible serving

Distributed, peer-to-peer, decentralized network for LLM inference — peers seed & leech completions, metered by BitTorrent-protocol-style up

Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, re

RigMark benchmarks local AI the way coding agents actually use it.

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

Give Vercel AI SDK agents a real persistent filesystem with Amazon S3 Files.

Drop-in ggml backend for Rockchip NPUs: offloads llama.cpp/whisper.cpp prefill to the RK3588 NPU

Run GPT-OSS-120B on a 16 GB Mac. Streams mixture-of-experts weights from SSD so only the 2 GiB trunk stays resident. Swift + Metal, no depen

Local LLMs that run lean, load instantly and keep getting smarter. A from-scratch runtime + model format in Zig: zero-copy mmap, CPU/CUDA/Vu

Transformer inference via unconventional compute: a single algebraic primitive, a tag-system scheduler, and Apple's Neural Engine through co

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic 8 GB RAM path. Native C, no GP

Declarative Python SDK for building AI inference platforms on Kubernetes — bare machines to a served model, composed rather than hand-assemb

Annona — the sovereign execution kernel for AI agents. Decides where each step runs, enforces it, and records it.

Which local LLMs can your computer run? Hardware-aware CLI, TUI and GUI: yes/slow/no verdicts and tok/s from your GPU VRAM and RAM, then ver

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.

Official implementation for "K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling"

Native runtime, integrations, and reproducible packaging for BTL-3 and BTL-3 Compact

LLM Infra 工程实战 — 面向应用层工程师的 LLM 基础设施

Run models too big for RAM on Apple Silicon — any Hugging Face model, dense or MoE — by streaming weights from disk

Grammar-aware adaptive speculative decoding for structured LLM output — custom CUDA/Triton kernels, 2.7× throughput on enum-heavy schemas

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable (arXiv:2606.17107) — paper, code, results, and interactive companion.

SDK for building on-device AI applications on Arm with reusable building blocks across inference runtimes.

[NeurIPS 2026] Official implementation of Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion

Multi-GPU prefill acceleration for llama.cpp

Tiny language models that never touch DRAM.

Cross-model transfer of persistent recurrent inference state without prefix replay

Unofficial .NET bindings for Google's LiteRT-LM — on-device LLM inference (Gemma) for Windows, Linux, Android, macOS & MAUI

AI inference gateway and control plane for multi-provider routing, policies, caching, tools, and observability.

A free textbook on building an LLM inference stack from scratch in Rust.

AnyFlow - flow-matching video generation: training, inference and VBench evaluation for t2v and i2v models.

⚡ Ultra-Reliable AI Inference Gateway & Verifiable Agent Control Plane. Dual-Dispatch, Zero-Token JSON Auto-Healer, VaaS On-Chain Attestatio