Havuz

Memory-aware GGUF inference runtime with OpenAI-compatible serving

SGLang Docker image for AMD Strix Halo (gfx1151 / Radeon 8060S)

ML model ports (ASR · TTS · LLM · VLM · vision · audio codecs) on the rlx multi-backend compiler + runtime — CPU/Metal/MLX/CUDA/wgpu

Distributed, peer-to-peer, decentralized network for LLM inference — peers seed & leech completions, metered by BitTorrent-protocol-style up

Fast WebGPU runtime for 1-bit (binary-weight) LLMs in the browser. Bit-exact, zero runtime dependencies.

Tiny language models that never touch DRAM.

Spark-optimized MLX LM for local inference and deployment on Apple Silicon

Cross-model transfer of persistent recurrent inference state without prefix replay

Drop-in ggml backend for Rockchip NPUs: offloads llama.cpp/whisper.cpp prefill to the RK3588 NPU

Which local LLMs can your computer run? Hardware-aware CLI, TUI and GUI: yes/slow/no verdicts and tok/s from your GPU VRAM and RAM, then ver

Community benchmark database for running LLMs on Apple Silicon Macs

a LLM inference engine to run on consumer hardware

First open-source TurboQuant KV cache compression for LLM inference. Drop-in for HuggingFace. pip install turboquant.

Recomputation-free, context-independent KV caching for large language models

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

Annona — the sovereign execution kernel for AI agents. Decides where each step runs, enforces it, and records it.

Local LLM inference engine written from scratch in Rust — hand-written AVX-512 assembly kernels, Metal & Vulkan compute shaders. Supports Qw

One command spins up Qwen3.8-27B (abliterated, NVFP4) on a rented Blackwell GPU at ~150 tok/s and wires it into your local coding agent via

Native runtime, integrations, and reproducible packaging for BTL-3 and BTL-3 Compact

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

SDK for building on-device AI applications on Arm with reusable building blocks across inference runtimes.

Tenstorrent backend plugin for vLLM.

A web-based memory usage and performance calculator for Huggingface GGUF models

The ultimate memory backend for OpenClaw and Claude Code. Persistent conversation memory with LLM fact extraction, foundation-model rerankin

Dynamic per-token early exit for LLM inference. Skip layers tokens don't need

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

neural networks in pure C

macOS-Port von kimi-k3-in-c: Patch, Installer und Validierung für Apple Silicon. Ohne Modellgewichte.

OpenMayhem is an open-source and decentralized inference router for consumer & pro hardware. Providers earn, users/agents access cheap infer

Run models too big for RAM on Apple Silicon — any Hugging Face model, dense or MoE — by streaming weights from disk

Grammar-aware adaptive speculative decoding for structured LLM output — custom CUDA/Triton kernels, 2.7× throughput on enum-heavy schemas

Multi-model orchestration engine. Combine any LLM providers (Gemini, OpenAI, Anthropic, Groq, local GGUF) into a single Ollama-compatible AP

LLM inference engine for AMD RDNA4 — Rust + Vulkan compute shaders, gguf & native FP8.

Official implementation for "K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling"

Transformer inference via unconventional compute: a single algebraic primitive, a tag-system scheduler, and Apple's Neural Engine through co

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.

LLM Infra 工程实战 — 面向应用层工程师的 LLM 基础设施

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable (arXiv:2606.17107) — paper, code, results, and interactive companion.

Plug-and-play llama.cpp runtime for Intel Arc GPUs. Auto-detects your card, picks safe SYCL defaults, and exposes an OpenAI-compatible API.

Multi-GPU prefill acceleration for llama.cpp

[NeurIPS 2026] Official implementation of Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion

Unofficial .NET bindings for Google's LiteRT-LM — on-device LLM inference (Gemma) for Windows, Linux, Android, macOS & MAUI