Havuz

A free textbook on building an LLM inference stack from scratch in Rust.

Memory-aware GGUF inference runtime with OpenAI-compatible serving

Distributed, peer-to-peer, decentralized network for LLM inference — peers seed & leech completions, metered by BitTorrent-protocol-style up

Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, re

RigMark benchmarks local AI the way coding agents actually use it.

SDK-first, independently maintained SemIf fork for fast, self-hosted semantic decisions.

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

Give Vercel AI SDK agents a real persistent filesystem with Amazon S3 Files.

Drop-in ggml backend for Rockchip NPUs: offloads llama.cpp/whisper.cpp prefill to the RK3588 NPU

Run GPT-OSS-120B on a 16 GB Mac. Streams mixture-of-experts weights from SSD so only the 2 GiB trunk stays resident. Swift + Metal, no depen

Local LLMs that run lean, load instantly and keep getting smarter. A from-scratch runtime + model format in Zig: zero-copy mmap, CPU/CUDA/Vu

LLM inference engine for AMD RDNA4 — Rust + Vulkan compute shaders, gguf & native FP8.

Transformer inference via unconventional compute: a single algebraic primitive, a tag-system scheduler, and Apple's Neural Engine through co

Runs Hugging Face ONNX models on any Nx backend, with no per-architecture code

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic 8 GB RAM path. Native C, no GP

Declarative Python SDK for building AI inference platforms on Kubernetes — bare machines to a served model, composed rather than hand-assemb

Annona — the sovereign execution kernel for AI agents. Decides where each step runs, enforces it, and records it.

Which local LLMs can your computer run? Hardware-aware CLI, TUI and GUI: yes/slow/no verdicts and tok/s from your GPU VRAM and RAM, then ver

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.

Official implementation for "K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling"

Native runtime, integrations, and reproducible packaging for BTL-3 and BTL-3 Compact

LLM Infra 工程实战 — 面向应用层工程师的 LLM 基础设施

Plug-and-play llama.cpp runtime for Intel Arc GPUs. Auto-detects your card, picks safe SYCL defaults, and exposes an OpenAI-compatible API.

Model Dispatch and Memory Allocator: an LLM inference engine for the hardware you own. Custom CPU, ROCm, Vulkan, CUDA, Metal and Level Zero

Run models too big for RAM on Apple Silicon — any Hugging Face model, dense or MoE — by streaming weights from disk

Grammar-aware adaptive speculative decoding for structured LLM output — custom CUDA/Triton kernels, 2.7× throughput on enum-heavy schemas

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable (arXiv:2606.17107) — paper, code, results, and interactive companion.

A Manim extension for animated statistical visualisations — distributions, regression, probability, inference and more.

Spark-optimized MLX LM for local inference and deployment on Apple Silicon

SDK for building on-device AI applications on Arm with reusable building blocks across inference runtimes.

[NeurIPS 2026] Official implementation of Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion

Multi-GPU prefill acceleration for llama.cpp

One-command deployment of open-source world models on AWS. Each model is a self-contained cartridge served behind a uniform API on EC2 or Sa

Proof of concept: turn old Android phones into an LLM inference cluster. ADB zero-touch provisioning, llama.cpp on each phone, OpenAI-compat

Tiny language models that never touch DRAM.

Guía gratuita de IA generativa avanzada en español (311 págs.): Transformer, post-entrenamiento, inferencia, GPU, difusión, agentes, segurid

AnyFlow - flow-matching video generation: training, inference and VBench evaluation for t2v and i2v models.

One command spins up Qwen3.8-27B (abliterated, NVFP4) on a rented Blackwell GPU at ~150 tok/s and wires it into your local coding agent via

Did your change make your LLM server faster? Measure it: faster, slower, or too close to call. Linux and Apple Silicon.

Cross-model transfer of persistent recurrent inference state without prefix replay

Unofficial .NET bindings for Google's LiteRT-LM — on-device LLM inference (Gemma) for Windows, Linux, Android, macOS & MAUI

macOS-Port von kimi-k3-in-c: Patch, Installer und Validierung für Apple Silicon. Ohne Modellgewichte.

Your AI intranet: network the computers you already own for inference and training.

Frontier: A Discrete-Event Simulator for Modern LLM Serving

llama.cpp fork that abliterates at inference

Zero-leak online detection of LLM decoding corruption and Free Web Search for your AI agents! Catch repetition loops, language drift and gar

AI inference gateway and control plane for multi-provider routing, policies, caching, tools, and observability.