Havuz

Local LLMs that run lean, load instantly and keep getting smarter. A from-scratch runtime + model format in Zig: zero-copy mmap, CPU/CUDA/Vu

Model Dispatch and Memory Allocator: an LLM inference engine for the hardware you own. Custom CPU, ROCm, Vulkan, CUDA, Metal and Level Zero

Open source skill library for AI coding agents to write, optimize, and debug high performance compute kernels across CUDA, Triton, and quant

A desktop application that estimates where the user is looking on the selected display and renders a red gaze marker at the estimated hit po

Crow — a local AI agent, not a chat box: 30 tools, MCP, memory, skills, goals, subagents, vision, and image generation and editing with Qwen

Production-ready, bleeding-edge, upstream-first vLLM Docker image for the NVIDIA DGX Spark (GB10 / sm_121a), verified against real models on

Local, offline text-to-speech and voice cloning on audio.cpp — multi-voice narration, voice design, sound/music generation, and a real-time

Run LLMs and Ollama Qwen across AMD XDNA1 NPU, CPU, and GPU on Ryzen AI Linux systems.

AI/LLM on Kubernetes 基础设施知识库：涵盖 GPU、大数据、调度、分布式训练、LLM 推理、RAG、Agent 与生产运维。| AI/LLM on Kubernetes Infrastructure Knowledge Base: Covering GPUs,

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

PantheonSim: GPU simulator for CUDA and ROCm. Run CUDA and HIP programs unmodified on a CPU, no GPU needed: simulated NVIDIA and AMD GPUs fo

Dual-engine (llama.cpp + vLLM) LLM benchmarking pipeline for GGUF & safetensors on NVIDIA GPUs — speed, quality, live dashboard, publishable

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

Memory-aware GGUF inference runtime with OpenAI-compatible serving

Local inference you can prove. A single-binary GGUF engine in C: serves, scores, and trains LoRA through the quantized weights it deploys, b

ML model ports (ASR · TTS · LLM · VLM · vision · audio codecs) on the rlx multi-backend compiler + runtime — CPU/Metal/MLX/CUDA/wgpu

High-performance CUDA inference engine for Qwen models, written from scratch in C++/CUDA

CLI-native local AI image and video generation for people, scripts, and agents — CUDA on Linux, Metal on macOS, plus desktop, web, iPhone, R

为 JAV 日语视频生成日文、中文或双语 SRT 字幕。支持本地音视频识别、时间轴对齐及 API 或本地模型翻译；网页界面操作，视频与音频留在本机。适用于 Windows + NVIDIA 显卡。

Implementation of LatentMoE,Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (Elango et al., NVIDIA 2026) — in Pytorch.

Dashboard for InferenceX™, Open Source Continuous Inference | InferenceX 仪表板

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

ScribeCrate Live: self-hosted real-time transcription server powered by WhisperLive and faster-whisper. WebSocket transcription and speech-t

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Minimal FlashAttention in CUDA C++/CuTe: readable WMMA/CuTe kernels, no NxN workspace, up to 4.5x faster than naive PyTorch

Native runtime, integrations, and reproducible packaging for BTL-3 and BTL-3 Compact

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

🔥 AI agent that ports CUDA code to AMD ROCm using local LLMs — FastAPI + React + Ollama | Live: rocmporter-agent.vercel.app

Operator-grade GPU monitor for NVIDIA GPUs with native GB10 / DGX Spark coherent UMA support — PSI pressure, clock detection, ConnectX-7 net

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Unified native AI runtime for Java 17+, Java 22+ (Panama FFM), Android & C++. Run GGUF LLMs, Vision, Video, Whisper ASR, TTS & Embeddings in

[ICML 2026] NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

SpeakCrate: self-hosted text-to-speech API powered by Kokoro. OpenAI-compatible speech endpoint, 50+ voices across 9 languages, streamed aud

Local AI engine in Rust: run GGUF and ONNX models for chat, vision, speech and embeddings on your own GPU, RAM and CPU, with automatic place

Catch local LLMs spilling into the CPU. One run shows GPU placement, prefill, decode, memory, power and a verdict.

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

neural networks in pure C

Grammar-aware adaptive speculative decoding for structured LLM output — custom CUDA/Triton kernels, 2.7× throughput on enum-heavy schemas

This is a comprehensive tool for merging, analyzing, and training Anima models (cosmos) in a hierarchical (layer-by-layer) manner.Multilingu

Tiny engine, immense models — run large MoE LLMs (gpt-oss, Mixtral, Qwen3-MoE) on ordinary machines by streaming experts from disk. OpenAI-c

Speech-first inference runtime in Rust — TTS / ASR / speech-to-speech / VC / speaker ID / VAD. An ONNX Runtime alternative that loads GGUF &

Multi-GPU prefill acceleration for llama.cpp

The old advice to avoid Windows for local LLMs used to be right. It isn't anymore. The fixes for display-GPU desktop crashes and WSL memory

ParseCrate: self-hosted document processing API powered by Docling and docling-serve. Converts PDF, DOCX, PPTX, HTML, and more to Markdown/J

CCO is an open-source system for managing, validating, and improving GPU.

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cl