Havuz

AI/LLM on Kubernetes 基础设施知识库：涵盖 GPU、大数据、调度、分布式训练、LLM 推理、RAG、Agent 与生产运维。| AI/LLM on Kubernetes Infrastructure Knowledge Base: Covering GPUs,

Local, offline text-to-speech and voice cloning on audio.cpp — multi-voice narration, voice design, sound/music generation, and a real-time

Run LLMs and Ollama Qwen across AMD XDNA1 NPU, CPU, and GPU on Ryzen AI Linux systems.

Dual-engine (llama.cpp + vLLM) LLM benchmarking pipeline for GGUF & safetensors on NVIDIA GPUs — speed, quality, live dashboard, publishable

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

Local inference you can prove. A single-binary GGUF engine in C: serves, scores, and trains LoRA through the quantized weights it deploys, b

Memory-aware GGUF inference runtime with OpenAI-compatible serving

Implementation of LatentMoE,Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (Elango et al., NVIDIA 2026) — in Pytorch.

CLI-native local AI image and video generation for people, scripts, and agents — CUDA on Linux, Metal on macOS, plus desktop, web, iPhone, R

ML model ports (ASR · TTS · LLM · VLM · vision · audio codecs) on the rlx multi-backend compiler + runtime — CPU/Metal/MLX/CUDA/wgpu

为 JAV 日语视频生成日文、中文或双语 SRT 字幕。支持本地音视频识别、时间轴对齐及 API 或本地模型翻译；网页界面操作，视频与音频留在本机。适用于 Windows + NVIDIA 显卡。

🔥 AI agent that ports CUDA code to AMD ROCm using local LLMs — FastAPI + React + Ollama | Live: rocmporter-agent.vercel.app

High-performance CUDA inference engine for Qwen models, written from scratch in C++/CUDA

Dashboard for InferenceX™, Open Source Continuous Inference | InferenceX 仪表板

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

ScribeCrate Live: self-hosted real-time transcription server powered by WhisperLive and faster-whisper. WebSocket transcription and speech-t

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Minimal FlashAttention in CUDA C++/CuTe: readable WMMA/CuTe kernels, no NxN workspace, up to 4.5x faster than naive PyTorch

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Native runtime, integrations, and reproducible packaging for BTL-3 and BTL-3 Compact

[ICML 2026] NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models

Unified native AI runtime for Java 17+, Java 22+ (Panama FFM), Android & C++. Run GGUF LLMs, Vision, Video, Whisper ASR, TTS & Embeddings in

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Catch local LLMs spilling into the CPU. One run shows GPU placement, prefill, decode, memory, power and a verdict.

SpeakCrate: self-hosted text-to-speech API powered by Kokoro. OpenAI-compatible speech endpoint, 50+ voices across 9 languages, streamed aud

Local AI engine in Rust: run GGUF and ONNX models for chat, vision, speech and embeddings on your own GPU, RAM and CPU, with automatic place

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

neural networks in pure C

Grammar-aware adaptive speculative decoding for structured LLM output — custom CUDA/Triton kernels, 2.7× throughput on enum-heavy schemas

This is a comprehensive tool for merging, analyzing, and training Anima models (cosmos) in a hierarchical (layer-by-layer) manner.Multilingu

Tiny engine, immense models — run large MoE LLMs (gpt-oss, Mixtral, Qwen3-MoE) on ordinary machines by streaming experts from disk. OpenAI-c

Speech-first inference runtime in Rust — TTS / ASR / speech-to-speech / VC / speaker ID / VAD. An ONNX Runtime alternative that loads GGUF &

Multi-GPU prefill acceleration for llama.cpp

The old advice to avoid Windows for local LLMs used to be right. It isn't anymore. The fixes for display-GPU desktop crashes and WSL memory

ParseCrate: self-hosted document processing API powered by Docling and docling-serve. Converts PDF, DOCX, PPTX, HTML, and more to Markdown/J

CCO is an open-source system for managing, validating, and improving GPU.

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cl