Havuz

为 JAV 日语视频生成日文、中文或双语 SRT 字幕。支持本地音视频识别、时间轴对齐及 API 或本地模型翻译；网页界面操作，视频与音频留在本机。适用于 Windows + NVIDIA 显卡。

Qwen Image 2.1 for DeepSeek Harness: native image editing on Apple Silicon MLX/MFLUX or self-hosted CUDA, LoRA, ControlNet, inpainting, pres

ML model ports (ASR · TTS · LLM · VLM · vision · audio codecs) on the rlx multi-backend compiler + runtime — CPU/Metal/MLX/CUDA/wgpu

ScribeCrate Live: self-hosted real-time transcription server powered by WhisperLive and faster-whisper. WebSocket transcription and speech-t

Crow — a local AI agent, not a chat box: 30 tools, MCP, memory, skills, goals, subagents, vision, and image generation and editing with Qwen

Ollama-compatible LLM server on a tuned llama.cpp fork. Beats vllm by 40% on single stream decode, and by 20% on multi stream decode, and be

Pure-Rust MoE inference engine for Qwen3.5/3.6 GGUF. CPU+GPU hybrid experts, NVRTC CUDA kernels, OpenAI-compatible server. 2x llama.cpp pref

Operator-grade GPU monitor for NVIDIA GPUs with native GB10 / DGX Spark coherent UMA support — PSI pressure, clock detection, ConnectX-7 net

Compute-for-memory research lab for running larger LLMs on memory-constrained consumer GPUs. Official website exvram.pro

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

Qwen3.8-Flash-Next (180B MoE) on one 12-16 GB NVIDIA GPU, Linux or Windows. Own engine, own compression.

Qwen3.8-Flash-Next (125B MoE) up to 1M context on one or two RTX cards (tuned on 3080 10gb). A measured, tuned and benchmarked fork of Str

A realtime assistant for Mac and Windows. Talk to it and it answers aloud, shows your schedule and mail as cards, and hands longer work to a

Run LLMs and Ollama Qwen across AMD XDNA1 NPU, CPU, and GPU on Ryzen AI Linux systems.

SpeakCrate: self-hosted text-to-speech API powered by Kokoro. OpenAI-compatible speech endpoint, 50+ voices across 9 languages, streamed aud

AI/LLM on Kubernetes 基础设施知识库：涵盖 GPU、大数据、调度、分布式训练、LLM 推理、RAG、Agent 与生产运维。| AI/LLM on Kubernetes Infrastructure Knowledge Base: Covering GPUs,

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

Minimal FlashAttention in CUDA C++/CuTe: readable WMMA/CuTe kernels, no NxN workspace, up to 4.5x faster than naive PyTorch

neural networks in pure C

Local inference you can prove. A single-binary GGUF engine in C: serves, scores, and trains LoRA through the quantized weights it deploys, b

Local AI engine in Rust: run GGUF and ONNX models for chat, vision, speech and embeddings on your own GPU, RAM and CPU, with automatic place

Memory-aware GGUF inference runtime with OpenAI-compatible serving

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

[ICML 2026] NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models

🔥 AI agent that ports CUDA code to AMD ROCm using local LLMs — FastAPI + React + Ollama | Live: rocmporter-agent.vercel.app

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Implementation of LatentMoE,Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (Elango et al., NVIDIA 2026) — in Pytorch.

Local LLMs that run lean, load instantly and keep getting smarter. A from-scratch runtime + model format in Zig: zero-copy mmap, CPU/CUDA/Vu

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

🔥🔥 Accelerating decision inference with exact state forking

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Windows utility focused on cuda setup, nvidia local AI, and cuda environment for local AI workflows.

This is a comprehensive tool for merging, analyzing, and training Anima models (cosmos) in a hierarchical (layer-by-layer) manner.Multilingu

Unified native AI runtime for Java 17+, Java 22+ (Panama FFM), Android & C++. Run GGUF LLMs, Vision, Video, Whisper ASR, TTS & Embeddings in

Native runtime, integrations, and reproducible packaging for BTL-3 and BTL-3 Compact

Model Dispatch and Memory Allocator: an LLM inference engine for the hardware you own. Custom CPU, ROCm, Vulkan, CUDA, Metal and Level Zero

Catch local LLMs spilling into the CPU. One run shows GPU placement, prefill, decode, memory, power and a verdict.

PantheonSim: GPU simulator for CUDA and ROCm. Run CUDA and HIP programs unmodified on a CPU, no GPU needed: simulated NVIDIA and AMD GPUs fo

Grammar-aware adaptive speculative decoding for structured LLM output — custom CUDA/Triton kernels, 2.7× throughput on enum-heavy schemas

Private, offline AI assistant for Android and Windows: run local LLMs (Gemma 4, Qwen, any GGUF) on your phone or PC with llama.cpp, CUDA and

Multi-GPU prefill acceleration for llama.cpp

ParseCrate: self-hosted document processing API powered by Docling and docling-serve. Converts PDF, DOCX, PPTX, HTML, and more to Markdown/J

The old advice to avoid Windows for local LLMs used to be right. It isn't anymore. The fixes for display-GPU desktop crashes and WSL memory

Speech-first inference runtime in Rust — TTS / ASR / speech-to-speech / VC / speaker ID / VAD. An ONNX Runtime alternative that loads GGUF &

Tiny engine, immense models — run large MoE LLMs (gpt-oss, Mixtral, Qwen3-MoE) on ordinary machines by streaming experts from disk. OpenAI-c

Local, offline text-to-speech and voice cloning on audio.cpp — multi-voice narration, voice design, sound/music generation, and a real-time

High-performance CUDA inference engine for Qwen models, written from scratch in C++/CUDA

CCO is an open-source system for managing, validating, and improving GPU.