Havuz

CPU-optimized LLM inference engine (C)

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

Hemmingway-1: a 27B model that writes the way a person writes. Open weights, free for non-commercial use (CC BY-NC 4.0); commercial use by a

Dual-engine (llama.cpp + vLLM) LLM benchmarking pipeline for GGUF & safetensors on NVIDIA GPUs — speed, quality, live dashboard, publishable

Agentic Runtime

一套功能完整、轻量稳定的 ComfyUI 实用节点集。初衷是为了平替一些断更插件的功能，会持续更新一些新开发的节点插件。有好的创意可以发Issues，我会力所能及想办法做出来。全中文界面，原生控件，不依赖第三方库。

Universal local LLM runtime. Drop-in Ollama replacement with HuggingFace integration, pipelines, structured output. Free & open source.

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducib

Crow — a local AI agent, not a chat box: 30 tools, MCP, memory, skills, goals, subagents, vision, and image generation and editing with Qwen

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embe

Open, calibrated System 1 decision models: typed choice / score / yes-no answers with a probability for every option. 400M to 80B; CPU, NVID

Ollama-compatible LLM server on a tuned llama.cpp fork. Beats vllm by 40% on single stream decode, and by 20% on multi stream decode, and be

WHIRL (Windows HIP Inference for RDNA LLMs): native Windows C++/HIP LLM inference engine for the AMD Ryzen AI Max+ 395 (Radeon 8060S) and Ra

One GPU. Full LLM workflow. Real benchmarks. No cloud required.

AI inference library for mobile devices

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

Native AI assistant for macOS, built with SwiftUI and llama.cpp. Run local LLMs offline, work with PDFs, and create custom AI assistants. Op

Impulse by Velocity: run AI models on your own computer, on Windows and Linux. Load any GGUF model and chat with web search, files, charts a

Local AI for your whole house: chat, images and voice on your own GPU. Computes each model's flags, fits them to your VRAM, and hot-swaps th

A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from flas

A TUI around llama.cpp for running, managing, and benchmarking local GGUF models and launching the pi coding agent against your local server

A web-based memory usage and performance calculator for Huggingface GGUF models

Local inference you can prove. A single-binary GGUF engine in C: serves, scores, and trains LoRA through the quantized weights it deploys, b

neural networks in pure C

Local AI engine in Rust: run GGUF and ONNX models for chat, vision, speech and embeddings on your own GPU, RAM and CPU, with automatic place

Memory-aware GGUF inference runtime with OpenAI-compatible serving

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Local AI workstation: chat by voice or text, transcribe, generate images and video (with sound on Apple Silicon), benchmark and serve open-w

Qwen3.8-Flash-Next 177B MoE (IQ3_S 3.44bpw) on RTX 5090 Laptop 24GB: full-model quality @ 97.7 tok/s · dual-lane 256K full-context unlocked

Winning Qwen and DeepSeek setups for AMD Strix Halo: pinned Ansible recipes and matched benchmarks on a 128 GiB Ryzen AI Max+ 395 / Radeon 8

A supercharged terminal for Mac with built-in local AI — suggestions, error explanations, and /ask chat. No API key required.

A beautifully crafted desktop client for running and managing local LLMs via llama.cpp.

Locally fine-tuned Russian RAG models: document splitter, query expansion, retrieval embedder. Teacher distillation → GGUF → llama.cpp on AM

Local LLMs that run lean, load instantly and keep getting smarter. A from-scratch runtime + model format in Zig: zero-copy mmap, CPU/CUDA/Vu

Local classification, routing, and scoring for AI agents, with option probabilities. Text + images, Python, MCP and agent skills.

LLM inference engine for AMD RDNA4 — Rust + Vulkan compute shaders, gguf & native FP8.

Fit-to-Size Intelligent Tensor Quantization for GGUF - deterministic, size-exact mixed quantization on llama.cpp presets. Turn any byte budg

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic 8 GB RAM path. Native C, no GP

Arabic-first expressive zero-shot speech synthesis — Audar-TTS-V1 (Flash + Turbo + Pro). Voice cloning, expression tags, benchmarks, technic

Rivo Agent is a private, local-first mobile AI assistant that runs compact GGUF language models directly on your phone. It lets users downlo

译声实时翻译（白色新版）— 免费的 Windows 实时日语字幕与翻译工具，支持系统声音、本地 GGUF / Ollama / OpenAI 兼容 API 翻译。

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Qwen3.8-27B on RTX 5090 Laptop 24GB — TWIN-TURBO NVFP4 (native llama.cpp) vs Coder390 Q3 LynnStyle (.ninfer + NInfer engine in WSL2/Docker):

Fine-tune Qwen2.5-Coder-7B for free (QLoRA on Kaggle T4) and run a local coding agent that runs your code instead of guessing — 8 GB Mac, ll

Unified native AI runtime for Java 17+, Java 22+ (Panama FFM), Android & C++. Run GGUF LLMs, Vision, Video, Whisper ASR, TTS & Embeddings in

Native runtime, integrations, and reproducible packaging for BTL-3 and BTL-3 Compact

Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Dock

DeepSeek V4 Flash 284B on AMD Strix Halo (gfx1151) — up to 32 tok/s decode & ~250 tok/s prefill via ROCmFPX, DSpark & ROCm 7.2