Havuz

First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression wit

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, depl

559 K パラメータの日本語 TTS を ESP32-S3 で実時間合成。漢字かな交じり文の形態素解析・アクセント推定まで端末内で走る（M5Stack CoreS3 実機で確認）。推論は依存ゼロの C99、ブラウザ demo あり。arXiv:2608.21378 sanoTT

Open-source platform for creating, distributing and running sovereign EU-compliant LLMs. Verticalize any model for your domain, language and

Frontier-class LLM inference on a laptop CPU — gpt-oss:20b at ~110 tok/s on Apple M4 Max, 7.5x llama.cpp, no GPU. From-scratch NEON/SME kern

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

edge-dit.cpp — a native C/C++ inference engine for Diffusion Transformers (DiT), designed for local and resource-constrained devices with au

Run AI models on Apple Silicon and Raspberry Pi

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

Fast WebGPU runtime for 1-bit (binary-weight) LLMs in the browser. Bit-exact, zero runtime dependencies.

Custom converter: open-weight Hugging Face LLMs/VLMs -> on-device LiteRT-LM (.litertlm), with one-command reproduction

Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, re

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Run GPT-OSS-120B on a 16 GB Mac. Streams mixture-of-experts weights from SSD so only the 2 GiB trunk stays resident. Swift + Metal, no depen

Memory backbone for AI agents. Four-tier store — working, episodic, semantic, procedural — with decay, consolidation, and association graphs

Fit-to-Size Intelligent Tensor Quantization for GGUF - deterministic, size-exact mixed quantization on llama.cpp presets. Turn any byte budg

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

Which local LLMs can your computer run? Hardware-aware CLI, TUI and GUI: yes/slow/no verdicts and tok/s from your GPU VRAM and RAM, then ver

Native Rust, single-binary MLX inference server for Apple Silicon. OpenAI/Anthropic-compatible LLM serving — text, vision, audio, embeddings

A ternary, zero-heap tiny language model that runs inside a $2 microcontroller — bit-exact Python <-> C99 <-> Cortex-M3 (QEMU) parity. Apach

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A

QuantClaw is a plug-and-play task-type routing quantization plugin for OpenClaw.

TurboQuant KV cache compression for MLX with fused Metal kernels. 4.6x compression at 98% FP16 speed.

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

Interactive canvas for taking an LLM apart tensor by tensor: measured INT8/INT4/FP8 error, distributions, spectra and outlier channels for e

🏆Winning Project | ModelGate is a contract-aware AI control plane that ingests customer contracts, extracts SLA/privacy/routing constraints,

Run a 28.9M-parameter TinyLM on ESP32-S3 with an RP2040 OLED display node for fully local embedded AI inference.

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

TurboQuant MLX: Flagship high-performance model quantization for Apple Silicon. Squeeze maximum inference speed and memory efficiency out of

Rust inference engine for a 28.9M-parameter PLE TinyLM on the ESP32-S3 — 9.4 tok/s, faster than the C implementation it was ported from

First open-source TurboQuant KV cache compression for LLM inference. Drop-in for HuggingFace. pip install turboquant.

Can my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducib

Open-source vector database and sub-microsecond key-value store in one Go engine — embed it as a library, run it standalone, or replicate it

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Consumer-aware compression for embedding indexes and LLM KV caches — compress by the metric the downstream consumer actually uses. PCA-Matry

vLLM recipe: GLM-5.3-Flash EXL3 K2 on one DGX Spark GB10. Native MTP k=2. Measured tok/s.

Local LLMs that run lean, load instantly and keep getting smarter. A from-scratch runtime + model format in Zig: zero-copy mmap, CPU/CUDA/Vu

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Dock

Measures which GGUF quant of a model to run on your GPU: real speed and quality loss (KL divergence), not estimates.

ComfyUI deployment kit for Qwen-Image-2.1 Uncensored (GGUF): ready-to-use t2i & image-edit workflows, ComfyUI-GGUF architecture patches, and

Docker compose configs for serving Qwen3.8-27B on a DGX Spark (GB10): vLLM+MTP current, SGLang+DSPARK + FP8 rollback stacks

Tiny engine, immense models — run large MoE LLMs (gpt-oss, Mixtral, Qwen3-MoE) on ordinary machines by streaming experts from disk. OpenAI-c

A 4B decision model built for Korean.