Havuz

📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python

A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decodi

Context management proxy for AI coding agents: verbatim archive, on-demand recall, and saved engine state on supported backends.

Glyd, model ağırlıklarını ve KV cache'i bit düzeyinde kayıpsız biçimde sıkıştırıp GPU'da kullanmayı hedefleyen bir compression yazılımıdır.

Neden yükseliyor · README, yaygın bir maliyet sorununa doğrudan yanıt verdiğini ve 32B ile 72B modellerde daha az GPU ile çalışmaya dair ölçümler sunuyor; ancak 9 günlük repoda 170 yıldız ve 474 commit'in yalnızca 2 farklı yazardan gelmesi, ilginin ve geliştirme temposunun geniş bir katkıcı tabanına dayandığını göstermiyor. Yükselişin kesin kaynağı veya bağımsız benimsenme verisi bilinmiyor.

Refusal removal as a loadable KV-cache graft — zero weight modification, no refusal-direction projection, fully reversible.

Qwen3.8-27B on RTX 5090 Laptop 24GB — TWIN-TURBO NVFP4 (native llama.cpp) vs Coder390 Q3 LynnStyle (.ninfer + NInfer engine in WSL2/Docker):

Pyramis-L3D: hierarchical sparse attention research prototype (code only; weights on Hugging Face)

DeepSeek Harness plugin: DietrichGebert/ponytail lazy senior mode & 7-rung ladder port, 6 skills with bilingual descriptions & per-skill tog

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducib

Proxima lets existing GPUs serve 4x more concurrent requests

Build and train a mini Kimi K3 from scratch — a pure-PyTorch playground from a custom 88M model to 2T-scale LLM profiles.

A minimal coding-agent harness built to measure what context management actually costs: constraint survival after compaction (8%→83%), def

qMLX: Custom inference engine for Qwen 3.5 122B on Apple Silicon, extending MLX with hybrid attention support, SSD-backed KV cache, and RYS

CPU-first LLM inference server on llama.cpp. Runs useful models on free-tier ARM boxes; rewriting the input made it ~6x faster and more accu

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Agentic Runtime

Security-First Agentic IDE Coding Harness

Semantic retrieval for AI agents: high-recall memory and code search that cuts coding-agent token usage by nearly 50%

An AI-native distributed data plane built in Rust that supports high performance RPC, KV Cache, Message Queue, and File & Object Acceleratio

Native Rust, single-binary MLX inference server for Apple Silicon. OpenAI/Anthropic-compatible LLM serving — text, vision, audio, embeddings

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable (arXiv:2606.17107) — paper, code, results, and interactive companion.

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Cal

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

Extreme weight + KV cache compression for LLMs on Apple Silicon (MLX implementation of Google's TurboQuant)

Consumer-aware compression for embedding indexes and LLM KV caches — compress by the metric the downstream consumer actually uses. PCA-Matry

LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.

TurboQuant KV cache compression for MLX with fused Metal kernels. 4.6x compression at 98% FP16 speed.

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression wit

TurboQuant MLX: Flagship high-performance model quantization for Apple Silicon. Squeeze maximum inference speed and memory efficiency out of

First open-source TurboQuant KV cache compression for LLM inference. Drop-in for HuggingFace. pip install turboquant.

Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and

This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed

Soul: The Persistent Memory Layer for Multi-Agent Systems. High-performance session orchestration with SQLite-based KV-Cache for MCP. Give y