Havuz

llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s

Camelid: a Rust-native local inference backend with evidence-gated model compatibility.

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cp

Python package for LLM compression

A tiny neural-network framework in pure Go with AVX2 SIMD kernels (GOEXPERIMENT=simd)

Halo is an open-source framework built by White Circle for training large language and multimodal models

Bilingual (中文+EN) ML / LLM / diffusion / agent interview cheat sheets for AI 秋招 — generated by ARIS /interview-cheatsheet, rendered by /rend

turbovec, TurboQuant ile vektörleri sıkıştırıp Rust ve Python üzerinden arama yapmayı sağlayan yerel bir vector index kütüphanesidir.

Neden yükseliyor · 185 günde 17.245 yıldız almış olması ve bellek tasarrufu ile hızlı yerel arama vaadi dikkat çekiyor. Ancak son 28 günde commit ve farklı yazar sayısı sıfır; bu veriler yükselişin güncel olarak sürdüğünü göstermiyor. README'deki performans karşılaştırmaları proje tarafından bildiriliyor, bağımsız doğrulama bilgisi verilen ölçümlerde yok.

kimi-k3-in-c, Kimi K3 modelinde inference çalıştırmayı hedefleyen bağımlılıksız bir C99 motorudur.

Neden yükseliyor · 57 günde 8.683 yıldız alması ve Kimi K3'ü 8 GB RAM'li CPU'da çalıştırma iddiası dikkat çekiyor; ancak son 28 gündeki 12 commit'in tek yazardan gelmesi, ilginin bağımsız teknik doğrulama veya geniş katkıcı tabanıyla desteklendiğini göstermiyor.

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Cal

Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/a

LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs

Phonon: open speech recognition models (Phonon-2, Phonon-1) — CLI, CPU and CUDA images

JANG — GGUF for MLX. YOU MUST USE JANG_Q RUNTIME. Adaptive Mixed-Precision Quantization + Runtime for Apple Silicon

Kyojin: open-source engine that runs the largest open models on one mini PC or one graphics card, about 2× faster than llama.cpp. AMD (ROCm)

A ternary, zero-heap tiny language model that runs inside a $2 microcontroller — bit-exact Python <-> C99 <-> Cortex-M3 (QEMU) parity. Apach

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A

Gumbel-Softmax post-training quantization for LLMs (1–3 bit scalar, INT/GGUF-compatible).

ScribeCrate: open-source, self-hosted Whisper speech-to-text server powered by faster-whisper. OpenAI-compatible transcription and English t

QuantClaw is a plug-and-play task-type routing quantization plugin for OpenClaw.

TurboQuant KV cache compression for MLX with fused Metal kernels. 4.6x compression at 98% FP16 speed.

🦀 Decoder-only LLM built from scratch in pure Rust using Candle — no Python, no PyTorch. Gated DeltaNet + sparse attention, fine-grained MoE

600 KB WASM runtime for Cactus Compute's Needle AI tool-calling models, Needle 3, 2 and 1 from one build. Browser, Cloudflare Workers, Node.

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Get more intelligence from every bit. Better quantization formats and smarter calibration let larger, stronger models run smoothly on the ha

Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGU

Extreme weight + KV cache compression for LLMs on Apple Silicon (MLX implementation of Google's TurboQuant)

Open source skill library for AI coding agents to write, optimize, and debug high performance compute kernels across CUDA, Triton, and quant

First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression wit

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

Interactive canvas for taking an LLM apart tensor by tensor: measured INT8/INT4/FP8 error, distributions, spectra and outlier channels for e

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, depl

559 K パラメータの日本語 TTS を ESP32-S3 で実時間合成。漢字かな交じり文の形態素解析・アクセント推定まで端末内で走る（M5Stack CoreS3 実機で確認）。推論は依存ゼロの C99、ブラウザ demo あり。arXiv:2608.21378 sanoTT

Open-source platform for creating, distributing and running sovereign EU-compliant LLMs. Verticalize any model for your domain, language and

Frontier-class LLM inference on a laptop CPU — gpt-oss:20b at ~110 tok/s on Apple M4 Max, 7.5x llama.cpp, no GPU. From-scratch NEON/SME kern

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

🏆Winning Project | ModelGate is a contract-aware AI control plane that ingests customer contracts, extracts SLA/privacy/routing constraints,

Run a 28.9M-parameter TinyLM on ESP32-S3 with an RP2040 OLED display node for fully local embedded AI inference.

edge-dit.cpp — a native C/C++ inference engine for Diffusion Transformers (DiT), designed for local and resource-constrained devices with au

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

TurboQuant MLX: Flagship high-performance model quantization for Apple Silicon. Squeeze maximum inference speed and memory efficiency out of

Rust inference engine for a 28.9M-parameter PLE TinyLM on the ESP32-S3 — 9.4 tok/s, faster than the C implementation it was ported from

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

First open-source TurboQuant KV cache compression for LLM inference. Drop-in for HuggingFace. pip install turboquant.

Run AI models on Apple Silicon and Raspberry Pi

Can my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.