Havuz

Run large AI models across your computers and servers. Powered by Colibri. Pure C.

kimi-k3-in-c, Kimi K3 modelinde inference çalıştırmayı hedefleyen bağımlılıksız bir C99 motorudur.

Neden yükseliyor · 57 günde 8.683 yıldız alması ve Kimi K3'ü 8 GB RAM'li CPU'da çalıştırma iddiası dikkat çekiyor; ancak son 28 gündeki 12 commit'in tek yazardan gelmesi, ilginin bağımsız teknik doğrulama veya geniş katkıcı tabanıyla desteklendiğini göstermiyor.

Proxima lets existing GPUs serve 4x more concurrent requests

Distributed 56M-parameter LLM inference across 3 ESP32-S3 boards via ESP-NOW , Split-PLE + KV cache, fully offline.

Get more intelligence from every bit. Better quantization formats and smarter calibration let larger, stronger models run smoothly on the ha

Plug-and-play, dependency-free web command center for DGX Spark fleets (and any GPU boxes): live per-node GPU/CPU/mem/temps/power, optional

Can my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

TurboFieldfare, Gemma 4 26B-A4B modelini Apple Silicon Mac’lerde Swift ve Metal tabanlı özel bir runtime ile yerel olarak çalıştırır.

Neden yükseliyor · 72 günde 6828 yıldız alması ve 8 GB RAM’li Mac’te büyük bir model çalıştırma iddiası dikkat çekiyor. README belirli cihazlarda ölçülmüş decode hızları ve deney kayıtları sunuyor; ancak yıldızlar benimseme veya kalite kanıtı değil. Son 28 günde 4 commit ve 1 farklı yazar olması, geliştirme etkinliği ve ekip genişliği açısından sınırlı kanıt sağlıyor.

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, depl

Open-source local AI workspace — advancing on-device inference.

The Continuous Verification and Optimization layer for self-hosted LLMs

Distributed, peer-to-peer, decentralized network for LLM inference — peers seed & leech completions, metered by BitTorrent-protocol-style up

Open-source control panel for the real llama.cpp: see where a model lands in your VRAM before loading, every llama-server flag per model, on

llambda.lisp inferencer for LLMs in GGUF format

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embe

BuildArena: Physics-grounded Besiege Building for LLMs

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Enchant Version of 9Router. 304+ Providers (API-key, OAuth, free-tier, and 39 web-cookie providers), 6 combo strategies (fallback, round-rob

AI Infra 推理系统学习资料：从深度学习算子到 vLLM 服务端部署

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

Context optimization for AI coding agents — lower token cost and latency without sacrificing accuracy.

Retrieval-augmented generation: training-free document-assessment routing for cost-efficient multi-agent RAG.

Agentic Runtime

TensorFold, Apple Silicon ve NVIDIA GPU’larda çeşitli LLM’leri yerel olarak çalıştırıp OpenAI-compatible ve Anthropic API’leri üzerinden sunan bir inference engine’dir.

Neden yükseliyor · 109 günlük yaşına göre 1.055 yıldız ve son 28 günde 143 commit ile 20 farklı yazar kayda değer geliştirme hareketliliği gösteriyor; model ve checkpoint desteğinin genişliği de dikkat çekiyor. Bu veriler ilgiyi açıklayabilir, ancak yıldızların ne zaman kazanıldığı veya üretim kullanımının yaygınlığı bu ölçümlerden anlaşılamaz.

Local AI for your whole house: chat, images and voice on your own GPU. Computes each model's flags, fits them to your VRAM, and hot-swaps th

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cl

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Runtime security for AI agents. In-process, zero dependencies, Apache 2.0.

Honest local LLM deployment planning and benchmarking for high unified-memory Macs and future Linux/NVIDIA rigs.

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Cal

Bounded, inspectable LLM inference pipelines from declared YAML — runs offline against Ollama or any OpenAI-compatible local server, emittin

Apple Silicon mlx with Zero Dependency Java

single-executable / library which combines llama.cpp, whisper.cpp, and stable-diffusion.cpp

[NeurIPS 2026 Spotlight] Fast, lossless LLM inference via dual-view diffusion decoding.

Diagnose vLLM inference servers

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Reproducible on-device LLM benchmarks for Apple Silicon (iPhone 17 Pro, M4 Max): Apple Core AI, MLX, llama.cpp, LiteRT-LM and Core ML on the

Neura lets you run long, tool heavy coding sessions without blowing up token costs by compressing old context into references and only resto

AI inference library for mobile devices

Local AI engine in Rust: run GGUF and ONNX models for chat, vision, speech and embeddings on your own GPU, RAM and CPU, with automatic place

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

User friendly GUI for configuring and launching llama.cpp

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code. Python SDK, LLMLingua-2, MIT.

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A

Turn your Android phone into an OpenAI-compatible LLM inference server - Fully local, private and Open Source