Havuz

Slotstream, büyük bir MoE modelini SSD’den parça parça yükleyip yoğun kullanılan expert’leri bellekte önbelleğe alarak Apple Silicon Mac’lerde çalıştıran yerel bir inference engine’dir.

Neden yükseliyor · README, 48 GB M5 Pro üzerinde belirli koşullarda 15.86 tok/s ölçümü ve prompt işleme iyileştirmeleri yayımlıyor; son 28 günde 271 commit ve 5 farklı yazar da yoğun geliştirme faaliyetine işaret ediyor. Ancak proje yalnızca 30 günlük ve mevcut sürüm için nitelikli tam yanıt hızı ölçümünün hâlâ beklediği belirtiliyor.

halogen-flash-server, Qwen3.8-Flash-Next modelini AMD Strix Halo (gfx1151) donanımında çalıştırmak için özelleştirilmiş bir inference server'dır.

Neden yükseliyor · 32 günlük repo için 722 yıldız ve son 28 günde 99 commit dikkat çekiyor. README, belirli donanım üzerindeki ölçümleri ve ölçüm koşullarını açıklıyor; ancak performans iddiaları proje tarafından sunulmuş ve bağımsız doğrulama bilgisi burada yok. 99 commit'in tek yazardan gelmesi geliştirme hızını gösterirken bakım kapasitesine dair kanıtı sınırlıyor.

kimi-k3-in-c, Kimi K3 modelinde inference çalıştırmayı hedefleyen bağımlılıksız bir C99 motorudur.

Neden yükseliyor · 57 günde 8.683 yıldız alması ve Kimi K3'ü 8 GB RAM'li CPU'da çalıştırma iddiası dikkat çekiyor; ancak son 28 gündeki 12 commit'in tek yazardan gelmesi, ilginin bağımsız teknik doğrulama veya geniş katkıcı tabanıyla desteklendiğini göstermiyor.

A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interf

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cp

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous

Run many AI agents at once on your Mac, up to 3x faster.

audio.cpp with a full-task WebUI - pure C++ audio-model inference engine powered by ggml. TTS, ASR/STT, VAD, voice conversion, speaker diari

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs

A vLLM-inspired inference engine for looped transformers, with loop-level scheduling and execution across composable stages

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share

The fastest way to run Qwen3.8 27B on Strix Halo (gfx1151)

600 KB WASM runtime for Cactus Compute's Needle AI tool-calling models, Needle 3, 2 and 1 from one build. Browser, Cloudflare Workers, Node.

Fastest MoE/LLM inference runtime for consumer and edge Blackwell GPUs. SN74 on Gittensor.

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, depl

An environment where open models work as agents: a first-level harness with routing, skills, memory and sandboxed tools, on an engine that d

hummingbird is a lightweight, zero dependency runtime for massive open source Mixture of Experts (MoE) language models. It unifies SSD, RAM,

Big models. Small Macs. Zero excuses.

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

edge-dit.cpp — a native C/C++ inference engine for Diffusion Transformers (DiT), designed for local and resource-constrained devices with au

The fastest inference engine for Qwen3.8-27B on the NVIDIA RTX 5090: up to 500 tokens/s for one agent and up to 2,000 tokens/s for many, con

OpenAI-compatible inference server for Qwen3.8 on one RTX 5090 / RTX PRO 6000: the 27B at ~170 tok/s (peaks ~370), Flash-Next (125B MoE) at

Rust inference engine for a 28.9M-parameter PLE TinyLM on the ESP32-S3 — 9.4 tok/s, faster than the C implementation it was ported from

🚿 Llaminar: an LLM inferencing engine in C++.

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

Local LLM inference engine written from scratch in Rust — hand-written AVX-512 assembly kernels, Metal & Vulkan compute shaders. Supports Qw

Atlas Inference is a dependency free lightweight inference engine meant to run LLMs on YOUR compute seamlessly. Supports GB10 and Strix Halo

Ollama-compatible LLM server on a tuned llama.cpp fork. Beats vllm by 40% on single stream decode, and by 20% on multi stream decode, and be

AI inference library for mobile devices

Dynamic per-token early exit for LLM inference. Skip layers tokens don't need

neural networks in pure C

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, re

The Kimi K3 and DeepSeek-V4-Flash-0731 architectures, reimplemented from scratch in pure Rust (zero dependencies) and verified 1:1 against t

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic 8 GB RAM path. Native C, no GP

Run a coding model at 300+ tok/s on your Mac. Native Rust + Metal inference engine for LiquidAI LFM2.5-24B-A2B (MLX 4-bit) on Apple Silicon.

Bulk classification and decision inference on one GPU: the content is read once, every question about it is scored in the same pass. OpenAI-

Model Dispatch and Memory Allocator: an LLM inference engine for the hardware you own. Custom CPU, ROCm, Vulkan, CUDA, Metal and Level Zero

🔥 Run large open language models on your phone. A C++20 GGUF inference engine that streams models larger than RAM from flash, paces itself b

A dependency-free C inference engine for Liquid AI's LFM2 / LFM2.5 models. Terminal chat, an OpenAI-compatible server, a web agent and a ter

Speech-first inference runtime in Rust — TTS / ASR / speech-to-speech / VC / speaker ID / VAD. An ONNX Runtime alternative that loads GGUF &