Havuz

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, depl

Reproducible on-device LLM benchmarks for Apple Silicon (iPhone 17 Pro, M4 Max): Apple Core AI, MLX, llama.cpp, LiteRT-LM and Core ML on the

Fast Gemma 4 inference in pure Java

Apple Silicon mlx with Zero Dependency Java

llmBench is a high-depth benchmarking tool designed to measure the raw performance of local LLM runtimes (Ollama, llama.cpp) while providing

Plug-and-play, dependency-free web command center for DGX Spark fleets (and any GPU boxes): live per-node GPU/CPU/mem/temps/power, optional

BuildArena: Physics-grounded Besiege Building for LLMs

Local AI engine in Rust: run GGUF and ONNX models for chat, vision, speech and embeddings on your own GPU, RAM and CPU, with automatic place

The agent that evolves with you 𖤍

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

A static performance model for LLM inference: given (model, GPU, interconnect, workload), instantly answer VRAM feasibility, latency(TTFT, T

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

Open-source, local-first AI team workspace. Several AI models work together in one conversation, each with its own provider, model and tool

What I run on my inference server: Qwen3.8-Flash-Next on two RTX 3090s, and how I got here

DeepSeek-V4.1-Flash-1017 on one 256 GB M5 Ultra: 1M context, 70.7-104.9 tok/s decode with DSpark drafts, 1,739-1,867 tok/s cold prompts (8k-

[NeurIPS 2026 Spotlight] Fast, lossless LLM inference via dual-view diffusion decoding.

Method for Long Context RLMs using verifiable Lambda Calculus

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed

GLM-5.3-Flash EXL3 (320B MoE) on 2x NVIDIA DGX Spark — production serving kit, 1M context, 97%+ multi-session prefix caching, DFlash2 spec d

Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4

Bounded, inspectable LLM inference pipelines from declared YAML — runs offline against Ollama or any OpenAI-compatible local server, emittin

Can my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducib

Reproduce Lium's LocalMaxxing leaderboard runs (7 boards, 20 Verified) on a rented Lium GPU in one script — vLLM/llama.cpp recipes, measured

AI inference library for mobile devices

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

llambda.lisp inferencer for LLMs in GGUF format

Runtime security for AI agents. In-process, zero dependencies, Apache 2.0.

Human as Agent（人即智能体）· Human-as-LLM 人工代理网关：把工程师变成模型，OpenAI 兼容 /v1 接入 Agent 调度池，涉密/需人工任务路由给真实工程师。为 AI Agent 提效、为平台/运维收集工单。Human-as-LLM Gate

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Distributed, peer-to-peer, decentralized network for LLM inference — peers seed & leech completions, metered by BitTorrent-protocol-style up

Foundational ternary-model inference & training library — CUDA + CPU, BitNet/TQ ternary kernels

Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, re

🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit

Proxima lets existing GPUs serve 4x more concurrent requests

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Neura lets you run long, tool heavy coding sessions without blowing up token costs by compressing old context into references and only resto

Diagnose vLLM inference servers

Retrieval-augmented generation: training-free document-assessment routing for cost-efficient multi-agent RAG.

AI Infra 推理系统学习资料：从深度学习算子到 vLLM 服务端部署

Benchmark llama.cpp on your own GPU/CPU boxes and compare against everyone else's — sweep runner, an sha256-keyed index of ~4M GGUF files, a

Run stock llama.cpp with the stock Unsloth Qwen3.6-35B-A3B 4-bit on a 16 GB Mac.

The Continuous Verification and Optimization layer for self-hosted LLMs

single-executable / library which combines llama.cpp, whisper.cpp, and stable-diffusion.cpp

Run Qwen3.8-Flash-Next on ONE RTX 3090 (24 GB) + 64 GB RAM: 128K context, up to 2,100 tok/s prefill, 43–51 tok/s decode. vLLM runtime with h