Havuz

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

Network-faithful simulation of LLM serving and training deployments

Legacy: self-hosted LLM/RAG stack installer for DGX Spark (arm64). The work continued as AGmind Systems Lab — measured local-LLM performance

Plug-and-play, dependency-free web command center for DGX Spark fleets (and any GPU boxes): live per-node GPU/CPU/mem/temps/power, optional

Black-box LLM distillation for any verifiable task, with text or image inputs: teacher API -> rejection sampling -> SFT -> eval -> GRPO

Dual-engine (llama.cpp + vLLM) LLM benchmarking pipeline for GGUF & safetensors on NVIDIA GPUs — speed, quality, live dashboard, publishable

Hemmingway-1: a 27B model that writes the way a person writes. Open weights, free for non-commercial use (CC BY-NC 4.0); commercial use by a

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind

Can my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.

Lightweight proxy for LLM

Open, calibrated, single-pass typed-decision models (4B & 27B) for finance and trading

Self-hosted, distributed, multi-model LLM inference

A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decodi

one-year LLM serving trace from Chutes

Open, calibrated System 1 decision models: typed choice / score / yes-no answers with a probability for every option. 400M to 80B; CPU, NVID

AI coding agent harness for any hardware, local or cloud (OpenAI-compatible engines). Tuned for AMD Strix Halo: Qwen3.8 125B on the iGPU, NP

Reproduce Lium's LocalMaxxing leaderboard runs (7 boards, 20 Verified) on a rented Lium GPU in one script — vLLM/llama.cpp recipes, measured

Disaggregated quantization official codebase

A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from flas

AI/LLM on Kubernetes 基础设施知识库：涵盖 GPU、大数据、调度、分布式训练、LLM 推理、RAG、Agent 与生产运维。| AI/LLM on Kubernetes Infrastructure Knowledge Base: Covering GPUs,

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simula

croit AIplane — one plane for all your AI. Self-hosted AI infrastructure connecting applications and users with models, agents, tools and en

A web-based memory usage and performance calculator for Huggingface GGUF models

Local-first desktop assistant for the whole job hunt — scrape listings, match with a local LLM, generate applications in your own voice, and

A MoE inference engine you can read in an afternoon.

Consumer-aware compression for embedding indexes and LLM KV caches — compress by the metric the downstream consumer actually uses. PCA-Matry

📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python

RigMark benchmarks local AI the way coding agents actually use it.

Open-source local-first computer-use agent for Linux, Windows, and macOS, powered by local VLMs or compatible APIs.

Local AI workstation: chat by voice or text, transcribe, generate images and video (with sound on Apple Silicon), benchmark and serve open-w

Run the LLM Swarm Router on machines to distribute Local Ai to the Swarm - More Machines - MORE SPEED

Proxima lets existing GPUs serve 4x more concurrent requests

vLLM recipe: GLM-5.3-Flash EXL3 K2 on one DGX Spark GB10. Native MTP k=2. Measured tok/s.

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

Self-hosted GPU cluster orchestration: deploy and serve open-source AI models on your own hardware — FastAPI control plane, worker agents, v

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

Declarative Python SDK for building AI inference platforms on Kubernetes — bare machines to a served model, composed rather than hand-assemb

Hands-on GPU/HPC infrastructure operations: K8s GPU scheduling, HAMi sharing, Slurm, observability & vLLM inference. Learn it free on a lapt

QwenCleo-ASR is the currently State-Of-The-Art (SOTA) open-source model for Egyptian Arabic & code-switching speech recognition Built on Qwe

A small chat model trained from scratch: tokenizer → pretraining → SFT → RL → multimodal. 0.78B params, CORE 0.248, tool calling, vLLM. Weig

Tenstorrent backend plugin for vLLM.

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.

MCP server that delegates Claude Code agent work to local models (llama.cpp / vLLM) on your own hardware

LLM Infra 工程实战 — 面向应用层工程师的 LLM 基础设施

MLOps — vLLM + LiteLLM + Portainer CE + Netdata. Case study song ngữ: kiến trúc, metrics, bài học vận hành. · Self-hosted LLM inference don

Decisions, not essays: a choice, a yes or no, or a score with a calibrated probability. Open weights, Jev-compatible API, adaptive thinking.

nvtop for vLLM — an interactive terminal dashboard for vLLM serving performance (concurrency, throughput, cache & KV memory, latency, spec-d