Havuz

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cp

halogen-flash-server, Qwen3.8-Flash-Next modelini AMD Strix Halo (gfx1151) donanımında çalıştırmak için özelleştirilmiş bir inference server'dır.

Neden yükseliyor · 32 günlük repo için 722 yıldız ve son 28 günde 99 commit dikkat çekiyor. README, belirli donanım üzerindeki ölçümleri ve ölçüm koşullarını açıklıyor; ancak performans iddiaları proje tarafından sunulmuş ve bağımsız doğrulama bilgisi burada yok. 99 commit'in tek yazardan gelmesi geliştirme hızını gösterirken bakım kapasitesine dair kanıtı sınırlıyor.

Running a 30B MoE LLM on CPU at 30-50+ tok/s with a 256K Context Window via Hybrid Mamba-2 SSM, L3 NTA Slicing, AVX-512 VNNI Tile Repacking

This project is for learning: it shows how an agent loop works underneath, in a way more intuitive and visual than reading about it. Behind

llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s

pooled, birden fazla cihazın tarayıcı GPU’larını WebRTC üzerinden birleştirerek büyük open model’leri birlikte çalıştırmasını sağlayan bir LLM inference projesidir.

Neden yükseliyor · 27 günlük yaşına göre 461 yıldız ve son 28 günde 90 commit güçlü bir erken ilgi ve geliştirme temposuna işaret ediyor. Tarayıcıdan katılımla cihazları birleştirme fikri, yerel inference ve donanım paylaşımı ilgisini çekebilir; ancak yalnızca 2 farklı yazar olduğundan aktivite geniş bir katkıcı topluluğunun kanıtı değil. İlginin nedenini kesin olarak belirlemek mümkün değil.

OpenAI-compatible inference server for Qwen3.8 on one RTX 5090 / RTX PRO 6000: the 27B at ~170 tok/s (peaks ~370), Flash-Next (125B MoE) at

The fastest way to run Qwen3.8 27B on Strix Halo (gfx1151)

dgx-spark-qwen38 is an open-source installer that turns an NVIDIA DGX Spark, or any GB10 machine (ASUS, Dell, HP, Lenovo, MSI, GIGABYTE, Ace

Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant k

An environment where open models work as agents: a first-level harness with routing, skills, memory and sandboxed tools, on an engine that d

AI coding agent harness for any hardware, local or cloud (OpenAI-compatible engines). Tuned for AMD Strix Halo: Qwen3.8 125B on the iGPU, NP

WHIRL (Windows HIP Inference for RDNA LLMs): native Windows C++/HIP LLM inference engine for the AMD Ryzen AI Max+ 395 (Radeon 8060S) and Ra

Ollama-compatible LLM server on a tuned llama.cpp fork. Beats vllm by 40% on single stream decode, and by 20% on multi stream decode, and be

Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/a

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

Qwen3.8-Flash-Next 177B MoE (IQ3_S 3.44bpw) on RTX 5090 Laptop 24GB: full-model quality @ 97.7 tok/s · dual-lane 256K full-context unlocked

A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decodi

Durable local inference for Oh My Pi: NInfer + Qwen3.8 27B on one RTX 3090/4090/5090, with restart-resumable OpenAI Responses state.

Winning Qwen and DeepSeek setups for AMD Strix Halo: pinned Ansible recipes and matched benchmarks on a 128 GiB Ryzen AI Max+ 395 / Radeon 8

Qwen3.8-27B on RTX 5090 Laptop 24GB — TWIN-TURBO NVFP4 (native llama.cpp) vs Coder390 Q3 LynnStyle (.ninfer + NInfer engine in WSL2/Docker):

llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched multi-GPU rigs by exact VRAM math.

Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGU

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, depl

Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, re

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → paralleli

[NeurIPS 2026 Spotlight] Fast, lossless LLM inference via dual-view diffusion decoding.

Tree-based speculative decoding for Apple Silicon (MLX). ~10-15% faster than DFlash on code, ~1.5x over autoregressive. First MLX port with

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A

[ICML 2026] When Drafts Evolve: Speculative Decoding Meets Online Learning

CPU-first LLM inference server on llama.cpp. Runs useful models on free-tier ARM boxes; rewriting the input made it ~6x faster and more accu

DFlash vLLM for DGX Spark — Plug & Play Block-Diffusion Speculative Decoding

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

Local AI workstation: chat by voice or text, transcribe, generate images and video (with sound on Apple Silicon), benchmark and serve open-w

Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Dock

Grammar-aware adaptive speculative decoding for structured LLM output — custom CUDA/Triton kernels, 2.7× throughput on enum-heavy schemas

Native Rust, single-binary MLX inference server for Apple Silicon. OpenAI/Anthropic-compatible LLM serving — text, vision, audio, embeddings

Hikari's curated English measured notes on local LLM systems (serving, specdec, training, steering).

One command spins up Qwen3.8-27B (abliterated, NVFP4) on a rented Blackwell GPU at ~150 tok/s and wires it into your local coding agent via

Docker compose configs for serving Qwen3.8-27B on a DGX Spark (GB10): vLLM+MTP current, SGLang+DSPARK + FP8 rollback stacks

DeepSeek-V4.1-Flash-1017 on one 256 GB M5 Ultra: 1M context, 70.7-104.9 tok/s decode with DSpark drafts, 1,739-1,867 tok/s cold prompts (8k-

Serve DeepSeek and Qwen, run an on-demand model library, and fine-tune with Unsloth QLoRA on 2x NVIDIA DGX Spark. TP=2 vLLM lanes behind one

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Specul

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous

Uncensored DeepSeek-V4.1-Flash IQ2_XS on Google Colab