Havuz

Project Maya - run GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE) and MiniMax-M3.1-Flash (428B MoE) privately on your own GPU

Halo is an open-source framework built by White Circle for training large language and multimodal models

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache co

Slotstream, büyük bir MoE modelini SSD’den parça parça yükleyip yoğun kullanılan expert’leri bellekte önbelleğe alarak Apple Silicon Mac’lerde çalıştıran yerel bir inference engine’dir.

Neden yükseliyor · README, 48 GB M5 Pro üzerinde belirli koşullarda 15.86 tok/s ölçümü ve prompt işleme iyileştirmeleri yayımlıyor; son 28 günde 271 commit ve 5 farklı yazar da yoğun geliştirme faaliyetine işaret ediyor. Ancak proje yalnızca 30 günlük ve mevcut sürüm için nitelikli tam yanıt hızı ölçümünün hâlâ beklediği belirtiliyor.

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimiz

Automated alignment adjustment for LLMs — direct steering, LoRA, and MoE expert-granular abliteration, optimized via multi-objective Optuna

llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched multi-GPU rigs by exact VRAM math.

Bilingual (中文+EN) ML / LLM / diffusion / agent interview cheat sheets for AI 秋招 — generated by ARIS /interview-cheatsheet, rendered by /rend

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cp

Open-source local AI workspace — advancing on-device inference.

从 MiniMind 源码读起,再延伸到现代大模型技术体系的中文学习笔记。主线逐行精读预训练 / SFT / DPO / PPO / GRPO 与训练机制;附录 17 篇进阶卷覆盖量化、投机解码、RLHF 全景、模型代际史等 MiniMind 没涉及、但进阶绕不开的主题。

kimi-k3-in-c, Kimi K3 modelinde inference çalıştırmayı hedefleyen bağımlılıksız bir C99 motorudur.

Neden yükseliyor · 57 günde 8.683 yıldız alması ve Kimi K3'ü 8 GB RAM'li CPU'da çalıştırma iddiası dikkat çekiyor; ancak son 28 gündeki 12 commit'in tek yazardan gelmesi, ilginin bağımsız teknik doğrulama veya geniş katkıcı tabanıyla desteklendiğini göstermiyor.

Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark,

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs

Get more intelligence from every bit. Better quantization formats and smarter calibration let larger, stronger models run smoothly on the ha

Curated list of open-weight AI models with commercially exploitable licenses, verified benchmarks, and no EU restrictions.

Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGU

Fastest MoE/LLM inference runtime for consumer and edge Blackwell GPUs. SN74 on Gittensor.

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, depl

The local AI app built for the Mac. Run LLMs like Qwen, Gemma and GPT-OSS on Apple Silicon, including models far bigger than your Mac's memo

GleamLM大语言模型全栈训练平台——核心代码用 PyTorch 从零实现，最低仅需 12GB 消费级显卡即可完整训练，另有一套工业代码与手写实现消费同一份数据逐层对照。模型训练支持CLI和可视化训练。

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

CPU-optimized LLM inference engine (C)

BOAR - Adaptive Local Intelligence

Local LLM inference engine written from scratch in Rust — hand-written AVX-512 assembly kernels, Metal & Vulkan compute shaders. Supports Qw

MLX-compatible REAP for pruning MoE models on Apple Silicon

Xing4.0-29B-Intent - schema-guided intent recognition and structured information extraction (intent and slot filling, entity, relation and e

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from

Qwen3.8-Flash-Next 177B MoE (IQ3_S 3.44bpw) on RTX 5090 Laptop 24GB: full-model quality @ 97.7 tok/s · dual-lane 256K full-context unlocked

The Kimi K3 and DeepSeek-V4-Flash-0731 architectures, reimplemented from scratch in pure Rust (zero dependencies) and verified 1:1 against t

Run a 177B model (Qwen3.8-Flash-Next) on an 8 GB laptop GPU. Measured: 6.6 GiB VRAM, 47.8 GiB RAM, 34-35 tok/s. The n-gram table stays on di

Run GPT-OSS-120B on a 16 GB Mac. Streams mixture-of-experts weights from SSD so only the 2 GiB trunk stays resident. Swift + Metal, no depen

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

Implementation of LatentMoE,Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (Elango et al., NVIDIA 2026) — in Pytorch.

Tiny-MoE is a lightweight Mixture-of-Experts language model built entirely from scratch in native PyTorch and trained end-to-end on Kaggle u

PyTorch-native framework for fault-tolerant distributed training of foundation models on AI clusters

A compact, readable PyTorch reference implementation of the DeepSeek-V4 architecture (Flash and Pro), in the spirit of nanoGPT. ~4,300 LOC.

Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Dock

👻 MoEWatch: A lightweight diagnostic and audit library for Mixture-of-Experts (MoE) models in HuggingFace Transformers — the pytest for MoE

I Ran Qwen 3.8 27B on Just 4GB VRAM! (FreeToken Tutorial) - Edge-native MoE serving engine for local LLMs on low VRAM GPUs

Open benchmark for expert collapse and routing efficiency in sparse MoE LLMs

面向内存受限端侧设备的 MoE 大模型推理运行时优化系统

Run stock llama.cpp with the stock Unsloth Qwen3.6-35B-A3B 4-bit on a 16 GB Mac.

Run Qwen3.8-Flash-Next on ONE RTX 3090 (24 GB) + 64 GB RAM: 128K context, up to 2,100 tok/s prefill, 43–51 tok/s decode. vLLM runtime with h

Tiny engine, immense models — run large MoE LLMs (gpt-oss, Mixtral, Qwen3-MoE) on ordinary machines by streaming experts from disk. OpenAI-c

MoE expert compression toolkit — prune, merge, and forge specialist models for local inference.

Evidence-backed structural validation of Kimi K3 UD-IQ1_M and UD-Q4_K_XL split GGUF releases using OMIV.