Havuz

Qwen3.8-Flash-Next (125B MoE) up to 1M context on one or two RTX cards (tuned on 3080 10gb). A measured, tuned and benchmarked fork of Str

ReAlign models instantly, on demand. Per-request alignment for verified users and channels, or Always On. Use directly or with Hermes, OpenC

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

llama.cpp fork that abliterates at inference

Runs Hugging Face ONNX models on any Nx backend, with no per-architecture code

Qwen3.8-Flash-Next (180B MoE) on one 12-16 GB NVIDIA GPU, Linux or Windows. Own engine, own compression.

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how

Realtime dashboard for the Halogen LLM inference server

Pure-Rust MoE inference engine for Qwen3.5/3.6 GGUF. CPU+GPU hybrid experts, NVRTC CUDA kernels, OpenAI-compatible server. 2x llama.cpp pref

Serve your models fast on your own infra in 3 steps.

AnyFlow - flow-matching video generation: training, inference and VBench evaluation for t2v and i2v models.

Local inference for diffusion language models (LLaDA, Dream), written in Mojo

Strata 0.2.3 (engine 0.1.48) for AMD gfx906 (MI50/MI60/Radeon VII): rocBLAS solution table, Qwen3.6/Ornith engine restored, DFlash2 build fi

WHIRL (Windows HIP Inference for RDNA LLMs): native Windows C++/HIP LLM inference engine for the AMD Ryzen AI Max+ 395 (Radeon 8060S) and Ra

Kyojin: open-source engine that runs the largest open models on one mini PC or one graphics card, about 2× faster than llama.cpp. AMD (ROCm)

Guía gratuita de IA generativa avanzada en español (311 págs.): Transformer, post-entrenamiento, inferencia, GPU, difusión, agentes, segurid

AI inference gateway and control plane for multi-provider routing, policies, caching, tools, and observability.

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine f

ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuous

Benchmark local LLM serving by replaying real agent trajectories

Compute-for-memory research lab for running larger LLMs on memory-constrained consumer GPUs. Official website exvram.pro

Proof of concept: turn old Android phones into an LLM inference cluster. ADB zero-touch provisioning, llama.cpp on each phone, OpenAI-compat

BOAR - Adaptive Local Intelligence

Model Dispatch and Memory Allocator: an LLM inference engine for the hardware you own. Custom CPU, ROCm, Vulkan, CUDA, Metal and Level Zero

SDK-first, independently maintained SemIf fork for fast, self-hosted semantic decisions.

Run Laya, the open-source Jev-compatible System-1 decision model, from Node.js / TypeScript via ONNX Runtime

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

One-command deployment of open-source world models on AWS. Each model is a self-contained cartridge served behind a uniform API on EC2 or Sa

JEV-inspired parallel decisions for CUDA LLMs. One context, many decisions. vLLM API, game-agent examples, and reproducible benchmarks.

Local LLMs that run lean, load instantly and keep getting smarter. A from-scratch runtime + model format in Zig: zero-copy mmap, CPU/CUDA/Vu

Declarative Python SDK for building AI inference platforms on Kubernetes — bare machines to a served model, composed rather than hand-assemb

Tiny language models that never touch DRAM.

Did your change make your LLM server faster? Measure it: faster, slower, or too close to call. Linux and Apple Silicon.

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

Cross-model transfer of persistent recurrent inference state without prefix replay

RigMark benchmarks local AI the way coding agents actually use it.

one-year LLM serving trace from Chutes

Project UAI is a universal AI agent that runs directly inside Roblox. It brings a Claude Code–style agent loop to the Roblox client, allowin

Give Vercel AI SDK agents a real persistent filesystem with Amazon S3 Files.

The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic 8 GB RAM path. Native C, no GP

pooled, birden fazla cihazın tarayıcı GPU’larını WebRTC üzerinden birleştirerek büyük open model’leri birlikte çalıştırmasını sağlayan bir LLM inference projesidir.

Neden yükseliyor · 27 günlük yaşına göre 461 yıldız ve son 28 günde 90 commit güçlü bir erken ilgi ve geliştirme temposuna işaret ediyor. Tarayıcıdan katılımla cihazları birleştirme fikri, yerel inference ve donanım paylaşımı ilgisini çekebilir; ancak yalnızca 2 farklı yazar olduğundan aktivite geniş bir katkıcı topluluğunun kanıtı değil. İlginin nedenini kesin olarak belirlemek mümkün değil.

Reef, agent isteklerini sunma, feedback toplama, öğrenme döngüsü yürütme ve kabul edilen güncellemeleri sürümleyerek dağıtma işlevlerini bir araya getiren Python altyapısıdır.

Neden yükseliyor · Sürekli öğrenen agent'lar gündemde ve Reef, inference'tan feedback'e, eğitimden sürüm dağıtımına uzanan altyapıyı tek projede toplamayı hedefliyor. Verilen ölçümlerde 28 günde 6.122 yıldız, 362 commit ve 19 farklı yazar güçlü ilgi ve geliştirme faaliyeti gösteriyor; ancak bu veriler tek başına üretim kullanımını veya teknik başarıyı kanıtlamıyor.

Spark-optimized MLX LM for local inference and deployment on Apple Silicon

Feel how fast a local LLM runs before you buy the hardware

Bilingual English and Mandarin text to speech in C++ and GGUF. Voice design, cloning, direction and experimental voice conversion. Vulkan ba

Atlas Inference is a dependency free lightweight inference engine meant to run LLMs on YOUR compute seamlessly. Supports GB10 and Strix Halo

Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools;

One command spins up Qwen3.8-27B (abliterated, NVFP4) on a rented Blackwell GPU at ~150 tok/s and wires it into your local coding agent via

Don't memorize ML. Understand it from first principles including Inference, Cuda, Deep Learning, Data Science and Reinforcement Learning