Havuz

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine f

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

Local inference for diffusion language models (LLaDA, Dream), written in Mojo

🧱 Experimental library for building inference engines

ReAlign models instantly, on demand. Per-request alignment for verified users and channels, or Always On. Use directly or with Hermes, OpenC

Don't memorize ML. Understand it from first principles including Inference, Cuda, Deep Learning, Data Science and Reinforcement Learning

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how

llama.cpp fork that abliterates at inference

Reef, agent isteklerini sunma, feedback toplama, öğrenme döngüsü yürütme ve kabul edilen güncellemeleri sürümleyerek dağıtma işlevlerini bir araya getiren Python altyapısıdır.

Neden yükseliyor · Sürekli öğrenen agent'lar gündemde ve Reef, inference'tan feedback'e, eğitimden sürüm dağıtımına uzanan altyapıyı tek projede toplamayı hedefliyor. Verilen ölçümlerde 28 günde 6.122 yıldız, 362 commit ve 19 farklı yazar güçlü ilgi ve geliştirme faaliyeti gösteriyor; ancak bu veriler tek başına üretim kullanımını veya teknik başarıyı kanıtlamıyor.

Agent Mesh

Feel how fast a local LLM runs before you buy the hardware

Kyojin: open-source engine that runs the largest open models on one mini PC or one graphics card, about 2× faster than llama.cpp. AMD (ROCm)

ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuous

Run Laya, the open-source Jev-compatible System-1 decision model, from Node.js / TypeScript via ONNX Runtime

A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interf

Bilingual English and Mandarin text to speech in C++ and GGUF. Voice design, cloning, direction and experimental voice conversion. Vulkan ba

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimiz

Serve your models fast on your own infra in 3 steps.

AI inference gateway and control plane for multi-provider routing, policies, caching, tools, and observability.

Self-hosted vLLM made simple. Git-friendly YAML for models and profiles, full lifecycle from an intuitive CLI and TUI.

On-device AI SDKs for iOS, macOS, Android, and the web. Small, focused models that run fully offline in Swift, Kotlin, and JavaScript with C

NeMo-Speech.cpp is a lightweight C++ inference runtime for Speech models

⚡ Ultra-Reliable AI Inference Gateway & Verifiable Agent Control Plane. Dual-Dispatch, Zero-Token JSON Auto-Healer, VaaS On-Chain Attestatio

pooled, birden fazla cihazın tarayıcı GPU’larını WebRTC üzerinden birleştirerek büyük open model’leri birlikte çalıştırmasını sağlayan bir LLM inference projesidir.

Neden yükseliyor · 27 günlük yaşına göre 461 yıldız ve son 28 günde 90 commit güçlü bir erken ilgi ve geliştirme temposuna işaret ediyor. Tarayıcıdan katılımla cihazları birleştirme fikri, yerel inference ve donanım paylaşımı ilgisini çekebilir; ancak yalnızca 2 farklı yazar olduğundan aktivite geniş bir katkıcı topluluğunun kanıtı değil. İlginin nedenini kesin olarak belirlemek mümkün değil.

Benchmark local LLM serving by replaying real agent trajectories

An agentic system that auto-optimizes LLM workloads on AMD GPUs.

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

An open-source inference engine for AI functions in SQL

Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-com

Open-source control panel for the real llama.cpp: see where a model lands in your VRAM before loading, every llama-server flag per model, on

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

One-command deployment of open-source world models on AWS. Each model is a self-contained cartridge served behind a uniform API on EC2 or Sa

WHIRL (Windows HIP Inference for RDNA LLMs): native Windows C++/HIP LLM inference engine for the AMD Ryzen AI Max+ 395 (Radeon 8060S) and Ra

Runs Hugging Face ONNX models on any Nx backend, with no per-architecture code

Tu propio "ChatGPT" con todos los modelos de NaN (qwen, deepseek, gemma, mimo, whisper, kokoro), self-hosted en Docker. Pones tu API key y l

BOAR - Adaptive Local Intelligence

Compute-for-memory research lab for running larger LLMs on memory-constrained consumer GPUs. Official website exvram.pro

Frontier: A Discrete-Event Simulator for Modern LLM Serving

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache co

From PyTorch model to end-to-end TensorRT inference experience in two commands—AI-native, cross-platform, and built for the best possible us

Local inference you can prove. A single-binary GGUF engine in C: serves, scores, and trains LoRA through the quantized weights it deploys, b

Realtime dashboard for the Halogen LLM inference server

Your AI intranet: network the computers you already own for inference and training.

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

AI Gateway

RigMark benchmarks local AI the way coding agents actually use it.

Plug-and-play llama.cpp runtime for Intel Arc GPUs. Auto-detects your card, picks safe SYCL defaults, and exposes an OpenAI-compatible API.

Strata 0.2.3 (engine 0.1.48) for AMD gfx906 (MI50/MI60/Radeon VII): rocBLAS solution table, Qwen3.6/Ornith engine restored, DFlash2 build fi

ScribeCrate: open-source, self-hosted Whisper speech-to-text server powered by faster-whisper. OpenAI-compatible transcription and English t

Open-source self-hosted home AI inference platform for AMD Strix Halo — multi-backend slots, OpenAI-compatible gateway, Vue 3 + FastAPI + sy