Havuz

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

Pure-Rust MoE inference engine for Qwen3.5/3.6 GGUF. CPU+GPU hybrid experts, NVRTC CUDA kernels, OpenAI-compatible server. 2x llama.cpp pref

🧱 Experimental library for building inference engines

Reef, agent isteklerini sunma, feedback toplama, öğrenme döngüsü yürütme ve kabul edilen güncellemeleri sürümleyerek dağıtma işlevlerini bir araya getiren Python altyapısıdır.

Neden yükseliyor · Sürekli öğrenen agent'lar gündemde ve Reef, inference'tan feedback'e, eğitimden sürüm dağıtımına uzanan altyapıyı tek projede toplamayı hedefliyor. Verilen ölçümlerde 28 günde 6.122 yıldız, 362 commit ve 19 farklı yazar güçlü ilgi ve geliştirme faaliyeti gösteriyor; ancak bu veriler tek başına üretim kullanımını veya teknik başarıyı kanıtlamıyor.

Don't memorize ML. Understand it from first principles including Inference, Cuda, Deep Learning, Data Science and Reinforcement Learning

Bilingual English and Mandarin text to speech in C++ and GGUF. Voice design, cloning, direction and experimental voice conversion. Vulkan ba

Local inference for diffusion language models (LLaDA, Dream), written in Mojo

Qwen3.8-Flash-Next (125B MoE) up to 1M context on one or two RTX cards (tuned on 3080 10gb). A measured, tuned and benchmarked fork of Str

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine f

NeMo-Speech.cpp is a lightweight C++ inference runtime for Speech models

Realtime dashboard for the Halogen LLM inference server

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimiz

On-device AI SDKs for iOS, macOS, Android, and the web. Small, focused models that run fully offline in Swift, Kotlin, and JavaScript with C

Run any local LLM engine, auto-tuned to your GPU — polished web UI + OpenAI/Anthropic-compatible API. Point Claude Code at your own machine

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embe

A MoE inference engine you can read in an afternoon.

Agent Mesh

From PyTorch model to end-to-end TensorRT inference experience in two commands—AI-native, cross-platform, and built for the best possible us

ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuous

Project UAI is a universal AI agent that runs directly inside Roblox. It brings a Claude Code–style agent loop to the Roblox client, allowin

Strata 0.2.3 (engine 0.1.48) for AMD gfx906 (MI50/MI60/Radeon VII): rocBLAS solution table, Qwen3.6/Ornith engine restored, DFlash2 build fi

Two-rank tensor-parallel LLM serving over USB4v2 between AMD Strix Halo hosts: a custom Linux thunderbolt stream transport, self-healing lin

pooled, birden fazla cihazın tarayıcı GPU’larını WebRTC üzerinden birleştirerek büyük open model’leri birlikte çalıştırmasını sağlayan bir LLM inference projesidir.

Neden yükseliyor · 27 günlük yaşına göre 461 yıldız ve son 28 günde 90 commit güçlü bir erken ilgi ve geliştirme temposuna işaret ediyor. Tarayıcıdan katılımla cihazları birleştirme fikri, yerel inference ve donanım paylaşımı ilgisini çekebilir; ancak yalnızca 2 farklı yazar olduğundan aktivite geniş bir katkıcı topluluğunun kanıtı değil. İlginin nedenini kesin olarak belirlemek mümkün değil.

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache co

An agentic system that auto-optimizes LLM workloads on AMD GPUs.

The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLL

one-year LLM serving trace from Chutes

Recomputation-free, context-independent KV caching for large language models

Hands-on GPU/HPC infrastructure operations: K8s GPU scheduling, HAMi sharing, Slurm, observability & vLLM inference. Learn it free on a lapt

Tenstorrent backend plugin for vLLM.

OpenMayhem is an open-source and decentralized inference router for consumer & pro hardware. Providers earn, users/agents access cheap infer

Run Laya, the open-source Jev-compatible System-1 decision model, from Node.js / TypeScript via ONNX Runtime

A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interf

Inference-native Tokenmaxxing Agent Harness for Loop Engineering

Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark,

Unified World Model Inference & Evaluation Infrastructure

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how

Kyojin: open-source engine that runs the largest open models on one mini PC or one graphics card, about 2× faster than llama.cpp. AMD (ROCm)

Camelid: a Rust-native local inference backend with evidence-gated model compatibility.

Ultrafast Qwen3-TTS: sub-50 ms time-to-first-audio at 10 requests per second.

An open-source inference engine for AI functions in SQL

Feel how fast a local LLM runs before you buy the hardware

Provider-pluggable orchestration runtime for multi-model AI inference. ( Sakana Fugu style )

Tree-based speculative decoding for Apple Silicon (MLX). ~10-15% faster than DFlash on code, ~1.5x over autoregressive. First MLX port with

Benchmark local LLM serving by replaying real agent trajectories

An AI-native distributed data plane built in Rust that supports high performance RPC, KV Cache, Message Queue, and File & Object Acceleratio

AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric

ReAlign models instantly, on demand. Per-request alignment for verified users and channels, or Always On. Use directly or with Hermes, OpenC

ScribeCrate: open-source, self-hosted Whisper speech-to-text server powered by faster-whisper. OpenAI-compatible transcription and English t