Havuz

A counterfactual battery for instruction following in vision-language-action policies: change one word of the instruction, hold the scene, a

Real-event-anchored benchmark for detecting AI-generated videos in real-world crisis settings.

Same model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open model

GateMem: a benchmark and evaluation toolkit for memory governance in multi-principal shared-memory LLM agents.

Alignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface

A benchmark for evaluating LLM × harness performance.

The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it,

A Benchmark for Evaluating Agent Self-Evolution on Real Business Tasks

Ultra-Fast Python Solutions: 110x Speed Optimization Guide 2026

Anti-slop skills for AI agents: slopbeth cleans the shipped writing, slopgent cleans the conversation.

AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?

Local LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity re

Benchmark AI Agents on Enterprise Workflows

SRA-Bench and SR-Agents: a benchmark and toolkit for skill-retrieval-augmented LLM agents.

State-Aware Collaboration Protocol: auditable work receipts for AI agents.

🔍 The hardest search benchmark in the wild — vague, multi-turn, proactive. 200 long-horizon tasks with persona-driven progressive disclosure

Self-evolving agentic reward framework for image-editing evaluation — 47.4% on EditReward-Bench from only 100 preference demos, no reward-mo

NeurIPS2026 , Scalable annotation pipeline for action-aglined fine-grained instruciton for Visual-language-Action model

JevGym is an open-source platform designed to benchmark and facilitate better probabilistic estimation in Jev-alike models

Agents can work. They cannot declare success. A governed coordination protocol for multi-agent AI work: eight contracts (task, resource, own

Honest local LLM deployment planning and benchmarking for high unified-memory Macs and future Linux/NVIDIA rigs.

Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGU

NEXBENCH measures what an autonomous agent can actually do on-chain — execute transactions, route swaps, bridge funds, manage DeFi positions

[COLM'26] SkillLearnBench is the first benchmark for evaluating continual learning methods that automatically generate agent skills.

[Public preview] The first public quant trading benchmark scored against what professional traders actually made on the same asset

An agent skill that plays ARC-AGI-3. One rule: say what an action will do before you spend it. Claude Code on Opus 5 finished all 25 public

A benchmark for testing whether coding agents can resolve engineering tasks in scientific software

[NeurIPS 2026] MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare.

Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs

Benchmark self-evolving Agent upon realistic large-scale file workspaces

First LLM Studio: local-first LLM workbench for Apple Silicon + MLX. 本地优先 LLM 工作台，覆盖 Agent、对比、微调、评测与运行时遥测。

A composable framework for reproducible evaluation of long-running agents.

Compare two local Persian (Farsi) speech-to-text models side by side: a Persian Whisper fine-tune vs. general large-v3, on Apple Silicon.

Open benchmark on observability tasks built on Harbor

One-shot LLM eval cases by NAGI STUDIO - same prompt, different agents (model + harness), runnable artifacts side by side.

Χ-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

Benchmarking repository-borne prompt injection attacks and lightweight defenses for local coding agents. DL4C @ ICML 2026.

Official code for "CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges"

Reproducible on-device LLM benchmarks for Apple Silicon (iPhone 17 Pro, M4 Max): Apple Core AI, MLX, llama.cpp, LiteRT-LM and Core ML on the

LLM benchmark and leaderboard for narrator-bias sycophancy, opposite-narrator contradictions, and judgment consistency.

Measurement harness for proxy providers, browser engines and scraping targets. Every claim carries the run it came from.

Audio-Oscar is a multi-agent framework for generating long-form, controllable audio from complex audio scene descriptions.

The benchmark tasks and evaluation harness for "PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments".

Benchmark for comparing agent harnesses on everyday online tasks — fixes the base model, varies the harness. Sister project of ClawBench, sa

Continuous benchmarks for multi-device CoW filesystems (btrfs, ZFS, bcachefs): snapshots, aging, compression, redundancy layouts

Evaluating frontier models on real-world ai-infra engineering tasks

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

🚀 Automated scraper & injector for 100+ free AI coding models into 9router combos. Live pre-tested, ranked by coding benchmarks & zero-conf

📊 Automated hourly benchmarks for 20+ NVIDIA NIM models — interactive dashboard, zero infra, self-hostable. Open-source & community-driven.

Can LLM/MARL agents coordinate in long-horizon, open-ended tasks? NeurIPS 2026 (Evaluations and Datasets).