Havuz

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence

Open-source benchmark for browser AI agents on daily tasks.

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verifi

Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation,

Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim benchmark.

Swap one RAG pipeline stage, hold the rest fixed, and measure whether the gain survives composition. The framework behind "Retrieval Augment

Track 42,331+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates.

ParseBench - A Document Parsing Benchmark for AI Agents

Self-evolving agentic reward framework for image-editing evaluation — 47.4% on EditReward-Bench from only 100 preference demos, no reward-mo

Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Clau

Download official Windows ISOs directly from Microsoft's CDN. Web app + CLI tool. No account, no ads, no browser required.

An open-source Python SDK for analyzing, evaluating, and curating agent traces stored in BigQuery. Built on top of the BigQuery Agent Analyt

Открытый бенчмарк LLM: какая нейросеть лучше пишет код 1С:Предприятие (BSL). Объективная оценка LLM по методике SMOP с реальным исполнением

A composable framework for reproducible evaluation of long-running agents.

SciPIvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 Evaluations and Datasets Track)

Evaluate (lint, review, and security-audit) your AI coding agent harness.

AI Agent / LLM application engineer Chinese interview question bank: 2,038 questions with reference answers

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model

HarnessEval-W: Agentifying the Evaluation of Visual Worlds

DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with

Litefuse - Agent Observability and Evaluation Platform

Run cyber-AI agents against real missions in isolated environments. Record everything as OpenTelemetry evidence. Grade the evidence, not the

SWE-Serve: an agentic benchmark of 53 production inference-engineering tasks derived from merged SGLang pull requests, run with Harbor.

LoopArena, uzun süreli yazılım görevlerinde bir coding agent’ı yönlendiren modelleri ortak koşullarda karşılaştırmak için benchmark ve çalıştırma altyapısı sunuyor.

Neden yükseliyor · Coding agents’ın uzun süreli iş akışlarında nasıl yönlendirileceği güncel bir araştırma konusu; LoopArena bu konuya açık protokol, benchmark sonuçları ve bir arXiv paper ile dikkat çekiyor. Ancak 146 yıldız, 28 günlük repo yaşı ve son 28 günde tek farklı yazar, ilginin henüz yaygın benimsemeye dönüştüğünü göstermiyor.

EvoTrace, Claude Code ve Codex oturumlarını inceleyip eğitim, değerlendirme ve doğrulama için yeniden kullanılabilir varlıklara dönüştürür.

Neden yükseliyor · 39 günde 292 yıldız, coding agent oturumlarından eğitim ve değerlendirme varlığı üretme fikrine ilgi olduğunu gösteriyor; ancak bu ilginin nedeni ölçümlerden anlaşılamıyor. Son 28 günde commit ve farklı yazar olmaması, ilginin devam eden geliştirme veya benimsemeyle desteklendiğine dair kanıt sunmuyor.

deedchain, browser agent raporlarındaki iddiaları kaydedilmiş ground truth ile karşılaştırarak rapor doğruluğunu ölçen bir benchmark ve Python kütüphanesidir.

Neden yükseliyor · Repo yalnızca 23 günlük ve 28 yıldızlı; son 28 gün commit ve farklı yazar verisi ölçülmediği için yükselişin arkasındaki aktivite veya benimsenme düzeyi doğrulanamıyor. Rapor doğruluğunu ayrı bir değerlendirme konusu yapması dikkat çekici olsa da README'deki “first benchmark” iddiası bağımsız olarak doğrulanmış değil.

Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs

skilled: open-source agent skill evaluation CLI and MCP server for Codex and Claude Code. Test skills, detect regressions, and improve with

Benchmark for comparing agent harnesses on everyday online tasks — fixes the base model, varies the harness. Sister project of ClawBench, sa

PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions

An external memory for LLMs: a bi-temporal knowledge graph with hybrid (keyword + vector) retrieval and a self-improving reflection loop. Be

The experience graph for AI agents. Self-hosted memory, human-reviewed decisions, and outcome-backed datasets for model evaluation and learn

Precision-aware retrieval benchmark for LLM memory systems.

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Cairn runs your AI and ML pipelines behind budgets, policy, approvals and an audit trail.

Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered c

Unified World Model Inference & Evaluation Infrastructure

🦞 ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery

GAN-style self-improvement loop for any text artifact: mutate, grade with a SEPARATE model, keep only verified wins (pairwise-judged), rever

Alignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface

The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it,

⚡ Unit tests for AI. Test prompts, compare models, save money.

The trust layer for your AI agent on any platform. Trace and evaluate your agent. Full Observability. Run Model portability analysis. Autotu

A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The ga

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.