Havuz

Benchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.

Stateful ride-support agent: versioned claims, evidence-backed RAG, durable memory and guarded MCP tools.

Official code for the paper: "Multi-User Large Language Model Agents"

Evaluate (lint, review, and security-audit) your AI coding agent harness.

Download official Windows ISOs directly from Microsoft's CDN. Web app + CLI tool. No account, no ads, no browser required.

研迹 ResearchTrail｜本地 AI Native投资研究工作台。串联证券、组合、证据报告、论点复审与 Today 简报，让研究有据、判断留痕。Python 业务内核 · Electron 桌面 · 可复现 Agent 评测。

PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

Swap one RAG pipeline stage, hold the rest fixed, and measure whether the gain survives composition. The framework behind "Retrieval Augment

Run cyber-AI agents against real missions in isolated environments. Record everything as OpenTelemetry evidence. Grade the evidence, not the

Compare web access APIs (web unblockers / scrape APIs) head-to-head against a fixed suite of ~90 real-world, hard-to-scrape URLs

Multi-modal benchmark for measuring sensitive-information redaction in human recording data

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

WRBench: camera-controlled generation and diagnostic evaluation of video world models.

Give Vercel AI SDK agents a real persistent filesystem with Amazon S3 Files.

AI Agent / LLM application engineer Chinese interview question bank: 2,038 questions with reference answers

面向 AI 奥林匹克与认证（IOAI / NOAI / LMCC）的在线评测与竞赛平台：支持客观题、代码题、LLM 工程题与产物提交（类 Kaggle）评测。

OpenRouter TypeScript harness for reproducible LLM benchmarks and evaluations.

Benchmark and fix LLM hallucinations when extracting figures from SEC 10-K filings

Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions

An external memory for LLMs: a bi-temporal knowledge graph with hybrid (keyword + vector) retrieval and a self-improving reflection loop. Be

SWE-Serve: an agentic benchmark of 53 production inference-engineering tasks derived from merged SGLang pull requests, run with Harbor.

Behavioral regression testing for AI apps and agents: pytest plugin, behavior diff, CI gate, and document extraction metrics

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? (EMNLP 2026 Findings)

OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models

The experience graph for AI agents. Self-hosted memory, human-reviewed decisions, and outcome-backed datasets for model evaluation and learn

Gen AI Evaluation Toolkit on AWS is a flexible, cloud-native accelerator built on AWS serverless architecture that enables comprehensive eva

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with

Semantic cache strategy bench: replay recorded request traces through candidate caching policies and measure which one actually pays off.

AcademiClaw: When Students Set Challenges for AI Agents — a bilingual benchmark of 80 university student-sourced academic tasks.

RAG parameter sweep and evaluation toolkit — a practical way to test how different vector databases, embedding models, chunking strategies,

Local-first observability and evaluation for production forecasts - capture, validate, and compare forecasts with a local UI.

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model

Precision-aware retrieval benchmark for LLM memory systems.

A runnable multi-agent process with a built-in evaluation system — repeated rollouts scored on the git diff, not the agent's self-report. Op

Red Team AI Benchmark: Evaluating LLMs for authorized offensive-security tasks

Eval-driven multimodal agent for identifying trees, logs, bark, leaves and wood from photographs. Dendrology is the reference domain; the e

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Measures how much code LLM agents waste. 5 tasks, sandboxed, weekly snapshots, contestable references.

How much does an image model change what you told it to leave alone? Chained-edit benchmark for image models + installable scorer and Claude

SciPIvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Cairn runs your AI and ML pipelines behind budgets, policy, approvals and an audit trail.

Workstream is governed contribution infrastructure for coordinating, verifying, and recording work performed by humans, AI agents, or both.

Score any AI for trust — Eval, Monitor, Govern. 10 trust dimensions, one free API key (5 credits / $500).

A temporal visual reasoning benchmark over egocentric robot manipulation video — 39 scenarios, 7 dimensions, 34,713 questions

Benchmark for repository-level code understanding: 500 open-ended questions from real GitHub Discussions across 50 repositories, graded by a

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio