Havuz

Semantic cache strategy bench: replay recorded request traces through candidate caching policies and measure which one actually pays off.

Run cyber-AI agents against real missions in isolated environments. Record everything as OpenTelemetry evidence. Grade the evidence, not the

A design-of-experiments platform for evaluating compound AI systems - find which technique drives quality, by how much, and whether the di

🧪💥 Evaluation framework based on Vitest, the testing framework you familiar with, for agents, models, and more.

Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs

The experience graph for AI agents. Self-hosted memory, human-reviewed decisions, and outcome-backed datasets for model evaluation and learn

Benchmark for comparing agent harnesses on everyday online tasks — fixes the base model, varies the harness. Sister project of ClawBench, sa

Catches multilingual voice agents placing the wrong order. τ-bench style benchmark + CI gate for voice ordering agents in English and Hingli

Open benchmark datasets republished in stressed form to evaluate ML/LLM robustness. Curated by Santander AI Lab.

Model routing is a System 1 job: a 421M local decision engine (Laya) against a GPT-5 nano reasoning router. Same accuracy, 35x faster, free.

Benchmarking Agents on Real-World Terminal Tasks

PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

An open-source Python SDK for analyzing, evaluating, and curating agent traces stored in BigQuery. Built on top of the BigQuery Agent Analyt

Open-source Environment toolkit of claw-like agents, support task/harness generation and evaluation

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

Raw benchmark logs for Swift-Qwen3.8-27B vs the Qwen3.8-27B BF16 base: every response, score, and config across 9 benchmarks

A temporal visual reasoning benchmark over egocentric robot manipulation video — 39 scenarios, 7 dimensions, 34,713 questions

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

Compare web access APIs (web unblockers / scrape APIs) head-to-head against a fixed suite of ~90 real-world, hard-to-scrape URLs

从原作台词蒸馏角色的说话方式，落成可验证的 prompt 约束

OpenRouter TypeScript harness for reproducible LLM benchmarks and evaluations.

A lightweight, general-purpose harness for tool-using LLM agents, fair benchmark evaluation, harness baselines, and personal assistant workf

Evaluate (lint, review, and security-audit) your AI coding agent harness.

AI Agent / LLM application engineer Chinese interview question bank: 2,038 questions with reference answers

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

WRBench: camera-controlled generation and diagnostic evaluation of video world models.

面向 AI 奥林匹克与认证（IOAI / NOAI / LMCC）的在线评测与竞赛平台：支持客观题、代码题、LLM 工程题与产物提交（类 Kaggle）评测。

Eval-driven multimodal agent for identifying trees, logs, bark, leaves and wood from photographs. Dendrology is the reference domain; the e

Multi-modal benchmark for measuring sensitive-information redaction in human recording data

Gen AI Evaluation Toolkit on AWS is a flexible, cloud-native accelerator built on AWS serverless architecture that enables comprehensive eva

Download official Windows ISOs directly from Microsoft's CDN. Web app + CLI tool. No account, no ads, no browser required.

Benchmark for repository-level code understanding: 500 open-ended questions from real GitHub Discussions across 50 repositories, graded by a

An external memory for LLMs: a bi-temporal knowledge graph with hybrid (keyword + vector) retrieval and a self-improving reflection loop. Be

Official code for the paper: "Multi-User Large Language Model Agents"

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Red Team AI Benchmark: Evaluating LLMs for authorized offensive-security tasks

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? (EMNLP 2026 Findings)

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model

Local-first observability and evaluation for production forecasts - capture, validate, and compare forecasts with a local UI.

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

EvalPort — The Open LLM Evaluation Standard. Portable test cases, graders, suites, and results across DeepEval, Promptfoo, Inspect AI, and m

Open-source control plane for governed ModelOps — fine-tune, evaluate, promote, and serve models as cost-capped, policy-gated, audited comma

Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions

Precision-aware retrieval benchmark for LLM memory systems.

A production-oriented LLM engineering platform with OpenAI-compatible serving, streaming, observability, evaluation, reproducible experiment

RAG parameter sweep and evaluation toolkit — a practical way to test how different vector databases, embedding models, chunking strategies,

AcademiClaw: When Students Set Challenges for AI Agents — a bilingual benchmark of 80 university student-sourced academic tasks.