Havuz

Evaluate (lint, review, and security-audit) your AI coding agent harness.

PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

Compare web access APIs (web unblockers / scrape APIs) head-to-head against a fixed suite of ~90 real-world, hard-to-scrape URLs

Multi-modal benchmark for measuring sensitive-information redaction in human recording data

Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions

The experience graph for AI agents. Self-hosted memory, human-reviewed decisions, and outcome-backed datasets for model evaluation and learn

Gen AI Evaluation Toolkit on AWS is a flexible, cloud-native accelerator built on AWS serverless architecture that enables comprehensive eva

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model

Precision-aware retrieval benchmark for LLM memory systems.

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

How much does an image model change what you told it to leave alone? Chained-edit benchmark for image models + installable scorer and Claude

Cairn runs your AI and ML pipelines behind budgets, policy, approvals and an audit trail.

Workstream is governed contribution infrastructure for coordinating, verifying, and recording work performed by humans, AI agents, or both.

The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it,

⚡ Unit tests for AI. Test prompts, compare models, save money.

A survey of rubrics across the evolving LLM landscape.

A lightweight, general-purpose harness for tool-using LLM agents, fair benchmark evaluation, harness baselines, and personal assistant workf

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

An auditable and reproducible evaluation suite for LeWM-compatible latent world models, with official-compatible and difficulty-controlled p

Open benchmark datasets republished in stressed form to evaluate ML/LLM robustness. Curated by Santander AI Lab.

Official code for the paper: "Multi-User Large Language Model Agents"

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

WRBench: camera-controlled generation and diagnostic evaluation of video world models.

Give Vercel AI SDK agents a real persistent filesystem with Amazon S3 Files.

Benchmark and fix LLM hallucinations when extracting figures from SEC 10-K filings

An external memory for LLMs: a bi-temporal knowledge graph with hybrid (keyword + vector) retrieval and a self-improving reflection loop. Be

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? (EMNLP 2026 Findings)

OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models

AcademiClaw: When Students Set Challenges for AI Agents — a bilingual benchmark of 80 university student-sourced academic tasks.

RAG parameter sweep and evaluation toolkit — a practical way to test how different vector databases, embedding models, chunking strategies,

Local-first observability and evaluation for production forecasts - capture, validate, and compare forecasts with a local UI.

A runnable multi-agent process with a built-in evaluation system — repeated rollouts scored on the git diff, not the agent's self-report. Op

Red Team AI Benchmark: Evaluating LLMs for authorized offensive-security tasks

Eval-driven multimodal agent for identifying trees, logs, bark, leaves and wood from photographs. Dendrology is the reference domain; the e

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Measures how much code LLM agents waste. 5 tasks, sandboxed, weekly snapshots, contestable references.

Score any AI for trust — Eval, Monitor, Govern. 10 trust dimensions, one free API key (5 credits / $500).

A temporal visual reasoning benchmark over egocentric robot manipulation video — 39 scenarios, 7 dimensions, 34,713 questions

Benchmark for repository-level code understanding: 500 open-ended questions from real GitHub Discussions across 50 repositories, graded by a

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio

Model routing is a System 1 job: a 421M local decision engine (Laya) against a GPT-5 nano reasoning router. Same accuracy, 35x faster, free.

Robot data infrastructure and runtime: turn video, state, actions, and telemetry into training episodes, RL environments, policy rollouts, a

从原作台词蒸馏角色的说话方式，落成可验证的 prompt 约束

Raw benchmark logs for Swift-Qwen3.8-27B vs the Qwen3.8-27B BF16 base: every response, score, and config across 9 benchmarks

The open testing standard for voice AI agents. Deterministic + semantic + RAG augmented evaluation. Local first. Zero telemetry.

Project page for AV-Phys Bench: Do Joint Audio-Video Generation Models Understand Physics?