Havuz

PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

Open-source Environment toolkit of claw-like agents, support task/harness generation and evaluation

Model routing is a System 1 job: a 421M local decision engine (Laya) against a GPT-5 nano reasoning router. Same accuracy, 35x faster, free.

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

AI Agent / LLM application engineer Chinese interview question bank: 2,038 questions with reference answers

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

OpenRouter TypeScript harness for reproducible LLM benchmarks and evaluations.

Raw benchmark logs for Swift-Qwen3.8-27B vs the Qwen3.8-27B BF16 base: every response, score, and config across 9 benchmarks

从原作台词蒸馏角色的说话方式，落成可验证的 prompt 约束

A lightweight, general-purpose harness for tool-using LLM agents, fair benchmark evaluation, harness baselines, and personal assistant workf

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

面向 AI 奥林匹克与认证（IOAI / NOAI / LMCC）的在线评测与竞赛平台：支持客观题、代码题、LLM 工程题与产物提交（类 Kaggle）评测。

Eval-driven multimodal agent for identifying trees, logs, bark, leaves and wood from photographs. Dendrology is the reference domain; the e

An external memory for LLMs: a bi-temporal knowledge graph with hybrid (keyword + vector) retrieval and a self-improving reflection loop. Be

Gen AI Evaluation Toolkit on AWS is a flexible, cloud-native accelerator built on AWS serverless architecture that enables comprehensive eva

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Benchmark for repository-level code understanding: 500 open-ended questions from real GitHub Discussions across 50 repositories, graded by a

Red Team AI Benchmark: Evaluating LLMs for authorized offensive-security tasks

Official code for the paper: "Multi-User Large Language Model Agents"

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions

Cairn runs your AI and ML pipelines behind budgets, policy, approvals and an audit trail.

EvalPort — The Open LLM Evaluation Standard. Portable test cases, graders, suites, and results across DeepEval, Promptfoo, Inspect AI, and m

Precision-aware retrieval benchmark for LLM memory systems.

RAG parameter sweep and evaluation toolkit — a practical way to test how different vector databases, embedding models, chunking strategies,

A production-oriented LLM engineering platform with OpenAI-compatible serving, streaming, observability, evaluation, reproducible experiment

AcademiClaw: When Students Set Challenges for AI Agents — a bilingual benchmark of 80 university student-sourced academic tasks.

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

A runnable multi-agent process with a built-in evaluation system — repeated rollouts scored on the git diff, not the agent's self-report. Op

Mutation testing for your LLM evals — degrade the system under test and check whether your evals would actually catch a regression.

Workstream is governed contribution infrastructure for coordinating, verifying, and recording work performed by humans, AI agents, or both.

Score any AI for trust — Eval, Monitor, Govern. 10 trust dimensions, one free API key (5 credits / $500).

Robot data infrastructure and runtime: turn video, state, actions, and telemetry into training episodes, RL environments, policy rollouts, a

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio

The open testing standard for voice AI agents. Deterministic + semantic + RAG augmented evaluation. Local first. Zero telemetry.

Research prototype for trace-based observability and failure analysis in retrieval-augmented generation.