Havuz

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

A temporal visual reasoning benchmark over egocentric robot manipulation video — 39 scenarios, 7 dimensions, 34,713 questions

Compare web access APIs (web unblockers / scrape APIs) head-to-head against a fixed suite of ~90 real-world, hard-to-scrape URLs

AI Agent / LLM application engineer Chinese interview question bank: 2,038 questions with reference answers

Evaluate (lint, review, and security-audit) your AI coding agent harness.

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

WRBench: camera-controlled generation and diagnostic evaluation of video world models.

Download official Windows ISOs directly from Microsoft's CDN. Web app + CLI tool. No account, no ads, no browser required.

Multi-modal benchmark for measuring sensitive-information redaction in human recording data

An external memory for LLMs: a bi-temporal knowledge graph with hybrid (keyword + vector) retrieval and a self-improving reflection loop. Be

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Benchmark for repository-level code understanding: 500 open-ended questions from real GitHub Discussions across 50 repositories, graded by a

Red Team AI Benchmark: Evaluating LLMs for authorized offensive-security tasks

Official code for the paper: "Multi-User Large Language Model Agents"

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? (EMNLP 2026 Findings)

Local-first observability and evaluation for production forecasts - capture, validate, and compare forecasts with a local UI.

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions

Open-source control plane for governed ModelOps — fine-tune, evaluate, promote, and serve models as cost-capped, policy-gated, audited comma

EvalPort — The Open LLM Evaluation Standard. Portable test cases, graders, suites, and results across DeepEval, Promptfoo, Inspect AI, and m

Precision-aware retrieval benchmark for LLM memory systems.

AcademiClaw: When Students Set Challenges for AI Agents — a bilingual benchmark of 80 university student-sourced academic tasks.

OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Score any AI for trust — Eval, Monitor, Govern. 10 trust dimensions, one free API key (5 credits / $500).

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio

Project page for AV-Phys Bench: Do Joint Audio-Video Generation Models Understand Physics?

The open testing standard for voice AI agents. Deterministic + semantic + RAG augmented evaluation. Local first. Zero telemetry.

Research prototype for trace-based observability and failure analysis in retrieval-augmented generation.