Havuz

Unified World Model Inference & Evaluation Infrastructure

🦞 ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery

GAN-style self-improvement loop for any text artifact: mutate, grade with a SEPARATE model, keep only verified wins (pairwise-judged), rever

Alignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface

The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it,

⚡ Unit tests for AI. Test prompts, compare models, save money.

The trust layer for your AI agent on any platform. Trace and evaluate your agent. Full Observability. Run Model portability analysis. Autotu

A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The ga

A survey of rubrics across the evolving LLM landscape.

Open-source Environment toolkit of claw-like agents, support task/harness generation and evaluation

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

A design-of-experiments platform for evaluating compound AI systems - find which technique drives quality, by how much, and whether the di

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

An auditable and reproducible evaluation suite for LeWM-compatible latent world models, with official-compatible and difficulty-controlled p

Open benchmark datasets republished in stressed form to evaluate ML/LLM robustness. Curated by Santander AI Lab.

A minimal coding-agent harness built to measure what context management actually costs: constraint survival after compaction (8%→83%), def

Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

Benchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.

Official code for the paper: "Multi-User Large Language Model Agents"

研迹 ResearchTrail｜本地 AI Native投资研究工作台。串联证券、组合、证据报告、论点复审与 Today 简报，让研究有据、判断留痕。Python 业务内核 · Electron 桌面 · 可复现 Agent 评测。

Compare web access APIs (web unblockers / scrape APIs) head-to-head against a fixed suite of ~90 real-world, hard-to-scrape URLs

Multi-modal benchmark for measuring sensitive-information redaction in human recording data

WRBench: camera-controlled generation and diagnostic evaluation of video world models.

Give Vercel AI SDK agents a real persistent filesystem with Amazon S3 Files.

面向 AI 奥林匹克与认证（IOAI / NOAI / LMCC）的在线评测与竞赛平台：支持客观题、代码题、LLM 工程题与产物提交（类 Kaggle）评测。

OpenRouter TypeScript harness for reproducible LLM benchmarks and evaluations.

Benchmark and fix LLM hallucinations when extracting figures from SEC 10-K filings

Behavioral regression testing for AI apps and agents: pytest plugin, behavior diff, CI gate, and document extraction metrics

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? (EMNLP 2026 Findings)

OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with

AcademiClaw: When Students Set Challenges for AI Agents — a bilingual benchmark of 80 university student-sourced academic tasks.

Local-first observability and evaluation for production forecasts - capture, validate, and compare forecasts with a local UI.

A runnable multi-agent process with a built-in evaluation system — repeated rollouts scored on the git diff, not the agent's self-report. Op

Red Team AI Benchmark: Evaluating LLMs for authorized offensive-security tasks

Eval-driven multimodal agent for identifying trees, logs, bark, leaves and wood from photographs. Dendrology is the reference domain; the e

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Measures how much code LLM agents waste. 5 tasks, sandboxed, weekly snapshots, contestable references.

How much does an image model change what you told it to leave alone? Chained-edit benchmark for image models + installable scorer and Claude

Workstream is governed contribution infrastructure for coordinating, verifying, and recording work performed by humans, AI agents, or both.

Score any AI for trust — Eval, Monitor, Govern. 10 trust dimensions, one free API key (5 credits / $500).

A temporal visual reasoning benchmark over egocentric robot manipulation video — 39 scenarios, 7 dimensions, 34,713 questions

Benchmark for repository-level code understanding: 500 open-ended questions from real GitHub Discussions across 50 repositories, graded by a

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio

Model routing is a System 1 job: a 421M local decision engine (Laya) against a GPT-5 nano reasoning router. Same accuracy, 35x faster, free.

Robot data infrastructure and runtime: turn video, state, actions, and telemetry into training episodes, RL environments, policy rollouts, a

从原作台词蒸馏角色的说话方式，落成可验证的 prompt 约束