Havuz

Benchmark for comparing agent harnesses on everyday online tasks — fixes the base model, varies the harness. Sister project of ClawBench, sa

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

Jev-powered, rule-based grader for text files. Runs every rule against every line in parallel. No skimming, no missed lines.

Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI

Benchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests.

RLCDAlignBench: 44 alignment-failure detection benchmarks and code for 'Just Ask Jev' (RLCD zero-shot detector of AI alignment failures)

Caveman prompting, measured. A two-channel evaluation framework for prompt compression and LLM token efficiency: what input vs output compre

开源 AI 应用评测平台，支持 RAG、AI Agent、多轮对话、LLM-as-Judge、接口评测、评测报告和人工盲测。Open-source AI evaluation platform for RAG, AI Agents, multi-turn conversation

Commented SHIP? Full briefs for 3 pile-two AI engineering projects: LLM Ops, Agent Evals, Auto Routing. What to build, what to measure, what

Deletion-first Agent Skill for removing test bloat, verification theater, and speculative fallbacks while preserving behavior.

A work-centered runtime for agentic software engineering. Work persists; agents, context, and graphs assemble around it.

A benchmark for testing whether coding agents can resolve engineering tasks in scientific software

Optimize any AI agent’s skills, tools/MCP, and prompts against your own evals.

Multi-Provider AI Gateway, "Because we have LiteLLM at home" No personal logs by design. Model autodiscovery, Failover groups, High availabi

多智能体小说写作框架 | Multi-agent novel-writing framework — 每个角色都有自己的脑子,每一章都有质量门禁

A design-of-experiments platform for evaluating compound AI systems - find which technique drives quality, by how much, and whether the di

Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

CI for your coding-agent setup (CLAUDE.md, skills, hooks): pinned baselines, a canary on every Claude Code release, model and version matrix

Independent, evidence-based map of when TypeSafe's Jev actually holds up vs. breaks down — real API-call receipts, not a leaderboard. 中文為主的雙

A production GenAI architecture guide for Forward Deployed Engineer (FDE) and Applied AI interviews—RAG, agents, evals, security, and operat

Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions

Make AI agents aim higher without losing control: a Claude Code skill with two bars per task, a cheap gate check before expensive work, and

Your Evals Are Lying To You: a hands-on TypeScript curriculum for LLM evals, from a five-line assertion to a live diagnostic console. By Tej

AI 模拟面试官：每条扣分都强制引用你回答的逐字原文（机器校验，编造引用会被重试/丢弃），四维度加权由确定性代码计算而非模型现编，评分体系设计为可经真人盲评校准 · FastAPI + Vue3 + LangGraph

Real-world browser-agent benchmark: 210 tasks across 107 websites, multi-agent/multi-browser evaluation, reproducible leaderboard and result

Open-source agent reliability platform for discovering, investigating, and fixing production agent failures

Free Collaborative Dynamics Augment for risk-driven software verification, skeptical review, and isolated Agent behavioral evals.

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

How do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did

jev-calibrate, Jev sorularının etiketli örneklerde karar vermeye uygunluğunu ölçen bir TypeScript CLI aracıdır.

Neden yükseliyor · Kesin yükseliş nedeni mevcut verilerden anlaşılamıyor. Repo 6 günlük, 32 yıldızlı ve son 28 günde 11 commit almış; ancak farklı yazar sayısının 1 olması aktivitenin tek geliştiricide toplandığını gösteriyor.

Open-source skills for Claude Code and Codex, with an MCP server, a PyPI package and Suede Awesome Everything, a directory that scores AI ag

Mechanical Governance for LLM Decisions — model-agnostic governance regimes (R1/R2/R3), hard gates, entropy commit-reveal and governance met

discover LLMs punching above their weight

Assessment-driven, local-first learning agent built on an observable Agent Runtime and eval harness—grounded ingestion, trace replay, HITL a

When asked to pick a random number between 1 and 100, ChatGPT does not follow a random uniform distribution

Phoenix-native LLM evaluation, prompt testing, model comparison, red teaming, and observability for Elixir

Lightweight RAG provenance middleware. Verifies every claim in an LLM response is grounded in a retrieved source - without an LLM call.

Combine the results from a panel of models into an enhanced response

Catch AI agents that say "done" when the task isn't. Checks agent-trace/v1 traces with claim-evidence rules, LLM judges and a trained classi

A pure-Python structured memory benchmark for multi-agent LLM systems — context graph vs vector RAG vs raw history dump, five scenarios, 18

Runtime security for AI agents. In-process, zero dependencies, Apache 2.0.

pytest for AI Apps — Fast, free, offline testing and self-improving prompts for Python developers.

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

A preservation-aware benchmark for natural-language SVG repair with deterministic binary rewards and complete model traces.

AIRadar · 每日定时运行的 AI 行业情报工作流

Diff your AI agent's behavior between two runs. See exactly which tool calls, args, costs and outputs changed when you swap models or edit p

AMBER (sealed-in-amber historical replay evaluation): real-event replay, physical sealing, pre-registered scoring — method spec; cases priva

Diagnosing policy grounding in LLM judges via causal erasure. Companion code to the AIES 2026 submission READ.