Havuz

AMBER (sealed-in-amber historical replay evaluation): real-event replay, physical sealing, pre-registered scoring — method spec; cases priva

Diagnosing policy grounding in LLM judges via causal erasure. Companion code to the AIES 2026 submission READ.

Real-world browser-agent benchmark: 210 tasks across 107 websites, multi-agent/multi-browser evaluation, reproducible leaderboard and result

面向小团队的一站式 NLP 小模型平台:数据集版本管理 · 训练 · 评估 · 在线部署 · Badcase 闭环,内置大模型 Prompt 盲测评测(人工 + AI 双指标)。

Open-source agent reliability platform for discovering, investigating, and fixing production agent failures

Safety benchmark for personal AI agents under realistic prompt injection. 120 test cases across 5 professional domains, 3 injection vectors,

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring (EMNLP 2026 Findings) — multi-language benchmark for linter-guided refactoring

Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode,

Adversarial LLM Reversal for hermes-agent

My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementati

A runnable multi-agent process with a built-in evaluation system — repeated rollouts scored on the git diff, not the agent's self-report. Op

The same AI agent pipeline built in Mastra and LangChain. Runs in parallel, measures everything.

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

Free Collaborative Dynamics Augment for risk-driven software verification, skeptical review, and isolated Agent behavioral evals.

Benchmark of AI coding agents building web apps and deploying them to Cloudish, DigitalOcean, Render and Vercel, measuring success, time, to

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

자기개선 후보를 채택할지 파레토 전선으로 판정하는 게이트 + 그 판정을 지탱하는 계측 규율. 사전등록·계기검침·부재증명 실측 포함.

⚽🤖 11 frontier LLMs predicted the entire 2026 World Cup — frozen before kickoff. Live leaderboard: Brier score, bracket points & Polymarket

Production-grade LLM Evaluation & Benchmarking Framework - GPT-4, Claude, Gemini, Mistral. Accuracy, latency, cost, hallucination, reasonin

Typed decisions (choice / score / yes-no) from local Qwen models on Apple Silicon. Probabilities come straight from the logits, no text gene

Smriti (स्मृति): private, local-first agentic RAG with System 1 + System 2 thinking over your notes, books and documents. Guardrails, tracin

A benchmark of AI QA agents finding real bugs in 4 open-source web apps, with 1,396 labeled bugs.

Typed evaluation for OpenCode skills and AI agents: capability, regression and product checks; provider-neutral evidence grading, provenance

Agent skills for designing, training, evaluating and improving application-specific decision systems. Primitive/model selection, data assemb

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio

GEDD turns domain-expert judgment into one failure codebook, then derives your evaluation, your training reward and your response gate from

Open-source benchmark for evaluating LLMs on domain-specific geological reasoning, AI for geology, and source-grounded geological knowledge.

A lightweight LLM evaluation layer that turns subjective model outputs into deterministic decisions by scoring attribution, specificity, rel

What can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.

Multi-model consensus debate via the filesystem. LLMs propose, peer-review, rebut, vote and synthesize a group-confirmed answer. CLI + MCP.

Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lan

Don't trust an Agent. Test it. Open-source lab for agent credit — exams → evidence → explainable, recomputable scores. Built for the day age

The Regression Testing Framework for AI Agents. Replay · Evaluate · Assert · Catch Regressions — in CI. Like Jest for your AI layer.

Distil an expensive LLM API call on a narrow task into a small local model: capture traffic, curate, LoRA fine-tune with MLX on Apple Silico

DodoGuard: Take AI Agents to Production. Test, validate, and operate AI Agents before and after deployment.for Dify, Coze, FastGPT, n8n, Ten

[EMNLP 2026] WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

Open-source skills for Claude Code and Codex, with an MCP server, a PyPI package and Suede Awesome Everything, a directory that scores AI ag

Bounded, evidence-driven SKILL.md evolution under frozen evaluation and safety contracts.

pytest for AI Apps — Fast, free, offline testing and self-improving prompts for Python developers.

Same-brief agentic build comparison: GPT-6 Astra High vs Gemini 3.8 Flash High building a playable 3D AI civilization simulation game.

Evidence-first infrastructure for reproducible, isolated executable evaluation and online rewards.

Turn a production incident into a structured 9-section LLM response (severity, root cause, mitigation, postmortem). Ships with a 5-scenario

Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate,

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

skilled: open-source agent skill evaluation CLI and MCP server for Codex and Claude Code. Test skills, detect regressions, and improve with

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs tha

Self-hosted decision model (selfjev-4b) with a Jev-compatible API: Python SDK, server, fine-tuning and RLCD.