Havuz

Real-world browser-agent benchmark: 210 tasks across 107 websites, multi-agent/multi-browser evaluation, reproducible leaderboard and result

面向小团队的一站式 NLP 小模型平台:数据集版本管理 · 训练 · 评估 · 在线部署 · Badcase 闭环,内置大模型 Prompt 盲测评测(人工 + AI 双指标)。

Caveman prompting, measured. A two-channel evaluation framework for prompt compression and LLM token efficiency: what input vs output compre

Open-source agent reliability platform for discovering, investigating, and fixing production agent failures

Safety benchmark for personal AI agents under realistic prompt injection. 120 test cases across 5 professional domains, 3 injection vectors,

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring (EMNLP 2026 Findings) — multi-language benchmark for linter-guided refactoring

Same-brief agentic build comparison: GPT-6 Astra High vs Gemini 3.8 Flash High building a playable 3D AI civilization simulation game.

Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode,

Adversarial LLM Reversal for hermes-agent

My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementati

Intents is a native macOS workbench for evaluating Apple's on-device Foundation Models.

A runnable multi-agent process with a built-in evaluation system — repeated rollouts scored on the git diff, not the agent's self-report. Op

The same AI agent pipeline built in Mastra and LangChain. Runs in parallel, measures everything.

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

Free Collaborative Dynamics Augment for risk-driven software verification, skeptical review, and isolated Agent behavioral evals.

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

Benchmark of AI coding agents building web apps and deploying them to Cloudish, DigitalOcean, Render and Vercel, measuring success, time, to

How do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Commented SHIP? Full briefs for 3 pile-two AI engineering projects: LLM Ops, Agent Evals, Auto Routing. What to build, what to measure, what

자기개선 후보를 채택할지 파레토 전선으로 판정하는 게이트 + 그 판정을 지탱하는 계측 규율. 사전등록·계기검침·부재증명 실측 포함.

⚽🤖 11 frontier LLMs predicted the entire 2026 World Cup — frozen before kickoff. Live leaderboard: Brier score, bracket points & Polymarket

Red team vs blue team benchmark for scientific paper verification. Plant defects into open access biomedical papers, build gates to catch th

Production-grade LLM Evaluation & Benchmarking Framework - GPT-4, Claude, Gemini, Mistral. Accuracy, latency, cost, hallucination, reasonin

Typed decisions (choice / score / yes-no) from local Qwen models on Apple Silicon. Probabilities come straight from the logits, no text gene

Smriti (स्मृति): private, local-first agentic RAG with System 1 + System 2 thinking over your notes, books and documents. Guardrails, tracin

A benchmark of AI QA agents finding real bugs in 4 open-source web apps, with 1,396 labeled bugs.

Typed evaluation for OpenCode skills and AI agents: capability, regression and product checks; provider-neutral evidence grading, provenance

Agent skills for designing, training, evaluating and improving application-specific decision systems. Primitive/model selection, data assemb

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio

GEDD turns domain-expert judgment into one failure codebook, then derives your evaluation, your training reward and your response gate from

Open-source benchmark for evaluating LLMs on domain-specific geological reasoning, AI for geology, and source-grounded geological knowledge.

A lightweight LLM evaluation layer that turns subjective model outputs into deterministic decisions by scoring attribution, specificity, rel

What can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.

Multi-model consensus debate via the filesystem. LLMs propose, peer-review, rebut, vote and synthesize a group-confirmed answer. CLI + MCP.

Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lan

Don't trust an Agent. Test it. Open-source lab for agent credit — exams → evidence → explainable, recomputable scores. Built for the day age

The Regression Testing Framework for AI Agents. Replay · Evaluate · Assert · Catch Regressions — in CI. Like Jest for your AI layer.

Distil an expensive LLM API call on a narrow task into a small local model: capture traffic, curate, LoRA fine-tune with MLX on Apple Silico

Turn your Win, Mac or DGX Spark into an AI hotspot. Nightshift races hundreds of attempts overnight and keeps what passes. Plus the open con

Open-source skills for Claude Code and Codex, with an MCP server, a PyPI package and Suede Awesome Everything, a directory that scores AI ag

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate,

skilled: open-source agent skill evaluation CLI and MCP server for Codex and Claude Code. Test skills, detect regressions, and improve with

Self-hosted decision model (selfjev-4b) with a Jev-compatible API: Python SDK, server, fine-tuning and RLCD.

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs tha