Havuz

Swap one RAG pipeline stage, hold the rest fixed, and measure whether the gain survives composition. The framework behind "Retrieval Augment

DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with

SciPIvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The ga

Measures how much code LLM agents waste. 5 tasks, sandboxed, weekly snapshots, contestable references.

How much does an image model change what you told it to leave alone? Chained-edit benchmark for image models + installable scorer and Claude

Catches multilingual voice agents placing the wrong order. τ-bench style benchmark + CI gate for voice ordering agents in English and Hingli

Behavioral regression testing for AI apps and agents: pytest plugin, behavior diff, CI gate, and document extraction metrics

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verifi

SWE-Serve: an agentic benchmark of 53 production inference-engineering tasks derived from merged SGLang pull requests, run with Harbor.

Benchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.

A temporal visual reasoning benchmark over egocentric robot manipulation video — 39 scenarios, 7 dimensions, 34,713 questions

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence

deedchain, browser agent raporlarındaki iddiaları kaydedilmiş ground truth ile karşılaştırarak rapor doğruluğunu ölçen bir benchmark ve Python kütüphanesidir.

Neden yükseliyor · Repo yalnızca 23 günlük ve 28 yıldızlı; son 28 gün commit ve farklı yazar verisi ölçülmediği için yükselişin arkasındaki aktivite veya benimsenme düzeyi doğrulanamıyor. Rapor doğruluğunu ayrı bir değerlendirme konusu yapması dikkat çekici olsa da README'deki “first benchmark” iddiası bağımsız olarak doğrulanmış değil.

Give Vercel AI SDK agents a real persistent filesystem with Amazon S3 Files.

PromptLedger, prompt şablonlarını fingerprint ile sürümleyip aday yanıtları karşılaştıran ve sonuçları tekrarlanabilir raporlara dönüştüren bir Python aracı ile Go CLI sunuyor.

Neden yükseliyor · 25 günlük bir repo için 35 yıldız dikkat çekiyor; ancak son 28 günde yalnızca 1 commit ve 1 farklı yazar var. Bu nedenle ilginin kalıcı benimsemeye dönüştüğünü gösteren yeterli aktivite kanıtı henüz yok.

Semantic cache strategy bench: replay recorded request traces through candidate caching policies and measure which one actually pays off.

LoopArena, uzun süreli yazılım görevlerinde bir coding agent’ı yönlendiren modelleri ortak koşullarda karşılaştırmak için benchmark ve çalıştırma altyapısı sunuyor.

Neden yükseliyor · Coding agents’ın uzun süreli iş akışlarında nasıl yönlendirileceği güncel bir araştırma konusu; LoopArena bu konuya açık protokol, benchmark sonuçları ve bir arXiv paper ile dikkat çekiyor. Ancak 146 yıldız, 28 günlük repo yaşı ve son 28 günde tek farklı yazar, ilginin henüz yaygın benimsemeye dönüştüğünü göstermiyor.

The experience graph for AI agents. Self-hosted memory, human-reviewed decisions, and outcome-backed datasets for model evaluation and learn

skilled: open-source agent skill evaluation CLI and MCP server for Codex and Claude Code. Test skills, detect regressions, and improve with

EvoTrace, Claude Code ve Codex oturumlarını inceleyip eğitim, değerlendirme ve doğrulama için yeniden kullanılabilir varlıklara dönüştürür.

Neden yükseliyor · 39 günde 292 yıldız, coding agent oturumlarından eğitim ve değerlendirme varlığı üretme fikrine ilgi olduğunu gösteriyor; ancak bu ilginin nedeni ölçümlerden anlaşılamıyor. Son 28 günde commit ve farklı yazar olmaması, ilginin devam eden geliştirme veya benimsemeyle desteklendiğine dair kanıt sunmuyor.

HarnessEval-W: Agentifying the Evaluation of Visual Worlds

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

Benchmark for repository-level code understanding: 500 open-ended questions from real GitHub Discussions across 50 repositories, graded by a

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

Run cyber-AI agents against real missions in isolated environments. Record everything as OpenTelemetry evidence. Grade the evidence, not the

A composable framework for reproducible evaluation of long-running agents.

The trust layer for your AI agent on any platform. Trace and evaluate your agent. Full Observability. Run Model portability analysis. Autotu

EvalPort — The Open LLM Evaluation Standard. Portable test cases, graders, suites, and results across DeepEval, Promptfoo, Inspect AI, and m

Track 42,331+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates.

AI Agent / LLM application engineer Chinese interview question bank: 2,038 questions with reference answers

An auditable and reproducible evaluation suite for LeWM-compatible latent world models, with official-compatible and difficulty-controlled p

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Cairn runs your AI and ML pipelines behind budgets, policy, approvals and an audit trail.

PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

Unified World Model Inference & Evaluation Infrastructure

Red Team AI Benchmark: Evaluating LLMs for authorized offensive-security tasks

Compare web access APIs (web unblockers / scrape APIs) head-to-head against a fixed suite of ~90 real-world, hard-to-scrape URLs

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio ·

Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim benchmark.

Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation,

Research prototype for trace-based observability and failure analysis in retrieval-augmented generation.

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model

The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it,

Open benchmark datasets republished in stressed form to evaluate ML/LLM robustness. Curated by Santander AI Lab.

WRBench: camera-controlled generation and diagnostic evaluation of video world models.

Alignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface

Mutation testing for your LLM evals — degrade the system under test and check whether your evals would actually catch a regression.

An external memory for LLMs: a bi-temporal knowledge graph with hybrid (keyword + vector) retrieval and a self-improving reflection loop. Be