Havuz

Diagnosing policy grounding in LLM judges via causal erasure. Companion code to the AIES 2026 submission READ.

Real-world browser-agent benchmark: 210 tasks across 107 websites, multi-agent/multi-browser evaluation, reproducible leaderboard and result

Qwen3.8-27B on RTX 5090 Laptop 24GB — TWIN-TURBO NVFP4 (native llama.cpp) vs Coder390 Q3 LynnStyle (.ninfer + NInfer engine in WSL2/Docker):

EC2-Bench (EC²-Bench): benchmark for edge-cloud collaboration in long-horizon LLM agents — task performance, resource usage & privacy exposu

Bittensor Subnet Repository maintainer agent + a GitHub-history benchmark that scores it — can an agent make the maintainer decisions a stro

Evidence profiles for AI-generated RTL that expose what functional benchmarks miss. Seeking collaborators, design partners, and validation

SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

[NeurIPS 2026 Meta-Agents] SAEScientist-Bench: Can AI agents conduct autonomous SAE interpretability research?

Evaluation code for SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images (NeurIPS 2026).

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

Scenarios and tasks for evaluating AI agents on real-world AWS automation tasks. Use the aws-bench framework (https://github.com/aws-bench/a

Caveman prompting, measured. A two-channel evaluation framework for prompt compression and LLM token efficiency: what input vs output compre

Reproducible benchmark for measuring Jev reranking quality, latency, and cost in RAG

AcademiClaw: When Students Set Challenges for AI Agents — a bilingual benchmark of 80 university student-sourced academic tasks.

CLI for benchmarking PostgreSQL queries before and after optimization and generating an HTML report.

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving

Safety benchmark for personal AI agents under realistic prompt injection. 120 test cases across 5 professional domains, 3 injection vectors,

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring (EMNLP 2026 Findings) — multi-language benchmark for linter-guided refactoring

OMCBench is a benchmark suite for evaluating malicious-code detection capabilities. The benchmark consists of a labeled set of 800 Python an

Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode,

Use many on device and cloud models with Apple's Foundation Models framework in iOS and macOS with a full chat experience.

Precision-aware retrieval benchmark for LLM memory systems.

Generative UI benchmark

A Minesweeper benchmark for LLM agents. One identical board, up to nine models in parallel, one clock, one tool layer.

Manipulation benchmarks test acting to change; EmbodiedLab tests acting to know. Given only an objective and the apparatus, a robot runs its

A 125-task benchmark for knowledge-mediated implicit associations and retrieval blind spots in long-term agent memory.

Catch local LLMs spilling into the CPU. One run shows GPU placement, prefill, decode, memory, power and a verdict.

Official code for EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents.

Profine automatically profiles and optimizes PyTorch training jobs on real GPUs, delivering measurable speedups and lower GPU costs before t

Benchmark model duels in Doom. An MCP-native arena for real-time model-vs-model evaluations.

Red Team AI Benchmark: Evaluating LLMs for authorized offensive-security tasks

[NeurIPS 2026 Verify-Agents] SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

The Virtual Filesystem Interface for AI Agents

AMD Radeon 9700 AI PRO RDNA multi GPU Tools for llm/vLLM inference, tuning, overclocking, undervolting and LLM models benchmarking

Cross-Dataset Reimplementation of NAVSIM

⚡ Adaptive multi-model LLM router — 80+ providers, Jev System One single-pass routing (model=jev-auto), pheromone-trail failover, parallel e

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Simplified Technical English for Code - extract, adapt, and enforce ASD-STE100 rules for software documentation. 9-agent pipeline, 59-test b

What I run on my inference server: Qwen3.8-Flash-Next on two RTX 3090s, and how I got here

A longitudinal reliability benchmark foundation for agent lifespan engineering.

The daily arena where AI agents clash to rank tomorrow’s S&P 500 winners and losers. Their only judge is the future — the one truth that sim

OR-Space: a full-lifecycle workspace benchmark for industrial optimization agents

Benchmark of AI coding agents building web apps and deploying them to Cloudish, DigitalOcean, Render and Vercel, measuring success, time, to

[ICML 2026] Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

Accepted at ISCSLP 2026 · Target-level benchmark for raw-input Chinese news TTS pronunciation

EMNLP 2026 Industry Track | A benchmark for shopping agents on long-horizon tasks with distributed hidden intent.

dsh-model-master（模型选择大师）— model selector for DeepSeek Harness: search across providers, probe any model with a real streaming request to mea

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Live Benchmark for long Horizon tool using Agentic Systems