Havuz

Can LLM/MARL agents coordinate in long-horizon, open-ended tasks? NeurIPS 2026 (Evaluations and Datasets).

A JAX transport-economy learning environment for route planning, cargo flow, financing, and replayable agent benchmarks.

Blind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions.

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

Compare two local Persian (Farsi) speech-to-text models side by side: a Persian Whisper fine-tune vs. general large-v3, on Apple Silicon.

Audio-Oscar is a multi-agent framework for generating long-form, controllable audio from complex audio scene descriptions.

Repository for the implementation of MacArena

Hardware-in-the-loop benchmark for agentic embedded AI deployment. Agents iteratively edit model or firmware artifacts, the framework compil

Open-source CLI, API, web UI, and reproducible benchmarks for probabilistic AI-generated image detection.

⚡ DROS™ VajraClaw (Hacker Edition): Free Multi-Agent Runtime Security & Docker Governance Gateway with W3C DID & <1μs Enforcement

Benchmark for the quality of LLM-generated test suites — anti-fragility, rigor, mocking discipline, reuse — scored against human baselines,

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

Official benchmark reproduction code for the SAG paper

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? (EMNLP 2026 Findings)

A benchmark for AI-driven CAD generation and editing

OMCBench is a benchmark suite for evaluating malicious-code detection capabilities. The benchmark consists of a labeled set of 800 Python an

Precision-aware retrieval benchmark for LLM memory systems.

A benchmarking suite for Agent Skills (SKILL.md files) that measures whether a skill actually improves LLM output quality, and by how much.

Synthetic fraud graph generator for benchmarking graph-based fraud detection models in financial services.

NeurIPS2026 , Scalable annotation pipeline for action-aglined fine-grained instruciton for Visual-language-Action model

Benchmark and correctness oracle for matching engines, with CI/CD checks against 100 billion order messages of precomputed consensus. Five

Caveman prompting, measured. A two-channel evaluation framework for prompt compression and LLM token efficiency: what input vs output compre

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Verification-gated skill routing and self-improvement harness for Hermes-style agent skills

A preservation-aware benchmark for natural-language SVG repair with deterministic binary rewards and complete model traces.

Community-driven benchmark suite for MLX inference engines on Apple Silicon

AMD Radeon 9700 AI PRO RDNA multi GPU Tools for llm/vLLM inference, tuning, overclocking, undervolting and LLM models benchmarking

Generate degraded speech datasets for noise-robust ASR benchmarking

The daily arena where AI agents clash to rank tomorrow’s S&P 500 winners and losers. Their only judge is the future — the one truth that sim

🔍 The hardest search benchmark in the wild — vague, multi-turn, proactive. 200 long-horizon tasks with persona-driven progressive disclosure

This dataset provides metadata, official statistics, and official play-by-play annotations for full-length NBA game videos available on YouT

A longitudinal reliability benchmark foundation for agent lifespan engineering.

Open-source benchmark of 5 intentionally vulnerable web apps with 100 baseline vulnerabilities for evaluating AI penetration testing agents.

Decomposing and measuring evaluation awareness in existing benchmarks and our proposed EvalAwareBench.

Open-source computational lithography workflow, manufacturability benchmarking, and foundation model data engine for advanced EUV/curvilinea

Layang is a gRPC-first, workspace-based API workbench for testing, mocking, benchmarking, documenting, and automating API workflows across g

Practical local LLM recipes and benchmarks for RTX 5060 Ti setups

A benchmark for evaluating LLM × harness performance.

⚡ Adaptive multi-model LLM router — 80+ providers, Jev System One single-pass routing (model=jev-auto), pheromone-trail failover, parallel e

[EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted A

Self-evolving agentic reward framework for image-editing evaluation — 47.4% on EditReward-Bench from only 100 preference demos, no reward-mo

[ACM MM 2026 ORAL] PROVE: A Perceptual RemOVal cohErence Benchmark for Visual Media. Benchmark and removal metrics for image/video object r

Project page for AV-Phys Bench: Do Joint Audio-Video Generation Models Understand Physics?

面向 LLM 交易 Agent 的抗污染回测、确定性回放与训练轨迹生成，支持 A 股研究｜Leakage-resistant backtesting, deterministic replay & training trajectory generation for LLM tr

Cross-Dataset Reimplementation of NAVSIM

Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark,

[NeurIPS 2026] SpatialBench: Is Your Spatial Foundation Model an All-Round Player?

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

Protein Fold Classification at Scale: Benchmarking and Pretraining (ICML 2026 oral)