Havuz

MathPinpoint: problem-level math retrieval dataset. Find the web page that solves the same problem under the same conditions, correctly. 3.5

A composable framework for reproducible evaluation of long-running agents.

Compare two local Persian (Farsi) speech-to-text models side by side: a Persian Whisper fine-tune vs. general large-v3, on Apple Silicon.

Open benchmark on observability tasks built on Harbor

One-shot LLM eval cases by NAGI STUDIO - same prompt, different agents (model + harness), runnable artifacts side by side.

Χ-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

Benchmarking repository-borne prompt injection attacks and lightweight defenses for local coding agents. DL4C @ ICML 2026.

Official code for "CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges"

A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The ga

Reproducible on-device LLM benchmarks for Apple Silicon (iPhone 17 Pro, M4 Max): Apple Core AI, MLX, llama.cpp, LiteRT-LM and Core ML on the

LLM benchmark and leaderboard for narrator-bias sycophancy, opposite-narrator contradictions, and judgment consistency.

Measurement harness for proxy providers, browser engines and scraping targets. Every claim carries the run it came from.

Audio-Oscar is a multi-agent framework for generating long-form, controllable audio from complex audio scene descriptions.

The benchmark tasks and evaluation harness for "PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments".

Benchmark for comparing agent harnesses on everyday online tasks — fixes the base model, varies the harness. Sister project of ClawBench, sa

🔥 Orbia: Do Generated Videos Form a Consistent 3D World?

Continuous benchmarks for multi-device CoW filesystems (btrfs, ZFS, bcachefs): snapshots, aging, compression, redundancy layouts

Evaluating frontier models on real-world ai-infra engineering tasks

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

🚀 Automated scraper & injector for 100+ free AI coding models into 9router combos. Live pre-tested, ranked by coding benchmarks & zero-conf

📊 Automated hourly benchmarks for 20+ NVIDIA NIM models — interactive dashboard, zero infra, self-hostable. Open-source & community-driven.

Can LLM/MARL agents coordinate in long-horizon, open-ended tasks? NeurIPS 2026 (Evaluations and Datasets).

Открытый бенчмарк LLM: какая нейросеть лучше пишет код 1С:Предприятие (BSL). Объективная оценка LLM по методике SMOP с реальным исполнением

Best Open Source Aim Trainer Benchmark Tool 2026

Aim Genetics Lab Open Research Tools 2026: Build Precision Breeding Models

[ACL 2026] PIArena: A Platform for Prompt Injection Evaluation

多智能体小说写作框架 | Multi-agent novel-writing framework — 每个角色都有自己的脑子,每一章都有质量门禁

ICML 2026 benchmark and MechVL models for multimodal understanding of mechanical engineering drawings.

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Specul

Aim Genetics Lab: CRISPR Gene Editing & Bioinformatics Tools 2026

[ECCV 2026] AerialMetric: Benchmarking and Adapting UAV Monocular Metric Depth Estimation in the Real World

Official benchmark reproduction code for the SAG paper

TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 Evaluations and Datasets Track)

Move whole tables between databases fast — Postgres, MySQL, ClickHouse, BigQuery. Rust engine, one-line Python API, bounded memory.

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

Chinese-first LLM jailbreak benchmark & red-team toolkit | 中文LLM安全评测框架 | by POJIA.AI

Reproducible speech-to-text benchmark: WER, latency and cost for Velma, Deepgram, AssemblyAI, Chirp 3, MAI-Transcribe 2, Moonshine and whisp

A robot-painting benchmark for multimodal language models: reference images, text prompts, and custom goals. Built on AgentArk.

Benchmark for the quality of LLM-generated test suites — anti-fragility, rigor, mocking discipline, reuse — scored against human baselines,

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4

PureDocBench: source-traceable benchmark for document parsing across clean, degraded, and real-world settings

Open-source benchmark of 5 intentionally vulnerable web apps with 100 baseline vulnerabilities for evaluating AI penetration testing agents.

DolphinBench - Mapping the Pareto frontier of Agent Memory

PaperBench High-Difficulty and Expanded Tasks: paper reproduction beyond machine learning. 93 tasks in 12 research areas, run in domain-nati

Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic sco

Generate degraded speech datasets for noise-robust ASR benchmarking

Community benchmark database for running LLMs on Apple Silicon Macs

MODA: open fashion retrieval benchmark and models by Hopit AI. MODA (203M, open source), MODA Pro Lite (213M, open weights), MODA Pro (hoste

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

Create your own benchmark for coding AI Agents