Havuz

Developer and product skills for Claude Code, Cursor and Codex: 1-file AI evals and Chrome extension store assets.

甲作 Atelier：手机端 AI 美甲推荐与虚拟试戴；仓库公开其中的轻量评测工作台。

Real-world browser-agent benchmark: 210 tasks across 107 websites, multi-agent/multi-browser evaluation, reproducible leaderboard and result

面向小团队的一站式 NLP 小模型平台:数据集版本管理 · 训练 · 评估 · 在线部署 · Badcase 闭环,内置大模型 Prompt 盲测评测(人工 + AI 双指标)。

Caveman prompting, measured. A two-channel evaluation framework for prompt compression and LLM token efficiency: what input vs output compre

Open-source agent reliability platform for discovering, investigating, and fixing production agent failures

Safety benchmark for personal AI agents under realistic prompt injection. 120 test cases across 5 professional domains, 3 injection vectors,

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring (EMNLP 2026 Findings) — multi-language benchmark for linter-guided refactoring

Same-brief agentic build comparison: GPT-6 Astra High vs Gemini 3.8 Flash High building a playable 3D AI civilization simulation game.

Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode,

Adversarial LLM Reversal for hermes-agent

My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementati

Intents is a native macOS workbench for evaluating Apple's on-device Foundation Models.

Free interactive tools for data scientists to learn MLOps, ML system design and agentic analytics: a production ML system map, ML project st

A runnable multi-agent process with a built-in evaluation system — repeated rollouts scored on the git diff, not the agent's self-report. Op

The same AI agent pipeline built in Mastra and LangChain. Runs in parallel, measures everything.

A benchmark for AI agents as teammates. Paper: https://arxiv.org/abs/2611.02633

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

Benchmark and controlled study of LLM-based novelty judges for research ideas. Code and data for "Old Ideas, Novel Problems" (arXiv 2610.020

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

开源 AI 应用评测平台，支持 RAG、AI Agent、多轮对话、LLM-as-Judge、接口评测、评测报告和人工盲测。Open-source AI evaluation platform for RAG, AI Agents, multi-turn conversation

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

Free Collaborative Dynamics Augment for risk-driven software verification, skeptical review, and isolated Agent behavioral evals.

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

Benchmark of AI coding agents building web apps and deploying them to Cloudish, DigitalOcean, Render and Vercel, measuring success, time, to

How do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

Commented SHIP? Full briefs for 3 pile-two AI engineering projects: LLM Ops, Agent Evals, Auto Routing. What to build, what to measure, what

자기개선 후보를 채택할지 파레토 전선으로 판정하는 게이트 + 그 판정을 지탱하는 계측 규율. 사전등록·계기검침·부재증명 실측 포함.

⚽🤖 11 frontier LLMs predicted the entire 2026 World Cup — frozen before kickoff. Live leaderboard: Brier score, bracket points & Polymarket

Red team vs blue team benchmark for scientific paper verification. Plant defects into open access biomedical papers, build gates to catch th

Production-grade LLM Evaluation & Benchmarking Framework - GPT-4, Claude, Gemini, Mistral. Accuracy, latency, cost, hallucination, reasonin

Mutation testing for your LLM evals — degrade the system under test and check whether your evals would actually catch a regression.

Typed decisions (choice / score / yes-no) from local Qwen models on Apple Silicon. Probabilities come straight from the logits, no text gene

Smriti (स्मृति): private, local-first agentic RAG with System 1 + System 2 thinking over your notes, books and documents. Guardrails, tracin

A benchmark of AI QA agents finding real bugs in 4 open-source web apps, with 1,396 labeled bugs.

Typed evaluation for OpenCode skills and AI agents: capability, regression and product checks; provider-neutral evidence grading, provenance

Agent skills for designing, training, evaluating and improving application-specific decision systems. Primitive/model selection, data assemb

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio

GEDD turns domain-expert judgment into one failure codebook, then derives your evaluation, your training reward and your response gate from

Open-source benchmark for evaluating LLMs on domain-specific geological reasoning, AI for geology, and source-grounded geological knowledge.

A lightweight LLM evaluation layer that turns subjective model outputs into deterministic decisions by scoring attribution, specificity, rel

What can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.

Multi-model consensus debate via the filesystem. LLMs propose, peer-review, rebut, vote and synthesize a group-confirmed answer. CLI + MCP.

Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lan

Don't trust an Agent. Test it. Open-source lab for agent credit — exams → evidence → explainable, recomputable scores. Built for the day age

The Regression Testing Framework for AI Agents. Replay · Evaluate · Assert · Catch Regressions — in CI. Like Jest for your AI layer.

Distil an expensive LLM API call on a narrow task into a small local model: capture traffic, curate, LoRA fine-tune with MLX on Apple Silico