Havuz

End-to-end Langfuse workshop using a TypeScript Agent to teach the AI engineering loop: tracing, prompt management, monitoring, datasets, ex

Multi-Provider AI Gateway, "Because we have LiteLLM at home" No personal logs by design. Model autodiscovery, Failover groups, High availabi

Hybrid rule-based and context-aware LLM evaluation system for Chinese-to-Japanese game localization QA.

Открытый бенчмарк LLM: какая нейросеть лучше пишет код 1С:Предприятие (BSL). Объективная оценка LLM по методике SMOP с реальным исполнением

Assessment-driven, local-first learning agent built on an observable Agent Runtime and eval harness—grounded ingestion, trace replay, HITL a

多智能体小说写作框架 | Multi-agent novel-writing framework — 每个角色都有自己的脑子,每一章都有质量门禁

Grade every agent, scraper and model in a claim's chain — provenance, trust scoring and audit evidence for LLM pipelines

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

Chinese-first LLM jailbreak benchmark & red-team toolkit | 中文LLM安全评测框架 | by POJIA.AI

PaperBench High-Difficulty and Expanded Tasks: paper reproduction beyond machine learning. 93 tasks in 12 research areas, run in domain-nati

Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic sco

When asked to pick a random number between 1 and 100, ChatGPT does not follow a random uniform distribution

A design-of-experiments platform for evaluating compound AI systems - find which technique drives quality, by how much, and whether the di

Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate,

Find the best AI models you can run

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

Phoenix-native LLM evaluation, prompt testing, model comparison, red teaming, and observability for Elixir

Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

Jev-powered, rule-based grader for text files. Runs every rule against every line in parallel. No skimming, no missed lines.

Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI

Lightweight RAG provenance middleware. Verifies every claim in an LLM response is grounded in a retrieved source - without an LLM call.

PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

Cooking AGI / AGI 大锅烩 · Bilingual (English / 中文) ML & LLM notes with worked examples, step-by-step figures and coding practice. Free to read

Benchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests.

Combine the results from a panel of models into an enhanced response

DiagSafe: diagnostic LLM safety benchmark with prompt-level rubrics (3,489 bilingual prompts, 84-label taxonomy)

Myrtia — agent-native red teaming for LLMs and AI agents: a deliberating council of attacker models (propose, cross-critique, Borda vote, ch

CI for your coding-agent setup (CLAUDE.md, skills, hooks): pinned baselines, a canary on every Claude Code release, model and version matrix

Self-hosted decision model (selfjev-4b) with a Jev-compatible API: Python SDK, server, fine-tuning and RLCD.

Catch AI agents that say "done" when the task isn't. Checks agent-trace/v1 traces with claim-evidence rules, LLM judges and a trained classi

A pure-Python structured memory benchmark for multi-agent LLM systems — context graph vs vector RAG vs raw history dump, five scenarios, 18

RLCDAlignBench: 44 alignment-failure detection benchmarks and code for 'Just Ask Jev' (RLCD zero-shot detector of AI alignment failures)

Runtime security for AI agents. In-process, zero dependencies, Apache 2.0.

Independent, evidence-based map of when TypeSafe's Jev actually holds up vs. breaks down — real API-call receipts, not a leaderboard. 中文為主的雙

pytest for AI Apps — Fast, free, offline testing and self-improving prompts for Python developers.

A production GenAI architecture guide for Forward Deployed Engineer (FDE) and Applied AI interviews—RAG, agents, evals, security, and operat

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

A lightweight, zero-dependency Python checkpoint layer for structured LLM output reliability.

A preservation-aware benchmark for natural-language SVG repair with deterministic binary rewards and complete model traces.

Teach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it te

Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions

Make AI agents aim higher without losing control: a Claude Code skill with two bars per task, a cheap gate check before expensive work, and

Behavioral regression testing for AI apps and agents: pytest plugin, behavior diff, CI gate, and document extraction metrics

AIRadar · 每日定时运行的 AI 行业情报工作流

Your Evals Are Lying To You: a hands-on TypeScript curriculum for LLM evals, from a five-line assertion to a live diagnostic console. By Tej

Diff your AI agent's behavior between two runs. See exactly which tool calls, args, costs and outputs changed when you swap models or edit p

AI 模拟面试官：每条扣分都强制引用你回答的逐字原文（机器校验，编造引用会被重试/丢弃），四维度加权由确定性代码计算而非模型现编，评分体系设计为可经真人盲评校准 · FastAPI + Vue3 + LangGraph

AMBER (sealed-in-amber historical replay evaluation): real-event replay, physical sealing, pre-registered scoring — method spec; cases priva

Diagnosing policy grounding in LLM judges via causal erasure. Companion code to the AIES 2026 submission READ.