Havuz

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

开源 AI 应用评测平台，支持 RAG、AI Agent、多轮对话、LLM-as-Judge、接口评测、评测报告和人工盲测。Open-source AI evaluation platform for RAG, AI Agents, multi-turn conversation

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

How do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did

Commented SHIP? Full briefs for 3 pile-two AI engineering projects: LLM Ops, Agent Evals, Auto Routing. What to build, what to measure, what

EvalForge, AI evaluation sonuçlarını baseline karşılaştırmaları ve CI quality gate’leriyle değerlendirip standart rapor formatlarında yayımlar.

Neden yükseliyor · RAG evaluation sonuçlarını CI’da denetlenebilir kanıta ve release gate’lerine bağlama ihtiyacına odaklanıyor; repo 34 günde 211 yıldız almış. Ancak README bağımsız doğrulanmış downstream benimsemenin henüz bulunmadığını belirtiyor ve son 28 günde katkı tek yazardan gelmiş; ilginin gerçek kullanımın kanıtı olduğu söylenemez.

GameOps Investigator, oyun analitiğindeki düşüş ve anomali vakalarını read-only SQL, istatistiksel karşılaştırmalar ve incelemeye açık raporlarla araştıran bir araçtır.

Neden yükseliyor · 76 yıldız ve 28 günde 23 commit ilgi gördüğünü gösteriyor; oyun analitiğini agent ve MCP yaklaşımıyla birleştirmesi dikkat çekmiş olabilir. Yükselişin kesin nedenini gösteren veri yok; commit’lerin tek yazardan gelmesi topluluk katkısının henüz sınırlı olduğuna işaret ediyor.

jev-calibrate, Jev sorularının etiketli örneklerde karar vermeye uygunluğunu ölçen bir TypeScript CLI aracıdır.

Neden yükseliyor · Kesin yükseliş nedeni mevcut verilerden anlaşılamıyor. Repo 6 günlük, 32 yıldızlı ve son 28 günde 11 commit almış; ancak farklı yazar sayısının 1 olması aktivitenin tek geliştiricide toplandığını gösteriyor.

Turn your Win, Mac or DGX Spark into an AI hotspot. Nightshift races hundreds of attempts overnight and keeps what passes. Plus the open con

Engineering deterministic, production-grade systems around non-deterministic LLMs — FSM, durable execution, retries, DAGs, agent runtimes, m

AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?

A work-centered runtime for agentic software engineering. Work persists; agents, context, and graphs assemble around it.

Don't make LLMs honest. Make every factual claim auditable. — An LLM Claim Auditing Layer with T1-T7 truth gradients. 98.1% business effecti

A benchmark for testing whether coding agents can resolve engineering tasks in scientific software

Mechanical Governance for LLM Decisions — model-agnostic governance regimes (R1/R2/R3), hard gates, entropy commit-reveal and governance met

Paper artifacts, figures, aggregate data, and share cards for the DXRG paper: What LLM Trading Agents Actually Do in Production (six months,

Live trading arena for LLM agents: public reasoning, real market data, virtual capital

discover LLMs punching above their weight

End-to-end Langfuse workshop using a TypeScript Agent to teach the AI engineering loop: tracing, prompt management, monitoring, datasets, ex

Hybrid rule-based and context-aware LLM evaluation system for Chinese-to-Japanese game localization QA.

Assessment-driven, local-first learning agent built on an observable Agent Runtime and eval harness—grounded ingestion, trace replay, HITL a

多智能体小说写作框架 | Multi-agent novel-writing framework — 每个角色都有自己的脑子,每一章都有质量门禁

Grade every agent, scraper and model in a claim's chain — provenance, trust scoring and audit evidence for LLM pipelines

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

PaperBench High-Difficulty and Expanded Tasks: paper reproduction beyond machine learning. 93 tasks in 12 research areas, run in domain-nati

Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic sco

When asked to pick a random number between 1 and 100, ChatGPT does not follow a random uniform distribution

A design-of-experiments platform for evaluating compound AI systems - find which technique drives quality, by how much, and whether the di

Phoenix-native LLM evaluation, prompt testing, model comparison, red teaming, and observability for Elixir

Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

Lightweight RAG provenance middleware. Verifies every claim in an LLM response is grounded in a retrieved source - without an LLM call.

PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

Benchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests.

Combine the results from a panel of models into an enhanced response

CI for your coding-agent setup (CLAUDE.md, skills, hooks): pinned baselines, a canary on every Claude Code release, model and version matrix

Catch AI agents that say "done" when the task isn't. Checks agent-trace/v1 traces with claim-evidence rules, LLM judges and a trained classi

A pure-Python structured memory benchmark for multi-agent LLM systems — context graph vs vector RAG vs raw history dump, five scenarios, 18

RLCDAlignBench: 44 alignment-failure detection benchmarks and code for 'Just Ask Jev' (RLCD zero-shot detector of AI alignment failures)

Runtime security for AI agents. In-process, zero dependencies, Apache 2.0.

Independent, evidence-based map of when TypeSafe's Jev actually holds up vs. breaks down — real API-call receipts, not a leaderboard. 中文為主的雙

A production GenAI architecture guide for Forward Deployed Engineer (FDE) and Applied AI interviews—RAG, agents, evals, security, and operat

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

A preservation-aware benchmark for natural-language SVG repair with deterministic binary rewards and complete model traces.

Make AI agents aim higher without losing control: a Claude Code skill with two bars per task, a cheap gate check before expensive work, and

Behavioral regression testing for AI apps and agents: pytest plugin, behavior diff, CI gate, and document extraction metrics

AIRadar · 每日定时运行的 AI 行业情报工作流

Your Evals Are Lying To You: a hands-on TypeScript curriculum for LLM evals, from a five-line assertion to a live diagnostic console. By Tej

Diff your AI agent's behavior between two runs. See exactly which tool calls, args, costs and outputs changed when you swap models or edit p

AI 模拟面试官：每条扣分都强制引用你回答的逐字原文（机器校验，编造引用会被重试/丢弃），四维度加权由确定性代码计算而非模型现编，评分体系设计为可经真人盲评校准 · FastAPI + Vue3 + LangGraph