Havuz

甲作 Atelier：手机端 AI 美甲推荐与虚拟试戴；仓库公开其中的轻量评测工作台。

面向小团队的一站式 NLP 小模型平台:数据集版本管理 · 训练 · 评估 · 在线部署 · Badcase 闭环,内置大模型 Prompt 盲测评测(人工 + AI 双指标)。

Safety benchmark for personal AI agents under realistic prompt injection. 120 test cases across 5 professional domains, 3 injection vectors,

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring (EMNLP 2026 Findings) — multi-language benchmark for linter-guided refactoring

Same-brief agentic build comparison: GPT-6 Astra High vs Gemini 3.8 Flash High building a playable 3D AI civilization simulation game.

Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode,

Adversarial LLM Reversal for hermes-agent

My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementati

A runnable multi-agent process with a built-in evaluation system — repeated rollouts scored on the git diff, not the agent's self-report. Op

The same AI agent pipeline built in Mastra and LangChain. Runs in parallel, measures everything.

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

Benchmark of AI coding agents building web apps and deploying them to Cloudish, DigitalOcean, Render and Vercel, measuring success, time, to

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

자기개선 후보를 채택할지 파레토 전선으로 판정하는 게이트 + 그 판정을 지탱하는 계측 규율. 사전등록·계기검침·부재증명 실측 포함.

⚽🤖 11 frontier LLMs predicted the entire 2026 World Cup — frozen before kickoff. Live leaderboard: Brier score, bracket points & Polymarket

Production-grade LLM Evaluation & Benchmarking Framework - GPT-4, Claude, Gemini, Mistral. Accuracy, latency, cost, hallucination, reasonin

Typed decisions (choice / score / yes-no) from local Qwen models on Apple Silicon. Probabilities come straight from the logits, no text gene

Smriti (स्मृति): private, local-first agentic RAG with System 1 + System 2 thinking over your notes, books and documents. Guardrails, tracin

A benchmark of AI QA agents finding real bugs in 4 open-source web apps, with 1,396 labeled bugs.

Typed evaluation for OpenCode skills and AI agents: capability, regression and product checks; provider-neutral evidence grading, provenance

Agent skills for designing, training, evaluating and improving application-specific decision systems. Primitive/model selection, data assemb

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluatio

GEDD turns domain-expert judgment into one failure codebook, then derives your evaluation, your training reward and your response gate from

Open-source benchmark for evaluating LLMs on domain-specific geological reasoning, AI for geology, and source-grounded geological knowledge.

A lightweight LLM evaluation layer that turns subjective model outputs into deterministic decisions by scoring attribution, specificity, rel

What can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.

Multi-model consensus debate via the filesystem. LLMs propose, peer-review, rebut, vote and synthesize a group-confirmed answer. CLI + MCP.

Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lan

Don't trust an Agent. Test it. Open-source lab for agent credit — exams → evidence → explainable, recomputable scores. Built for the day age

Distil an expensive LLM API call on a narrow task into a small local model: capture traffic, curate, LoRA fine-tune with MLX on Apple Silico

Evolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible epochs to converge on optimizati

Don't make LLMs honest. Make every factual claim auditable. — An LLM Claim Auditing Layer with T1-T7 truth gradients. 98.1% business effecti

Turn a production incident into a structured 9-section LLM response (severity, root cause, mitigation, postmortem). Ships with a 5-scenario

PaperBench High-Difficulty and Expanded Tasks: paper reproduction beyond machine learning. 93 tasks in 12 research areas, run in domain-nati

Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic sco

Bounded, evidence-driven SKILL.md evolution under frozen evaluation and safety contracts.

Paper artifacts, figures, aggregate data, and share cards for the DXRG paper: What LLM Trading Agents Actually Do in Production (six months,

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

EvoTrace, Claude Code ve Codex oturumlarını inceleyip eğitim, değerlendirme ve doğrulama için yeniden kullanılabilir varlıklara dönüştürür.

Neden yükseliyor · 39 günde 292 yıldız, coding agent oturumlarından eğitim ve değerlendirme varlığı üretme fikrine ilgi olduğunu gösteriyor; ancak bu ilginin nedeni ölçümlerden anlaşılamıyor. Son 28 günde commit ve farklı yazar olmaması, ilginin devam eden geliştirme veya benimsemeyle desteklendiğine dair kanıt sunmuyor.

EvalForge, AI evaluation sonuçlarını baseline karşılaştırmaları ve CI quality gate’leriyle değerlendirip standart rapor formatlarında yayımlar.

Neden yükseliyor · RAG evaluation sonuçlarını CI’da denetlenebilir kanıta ve release gate’lerine bağlama ihtiyacına odaklanıyor; repo 34 günde 211 yıldız almış. Ancak README bağımsız doğrulanmış downstream benimsemenin henüz bulunmadığını belirtiyor ve son 28 günde katkı tek yazardan gelmiş; ilginin gerçek kullanımın kanıtı olduğu söylenemez.

The Regression Testing Framework for AI Agents. Replay · Evaluate · Assert · Catch Regressions — in CI. Like Jest for your AI layer.

Self-hosted decision model (selfjev-4b) with a Jev-compatible API: Python SDK, server, fine-tuning and RLCD.

Evidence-first infrastructure for reproducible, isolated executable evaluation and online rewards.

GameOps Investigator, oyun analitiğindeki düşüş ve anomali vakalarını read-only SQL, istatistiksel karşılaştırmalar ve incelemeye açık raporlarla araştıran bir araçtır.

Neden yükseliyor · 76 yıldız ve 28 günde 23 commit ilgi gördüğünü gösteriyor; oyun analitiğini agent ve MCP yaklaşımıyla birleştirmesi dikkat çekmiş olabilir. Yükselişin kesin nedenini gösteren veri yok; commit’lerin tek yazardan gelmesi topluluk katkısının henüz sınırlı olduğuna işaret ediyor.

Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate,

AI equity research agent with resilient workflows, evidence-grounded RAG, versioned reports, and automated quality evaluation.