Havuz

Live trading arena for LLM agents: public reasoning, real market data, virtual capital

Benchmark and controlled study of LLM-based novelty judges for research ideas. Code and data for "Old Ideas, Novel Problems" (arXiv 2610.020

Self-hosted decision model (selfjev-4b) with a Jev-compatible API: Python SDK, server, fine-tuning and RLCD.

Make AI agents aim higher without losing control: a Claude Code skill with two bars per task, a cheap gate check before expensive work, and

Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lan

DiagSafe: diagnostic LLM safety benchmark with prompt-level rubrics (3,489 bilingual prompts, 84-label taxonomy)

DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with

AI 模拟面试官：每条扣分都强制引用你回答的逐字原文（机器校验，编造引用会被重试/丢弃），四维度加权由确定性代码计算而非模型现编，评分体系设计为可经真人盲评校准 · FastAPI + Vue3 + LangGraph

A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.

Benchmark of AI coding agents building web apps and deploying them to Cloudish, DigitalOcean, Render and Vercel, measuring success, time, to

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs tha

Commented SHIP? Full briefs for 3 pile-two AI engineering projects: LLM Ops, Agent Evals, Auto Routing. What to build, what to measure, what

PaperBench High-Difficulty and Expanded Tasks: paper reproduction beyond machine learning. 93 tasks in 12 research areas, run in domain-nati

Developer and product skills for Claude Code, Cursor and Codex: 1-file AI evals and Chrome extension store assets.

Smriti (स्मृति): private, local-first agentic RAG with System 1 + System 2 thinking over your notes, books and documents. Guardrails, tracin

What can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.

Find the best AI models you can run

Catch AI agents that say "done" when the task isn't. Checks agent-trace/v1 traces with claim-evidence rules, LLM judges and a trained classi

Distil an expensive LLM API call on a narrow task into a small local model: capture traffic, curate, LoRA fine-tune with MLX on Apple Silico

Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifac

HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time

Chinese-first LLM jailbreak benchmark & red-team toolkit | 中文LLM安全评测框架 | by POJIA.AI

RLCDAlignBench: 44 alignment-failure detection benchmarks and code for 'Just Ask Jev' (RLCD zero-shot detector of AI alignment failures)

Behavioral regression testing for AI apps and agents: pytest plugin, behavior diff, CI gate, and document extraction metrics

Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate,

jev-calibrate, Jev sorularının etiketli örneklerde karar vermeye uygunluğunu ölçen bir TypeScript CLI aracıdır.

Neden yükseliyor · Kesin yükseliş nedeni mevcut verilerden anlaşılamıyor. Repo 6 günlük, 32 yıldızlı ve son 28 günde 11 commit almış; ancak farklı yazar sayısının 1 olması aktivitenin tek geliştiricide toplandığını gösteriyor.

Myrtia — agent-native red teaming for LLMs and AI agents: a deliberating council of attacker models (propose, cross-critique, Borda vote, ch

jevals, agent trace'lerini eval ve guardrail kontrollerinden tek istekte geçirerek production veya agent loop içinde değerlendirmeyi amaçlayan bir Python kütüphanesidir.

Neden yükseliyor · 7 günlük bir repoda 97 yıldız, 5 commit ve 4 farklı yazar erken ilgi ve katkı işareti. Tek istekte birden fazla kontrol, düşük maliyet ve agent loop'a uygun hız iddiaları dikkat çekiyor; performans ve maliyet rakamları README'de verilmiş, bağımsız doğrulama olarak kabul edilmemeli.

Jev-powered, rule-based grader for text files. Runs every rule against every line in parallel. No skimming, no missed lines.

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

Independent, evidence-based map of when TypeSafe's Jev actually holds up vs. breaks down — real API-call receipts, not a leaderboard. 中文為主的雙

Typed decisions (choice / score / yes-no) from local Qwen models on Apple Silicon. Probabilities come straight from the logits, no text gene

Bounded, evidence-driven SKILL.md evolution under frozen evaluation and safety contracts.

Red team vs blue team benchmark for scientific paper verification. Plant defects into open access biomedical papers, build gates to catch th

Teach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it te

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence

Same-brief agentic build comparison: GPT-6 Astra High vs Gemini 3.8 Flash High building a playable 3D AI civilization simulation game.

Intents is a native macOS workbench for evaluating Apple's on-device Foundation Models.

Hybrid rule-based and context-aware LLM evaluation system for Chinese-to-Japanese game localization QA.

Paper artifacts, figures, aggregate data, and share cards for the DXRG paper: What LLM Trading Agents Actually Do in Production (six months,

GameOps Investigator, oyun analitiğindeki düşüş ve anomali vakalarını read-only SQL, istatistiksel karşılaştırmalar ve incelemeye açık raporlarla araştıran bir araçtır.

Neden yükseliyor · 76 yıldız ve 28 günde 23 commit ilgi gördüğünü gösteriyor; oyun analitiğini agent ve MCP yaklaşımıyla birleştirmesi dikkat çekmiş olabilir. Yükselişin kesin nedenini gösteren veri yok; commit’lerin tek yazardan gelmesi topluluk katkısının henüz sınırlı olduğuna işaret ediyor.

OrcaReplay, AI agent çalıştırmalarını kaydedip çevrimdışı tekrar oynatmaya ve bir adımdan farklı bir modelle devam ettirmeye yarayan bir debugging aracıdır.

Neden yükseliyor · 29 günlük bir repo için 171 commit ve 5 farklı yazar kayda değer geliştirme etkinliği gösteriyor; 268 yıldız da erken dönem ilgisine işaret ediyor. Yükselişin belirli bir duyuruya veya kullanıcı benimsemesine bağlı olup olmadığı verilen verilerden anlaşılamıyor.

Don't trust an Agent. Test it. Open-source lab for agent credit — exams → evidence → explainable, recomputable scores. Built for the day age

AIRadar · 每日定时运行的 AI 行业情报工作流

甲作 Atelier：手机端 AI 美甲推荐与虚拟试戴；仓库公开其中的轻量评测工作台。

Open-source agent reliability platform for discovering, investigating, and fixing production agent failures

CI for your coding-agent setup (CLAUDE.md, skills, hooks): pinned baselines, a canary on every Claude Code release, model and version matrix

DodoGuard: Take AI Agents to Production. Test, validate, and operate AI Agents before and after deployment.for Dify, Coze, FastGPT, n8n, Ten

Bug Hunt Bench: 105 real bugs in two production repos, frontier coding models (GPT-6, Claude, Grok, Gemini, DeepSeek...) find and fix them i

EvalForge, AI evaluation sonuçlarını baseline karşılaştırmaları ve CI quality gate’leriyle değerlendirip standart rapor formatlarında yayımlar.

Neden yükseliyor · RAG evaluation sonuçlarını CI’da denetlenebilir kanıta ve release gate’lerine bağlama ihtiyacına odaklanıyor; repo 34 günde 211 yıldız almış. Ancak README bağımsız doğrulanmış downstream benimsemenin henüz bulunmadığını belirtiyor ve son 28 günde katkı tek yazardan gelmiş; ilginin gerçek kullanımın kanıtı olduğu söylenemez.