ICAIF '26 Workshop · Bocconi University, Milan
ZooBench: Localizing Failure in AI Replication of the Factor Zoo
University of Chicago
Overview
ZooBench takes a published asset-pricing anomaly paper and asks a model to rebuild it: classify the predictor, state the factor logic, write the signal code, and sort the portfolio. Every stage is scored twice — once where the model builds on its own earlier answers, and once where it is handed the ground truth. The distance between those two numbers is what the benchmark is for.
Where the error actually enters
Each stage is scored twice. First the model works on its own, building on its earlier answers, so any mistake it made upstream is still there. Then it gets the correct input for that stage instead. If the score jumps, the problem came from an earlier stage; if it doesn't, the stage itself is the hard part.
The finding that keeps holding
Logic and Code scores are not comparable across harness versions — the judge rubric changed at v1.2. Theme is an exact-match rate against a provisional 33-label taxonomy, so read it as agreement with that taxonomy rather than as accuracy.
Leaderboard
One row per model. Click a row for per-paper scores and telemetry.
Table 1. Compositional (E): each stage builds on the model's own previous output. Stage-isolated (O): each stage is handed ground-truth upstream input. Gap = O − E. Theme has no upstream stage, so it appears once. Full runs are the single-model batches over all 98 papers, grouped as in the paper; all runs pools every batch per model. Scores are means over (task, model) pairs, so a model measured on 1 paper sits beside one measured on 98 — the n column is the honest comparison guard. Backtest needs licensed data and only calibrated tasks are backtested, so its columns score fewer pairs than n; open a row to see how many.
Runs
Every batch, newest first. Click one to see its manifest, telemetry and per-model scores.
Harness timeline
Scores are only comparable within a version. Each entry is a fingerprint over the harness sources, so a change to scoring code shows up here whether or not it was committed.
Methodology
How a task is built, run and scored. Section order follows the paper.
1The Paper Assimilation Task
Each task is one published paper that documents a cross-sectional return predictor. The model receives the full paper text and has to rebuild the predictor in four steps. A separate reference implementation — written from the paper by hand, with its own published return series — is the ground truth.
Every task carries four fixed, version-controlled ground truths:
- Theme — the economic theme of the anomaly under the corpus taxonomy.
- Factor Logic — a natural-language business specification sufficient to regenerate the reference implementation.
- Reference implementation — the verbatim OSAP signal script for the predictor.
- Reference backtest — the published OSAP long–short return series, with OSAP's portfolio-construction conventions for it.
2Corpus
98 predictors from the Open Source Asset Pricing replication set, each with a reference implementation and a published long–short return series. Ground truth is built in a separate repository; this harness only consumes it.
| Papers cataloged | Count |
|---|---|
| Predictor (Paper Assimilation Tasks) | 98 |
| Placebo (auxiliary controls) | 3 |
| Reference (context and provenance only) | 9 |
| Total | 110 |
Counts as reported in the paper (Table 2). 59 of the 98 tasks have an executable reference backtest on the data the harness stages.
3The four stages
- Theme — classify the predictor into the corpus taxonomy (33 working labels). Scored by exact match, so it is agreement with a provisional taxonomy and not accuracy in any deeper sense.
- Factor Logic — state, in prose, the quantity to compute and how. Judged 1–5 against the reference logic.
- Code — write a signal script against a fixed environment specification. Judged 1–5 against the verbatim reference code.
- Backtest — execute the generated code in a sandbox, sort the long–short portfolio, and compare against the published returns. Needs licensed WRDS-derived data, so only some runs include it.
4Two evaluation regimes
Every stage runs twice. In the compositional regime the model consumes its own earlier output, so an error in Theme is still present by the time it writes Code. In the stage-isolated (oracle) regime each stage receives the ground-truth input instead, which isolates that stage's own competence.
The difference, stage-isolated minus compositional, is the error-propagation cost. A stage with a large gap is being held back by what reached it; a stage with no gap is failing on its own terms, and giving it perfect inputs would not help.
5Judging
Factor Logic and Code are scored by a model judge pinned for the whole run and held outside the roster under test, so the construction-model family does not also sit in the judge. The judge is recorded in every run's manifest and shown on the leaderboard.
6Limitations
- The theme taxonomy is provisional, not a decided closed set.
- Judge prompts are not yet validated; the Logic rubric changed at harness v1.2, so Logic and Code scores do not compare across that boundary.
- Part of the ground truth is Mode B — in-session isolation rather than the trusted sandboxed promote gate.
- Family confound: the ground truth was built with Claude models, so Claude results may benefit.
- API and configuration errors are scored at the floor rather than halting the job, which is why the failure column matters on every run.
7Cost accounting
Dollar figures are computed from the token counts each run recorded, priced with the provider catalog in the harness repo. Reasoning models bill hidden reasoning as output tokens — on some runs that is over 90% of the output — so cost per paper varies several-fold for the same model and cannot be estimated from paper count.