Paper Assimilation Task · ICAIF '26 Milan
Can a model turn a finance paper back into the signal it describes?
ZooBench takes a published asset-pricing anomaly paper and asks a model to rebuild it: classify the predictor, state the factor logic, write the signal code, and sort the portfolio. Every stage is scored twice — once where the model builds on its own earlier answers, and once where it is handed the ground truth. The distance between those two numbers is what the benchmark is for.
Next milestone
Corpus
98 predictors from the Open Source Asset Pricing replication set, each with a reference implementation and a published long–short return series. Ground truth is built in a separate repository; this harness only consumes it.
Where the error actually enters
Mean score per stage, every run pooled. Paired bars: compositional (the model's own chain) against oracle (ground-truth inputs).
The finding that keeps holding
Logic and Code scores are not comparable across harness versions — the judge rubric changed at v1.2. Theme is an exact-match rate against a provisional 33-label taxonomy, so read it as agreement with that taxonomy rather than as accuracy.
Leaderboard
One row per model, pooled across the runs you select. Click a row for telemetry.
Compositional (E): each stage builds on the model's own previous output · Oracle (O): each stage is handed ground-truth upstream input · Gap = O − E. Theme has no upstream stage, so it appears once. Scores are means over (task, model) pairs, so a model measured on 1 paper sits beside one measured on 98 — the n column is the honest comparison guard. Backtest needs licensed data and only calibrated tasks are backtested, so its columns score fewer pairs than n; open a row to see how many.
Runs
Every batch, newest first. Click one to see its manifest, telemetry and per-paper rows.
Harness timeline
Scores are only comparable within a version. Each entry is a fingerprint over the harness sources, so a change to scoring code shows up here whether or not it was committed.
What the task is
Each task is one published paper that documents a cross-sectional return predictor. The model receives the full paper text and has to rebuild the predictor in four steps. A separate reference implementation — written from the paper by hand, with its own published return series — is the ground truth.
The four stages
- Theme — classify the predictor into the corpus taxonomy (33 working labels). Scored by exact match, so it is agreement with a provisional taxonomy and not accuracy in any deeper sense.
- Factor Logic — state, in prose, the quantity to compute and how. Judged 1–5 against the reference logic.
- Code — write a signal script against a fixed environment specification. Judged 1–5 against the verbatim reference code.
- Backtest — execute the generated code in a sandbox, sort the long–short portfolio, and compare against the published returns. Needs licensed WRDS-derived data, so only some runs include it.
The two regimes
Every stage runs twice. In the compositional regime the model consumes its own earlier output, so an error in Theme is still present by the time it writes Code. In the oracle regime each stage receives the ground-truth input instead, which isolates that stage's own competence.
The difference, oracle minus compositional, is the error-propagation cost. A stage with a large gap is being held back by what reached it; a stage with no gap is failing on its own terms, and giving it perfect inputs would not help.
Judging
Factor Logic and Code are scored by a model judge pinned for the whole run and held outside the roster under test, so the construction-model family does not also sit in the judge. The judge is recorded in every run's manifest and shown on the leaderboard.
Carried caveats
- The theme taxonomy is provisional, not a decided closed set.
- Judge prompts are not yet validated; the Logic rubric changed at harness v1.2, so Logic and Code scores do not compare across that boundary.
- Part of the ground truth is Mode B — in-session isolation rather than the trusted sandboxed promote gate.
- Family confound: the ground truth was built with Claude models, so Claude results may benefit.
- API and configuration errors are scored at the floor rather than halting the job, which is why the failure column matters on every run.
Cost accounting
Dollar figures are computed from the token counts each run recorded, priced with the provider catalog in the harness repo. Reasoning models bill hidden reasoning as output tokens — on some runs that is over 90% of the output — so cost per paper varies several-fold for the same model and cannot be estimated from paper count.