Every number, with its receipt.
This page holds every quantitative claim we make, what produced it, and where it does not apply, including the systems that score above us. Nothing here is a projection.
The claims table
Everything the site says, in one table, each row traceable to a run anyone can repeat: the public LongMemEval dataset, the open v4.4n engine, and the benchmark authors' own scorer.
| Claim | Source |
|---|---|
| 3.0× cheaper per answer than the system ranked above us: about 10k of context per question against Mastra OM's ~30k, $0.025 against $0.075 at GPT-4o input pricing | official 500-question run, against Mastra's own published per-question context. Details |
| 83.00% against 60.60% full-context on the official 500-question chat-memory benchmark (LongMemEval) | official evaluate_qa.py, GPT-4o actor. Details |
| Portable across models, not independent of them: 14.0 points from the weakest actor (Claude Haiku 4.5, 78.00%) to the strongest (Claude Fable 5, 92.00%, 460/500) | identical stores, identical prompts, one shared set of rendered prompts, cross-actor run |
| Retrieval makes no model call, so recall runs on your Claude subscription | architecture: retrieval is code, your Claude answers. Details |
| Saving costs about $0.0009 per 1,000 tokens with the GPT-4.1-mini observer, or $0 on a Claude subscription, where the assistant writes each memory itself | measured, GPT-4.1-mini observer. Details |
Nothing is ever lost: originals stay byte-exact in _raw/ | verify-loop design. |
LongMemEval, the official run
LongMemEval is the standard academic benchmark for chat-history memory: 500 questions over long multi-session histories. Scoring used the benchmark authors' own evaluate_qa.py with its GPT-4o judge. That is the documented protocol for this benchmark, which publishes no leaderboard portal, so self-evaluation with the official script is the protocol. A reproduction needs three public things: the dataset is a published academic benchmark, the v4.4n engine goes up under Apache 2.0 at github.com/Kerneta/daidocs on launch day, and the scorer is the benchmark authors' script rather than ours.
LongMemEval_S accuracy, official scoring, GPT-4o actor
500 questions, higher is better. The dotted line is the oracle ceiling: what this same actor scores when it is handed the correct evidence.
Reading a folder of files beats pasting the whole history by 22.4 points, on the same model, from 10,065 tokens a question instead of 103,601. That closes 85% of the distance to the ceiling.
Full-context baseline (60.60) as published by the benchmark authors, feeding the whole history into the window per question. The .dai store reads a mean of 10,065 tokens per question instead, counted with gpt-tokenizer@3.4.0 on both sides of the ratio. That is the same figure quoted everywhere else.
Run configuration
| Dataset | LongMemEval_S, 500 q (abstentions included) |
| Actor (answers) | openai:gpt-4o, temp 0 |
| Observer (ingest) | openai:gpt-4.1-mini |
| Judge | official evaluate_qa.py · gpt-4o-2024-08-06 |
| Embedder (retrieval) | openai:text-embedding-3-small |
| Tokenizer | gpt-tokenizer@3.4.0 |
| Stores | 19,195 sessions, built fresh |
| Retrieval hit rate | 94.6% |
| Best config | v4.4n · 415/500 · 83.00% |
Reproducibility: GPT-4o's API is not fully deterministic at temperature 0, so expect some movement on a from-scratch re-run. At n=500 the binomial standard error on this run is 1.68 points. How to reproduce
On re-running this yourself. At n=500 the binomial standard error is 1.68 points, so treat any difference smaller than a couple of points, against any other system, as inside the noise. The run-to-run spreads we have measured directly are ±3 points at n=100 and ±7 points at n=20, and we will not quote a tighter full-500 band than the data supports. A reproduction runs the open engine on your own machine against the public dataset: ingest, 500 answers on the actor you choose, then the benchmark authors' own script over your own hypothesis file. How to reproduce
87% is the model's limit, not the memory's
The benchmark's authors measured it first. Table 4(b) of LongMemEval (Wu et al., ICLR 2025) puts plain GPT-4o at 0.870 when it is handed only the correct evidence, against 0.606 reading the whole history. We confirm it: our own oracle control, the same evidence through our reader, scored 87.00%, 435 of 500. Two independent measurements, one ceiling. This is a property of the answering model rather than of our retrieval, which is why it does not move when the engine does.
So perfect retrieval scores 87 on this actor, and our 83.00% closes 85% of the distance from the baseline to it. A much higher number on GPT-4o is not better retrieval, it is a different setup: the same table shows Chain-of-Note prompting lifting the oracle to 0.924. A stronger actor does it too, and we publish ours, 92.00% with Claude Fable 5, above this ceiling because the ceiling is GPT-4o's rather than the benchmark's.
Compare memory systems on one fixed answering model, or you are ranking models rather than memory.
Per-question-type (v4.4n official run)
| Type | Score | n |
|---|---|---|
| Single-session (user) | 92.86% | 65/70 |
| Single-session (assistant) | 92.86% | 52/56 |
| Temporal reasoning | 84.96% | 113/133 |
| Knowledge update | 84.62% | 66/78 |
| Preference | 76.67% | 23/30 |
| Multi-session | 72.18% | 96/133 |
Temporal reasoning is strong because date arithmetic is computed in code and handed to the model to read. Multi-session joins at 72.18% and preference at 76.67% are the weak types, shown here rather than hidden, and both fail on synthesis rather than on search: retrieval surfaced the gold session on 94.6% of questions, so most of what is left is reading error. Preference is n=30, where one question moves the row 3.33 points, so do not read small movements in it as signal. Task-averaged across the six types the run is 84.02%, against the 83.00% micro figure we headline. Per-type numbers come from the v4.4n run’s official eval-results file. Retrieval recall@k against Supermemory lives on the comparison page.
Per reading strategy (v4.4n official run)
| Strategy | Score | n |
|---|---|---|
| lookup single fact | 88.03% | 125/142 |
| advice recommend | 83.02% | 44/53 |
| tally count | 82.55% | 194/235 |
| timeline order | 74.29% | 52/70 |
The engine routes each question to one of these four strategies with a regex classifier over the question text alone: no second model, no caller override, no access to the dataset. The split above (142 / 53 / 235 / 70) is exactly what the shipped classifier produces on these 500 questions, which is the evidence that the run routed with the shipped regex and nothing else. Whether four strategies chosen while looking at this benchmark generalise beyond it is argued rather than demonstrated, and stays that way until a second corpus is published.
The systems that survive the same protocol
Almost every memory score in circulation changes something: the answering model, the judge, the dataset variant, the denominator, or micro versus macro averaging. Filter for one setup (LongMemEval-S, all 500 questions including abstentions, GPT-4o actor at temperature 0, micro-averaged) and a short list is left. The ranking below lists the systems on that list whose configuration is reproducible, because a number nobody outside the vendor can re-run is not a number you can rank. On that ranking we are second among memory systems whose configuration is reproducible, behind Mastra OM at 84.80%, by 1.80 points. Systems that match the setup but not the reproducibility test (EmergenceMem's three configurations, including the 86.00% that sits above ours) are held out and reported in full below, numbers included.
Strict same-setup ranking
LongMemEval-S · gpt-4o actor · 500/500 · micro · reproducible configurations only
Every figure in that ranking carries a caveat, and the caveat travels with the number. The entry criterion is that somebody who does not work for the vendor could re-run the configuration, and it is applied whatever the number is.
| Row | The caveat that travels with it |
|---|---|
| Mastra OM 84.80 | Their own published run, 424/500. They name gpt-4o as judge but not the snapshot, and their memory is built by gemini-2.5-flash where ours is gpt-4.1-mini: the answering model is held fixed between us, the memory-building model is not. |
| Supermemory 81.60 | Measured on a cleaned-dataset variant with their own harness. Their published per-category counts actually sum to 81.8 rather than the 81.60 they headline. |
| Mastra RAG topK 20 81.20 | 406/500, and it does not name its judge. |
| TiMem 76.88 Feather 69.30 | The projects' own figures, not audited by us. |
| Zep · Graphiti 71.20 | 356/500, from Zep's own harness with baselines they re-implemented themselves. A separate 72.27% circulates alongside it, unlabelled as to aggregation. |
| The bottom two rows | The benchmark authors' own baselines (arXiv:2410.10813), not memory systems. |
| EmergenceMem Internal 86.00 | 430/500 on the same dataset, actor and denominator, and it sits above ours. Held out because its configuration is closed, its own authors calling it not publicly reproducible, and its judge is never named. Reported below rather than dropped: a rule that only excluded systems below us would not be a rule. |
Micro against macro, and why we use the number that flatters them. Mastra's own page headlines 84.23%, which is the unweighted average across the six categories. Their per-category counts sum to 424/500 = 84.80% micro, and micro is what our 415/500 is, so 84.80 is the like-for-like comparison and it is the one we use. Averaged the same unweighted way our own run is 84.02% against their 84.23%, a gap of 0.21 points, well inside a single run's noise. Both aggregations of both runs are on this page and the order does not change under either.
The oracle is not in the ranking. GPT-4o handed the correct sessions, with no retrieval to do, scores 87.00% in the authors' own paper. That is a reading ceiling rather than a memory result. We are 4.0 points below it and we do not claim to beat it.
The 9 questions, against Mastra OM
We would like to tell you which categories the 9 questions sit in. We cannot: Mastra publishes a per-category breakdown, but we have not verified it against a run artifact of our own, and we will not reproduce a competitor's split as if we had. What is verified on both sides is the total. Our own per-category split is above, weak types included.
| System | n | Correct | Score |
|---|---|---|---|
| Mastra OM | 500 | 424 | 84.80% |
| .dai v4.4n | 500 | 415 | 83.00% |
| Difference | n/a | −9 | −1.80 |
Is 1.80 points a real difference? On a two-proportion test over independent 500-question runs the standard error of the difference is about 2.32 points, z ≈ 0.78, p ≈ 0.44. We are behind on the point estimate and say so. But the gap sits inside single-run variance, and neither side publishes multi-seed numbers, so it is not settled either way. Mastra Observational Memory is open source under Apache 2.0 and keeps its memory as a human-readable markdown log, so this is a peer, not a walled garden. Where we differ is cost rather than accuracy: their run averaged about 30k tokens of context per question against our 10,065, which is about 3.0× cheaper per answer at GPT-4o input pricing, $0.025 against $0.075. Their per-question context figure is their own published one, not something we measured.
The other numbers, and what they actually measure
Higher figures exist and you will see them quoted, and every score held out of the ranking above lands here with its figure intact, including the held-out score that beats ours. Here is what each one is actually measuring, in the words of our own audit rather than a competitor's.
| Claim | Figure | What it actually measures |
|---|---|---|
| Mastra OM, stronger actors | 94.87% | gpt-5-mini rather than gpt-4o; they also publish 93.27% on gemini-3-pro-preview. We have not run our own store on either of those actors, so we have no figure to place beside them. Different actor, different table. Not a number we get to leave out. |
| .dai v4.4n1, stronger actor | 92.00% | our own store read by Claude Fable 5 rather than GPT-4o, 460 of 500, same files and same retrieval as the 83.00% we headline. It belongs in this table and not in the ranking, for exactly the reason every other row is here: different actor. We hold ourselves to the rule we apply to everyone else, so our best number sits with the incomparables rather than at the top of a league table. |
| Hindsight | 91.40% | a gemini-3-pro actor and a GPT-OSS-120B judge, not the official one. We know of no gpt-4o-actor figure from them. |
| Zep, the unlabelled figures | 90.2% | one of two further figures that circulate for Zep on LongMemEval, the other being 72.27%. Neither arrives with a named actor and judge, so neither is placeable beside a gpt-4o row. Their 71.20 gpt-4o row, which does arrive labelled, is in the ranking above. |
| EmergenceMem Internal / Simple / Simple Fast | 86.00% / 82.40% / 79.00% | Correct variant, denominator and the same GPT-4o answering model as our own row, and none of the three names its judge. That last point is a caveat rather than a disqualifier: the ranking admits Mastra RAG topK 20 on exactly the same basis, and a rule we applied in one direction only would not be a rule. What holds Internal (430/500) and Simple (412/500) out is reproducibility: the Internal configuration is closed, its own authors calling it not publicly reproducible. Simple Fast is open-sourced, so it meets the entry criterion and now sits in the ranking above at 79.00%, below us. It belongs there whether or not that flatters us. |
| Graphify | 76% | LongMemEval-S at n=50, the English subset rather than all 500, answered and judged by Kimi K2.6 and graded on key-fact coverage rather than the official script. Their own harness scores plain dense RAG at the same 76%, which is how they report it: tied, not ahead. Different actor, different denominator, different grader, so it is not placeable beside a gpt-4o 500/500 row in either direction. It is also built for a different problem, which is worth more than the number: see below. |
Any table that ranks 94.87% (gpt-5-mini), 91.40% (gemini-3-pro) and 83.00% (gpt-4o) in a single column is ranking answering models, not memory systems. Every figure above is taken from the publisher's own published material, and we ship no audit document of our own, so each caveat here has to stand on the source you can go and read yourself.
Memory of the code, and memory of the work
Graphify is in the table above because it publishes a LongMemEval number, but ranking it against us rather misses what it does. Tools of that kind read the repository and build a map of the code as it stands right now: which function calls which, where a symbol is defined, what a rename would touch. .dai reads the conversations and keeps the record of the work: what was decided, what was tried and abandoned, why a number is the number it is, and what is still owed. One is the state of the code. The other is the history behind it, and the code does not carry that.
So they are complementary, and running both is the configuration we would actually recommend. A code index tells the assistant what the codebase is. The store tells it what happened to the codebase and what you already ruled out. Neither stands in for the other: a repository map cannot tell you why the retry limit is three, and no amount of conversation history will list every caller of the function you are about to rename. If you already run a code-graph or repository-index tool, an AST graph such as Graphify, your editor's own codebase index, or a language-server-backed search, keep it. .dai sits beside it rather than in its place, and neither one has to know about the other.
LongMemEval and LoCoMo measure two different problems
Some of the highest memory figures in circulation are LoCoMo scores rather than LongMemEval ones, so it is worth saying plainly what the difference is, because it is the difference this whole format is built around.
| Benchmark | What it tests | Roughly |
|---|---|---|
| LongMemEval what we are measured on | Long memory. Histories that have outgrown the window: many sessions, months apart, where the answer sits in one conversation out of dozens and nothing can hold all of it at once. | 103,601 tokens of history per question, on average median 103,706, min 97,121, max 105,842 |
| LoCoMo | Short memory. Single long-ish conversations that still fit comfortably inside a modern context window, so the model can simply be shown the whole thing. | short enough to fit inside a modern window not a long-history benchmark, so it does not test what .dai is for |
Why that matters here. Below roughly 20,000 tokens there is not much for a memory system to do. The history fits, so pasting it in is simpler, and usually more accurate, than converting it. .dai is designed for what happens above that line: where the history no longer fits, where you start paying to re-send the same context on every question, and where you eventually cannot buy your way past the wall at any price. That is where the cost curve bends and where the format earns its place.
This is where Mem0's 94.4% belongs. It is a LoCoMo score, published without a named answering model, and Mem0's peer-reviewed paper reports no LongMemEval result at all. So it is not a number that can be set against ours in either direction: it is a score on the short-memory problem in the row above, and we do not publish one of those. Quoting it beside a LongMemEval figure, whoever does it and whichever way it falls, compares two different tasks.
So: we are not currently designed for the short-window regime, and we do not publish a LoCoMo number for the current engine because we have not run one. A small-window profile is an open piece of work, listed with the rest of the contributor roadmap in the public repository at github.com/Kerneta/daidocs, along with measuring exactly where the crossover sits. Until that is measured, if your whole history fits in the window, use the window.
What memory costs you
Recall makes no model call
Retrieval is ordinary code over the index and the files, with one external dependency: a disk-cached text-embedding-3-small call that ranks which stored files the reading model sees, on the order of $0.00002 per 1,000 tokens. No model call reads your memory for you. The recalled context (a mean of 10,065 tokens per question on the benchmark run) is read by your Claude on the subscription or API plan you already have. This is architecture, not a promotion.
Save: about $0.0009 per 1,000 tokens
Measured with GPT-4.1-mini doing the single extraction call, the observer of record for every published number. It runs once per document and never again, so the cost is paid on the way in rather than on every question. On a Claude subscription it runs free, because the assistant in the session writes the extraction itself.
The alternative, measured
On the same benchmark, Mastra OM's run averaged about 30k tokens of context per question against our 10,065. That is about 3.0× more per answer, $0.075 against $0.025 at GPT-4o input pricing. Beyond the window, the full-text default simply cannot run at all, while recall still can.
Which observer to convert with
The observer writes the derived layer of a .dai file: the summary, the Understanding JSON, and the facts, events and profile indexes. It never touches the verbatim content, which every observer preserves in full. That derived layer is what retrieval ranks on, so a poor conversion is permanent in that file and shows up later as recall failing to surface the right session.
| Observer | Extraction recall | Cost per 1,000 tokens | Verdict |
|---|---|---|---|
| GPT-4.1-mini observer of record | 0.515 | $0.0009 | Use this one. Every published number was produced with it |
| Claude Opus 5 | 0.512 | $0.0129 | Extracts to a comparable standard, for about fourteen times the price and no accuracy we can demonstrate |
| Claude Sonnet 5 | 0.486 | $0.0058 | Works, and measures below both of the above |
| Claude Haiku 4.5 | 0.440 | $0.0026 | Extraction comes back malformed or thin. Cheap, and not equivalent |
Extraction recall is measured by swapping only the observer and scoring the derived layer it writes, over 251 questions. Read it for what it is: a lexical measure, no judge, and extraction recall is a necessary condition for a right answer rather than a sufficient one. No observer other than GPT-4.1-mini has been run through a full haystack benchmark, so none of them has been shown to reach 83.00%. Spend the quality on the answering model instead, where it is measured: Opus 5 reads the same store at 91.00% and Fable 5 at 92.00%, against the 83.00% we publish on GPT-4o.
Where these results apply, and where they don't
Where .dai wins
- Chat history. The LongMemEval domain: many sessions, long time spans, updates and contradictions.
- Many files. Corpora that no single context window holds coherently.
- Beyond the window. Histories that literally cannot fit in a context window. Recall reads a small, bounded slice (a mean of 10,065 tokens per question on the benchmark run, against a 103,601-token history) rather than the whole store.
- Agent sessions. What a coding agent learns and decides across a long project: Claude Code, Cursor and Windsurf read and write the same store over MCP, and the session hook archives each one as it ends. Structurally this is the benchmark's domain, many sessions accumulating over months, but we have not benchmarked agent sessions themselves, so treat it as the same mechanism rather than as a measured number.
When .dai routes you away
A single document fits in the context window. There, the full-text default is cheaper and at least as accurate, and the built-in decision layer chooses the full-text default instead of retrieving. We consider that a core feature, because a memory system that cannot name its own boundary will happily bill you inside it.
Raw agent traces, too. Tool calls, stack traces and file dumps have a different shape from a conversation, and the extraction is built for what an agent learns and discusses rather than for its execution log. Point it at the reasoning, not at the transcript of every command.
Also worth knowing: on a small own-data run of about twenty questions, an accuracy difference of a few answers is noise (±7 points at n=20 measured, against ±3 points at n=100), so full-500 benchmark runs are the citable numbers.
Three caveats that travel with 83.00%
| Caveat | What it means for the number |
|---|---|
| It is a development-set score | The engine was developed against LongMemEval-S, and several variants were evaluated against these same 500 questions before this one shipped. Diagnosing failures on a development set and fixing what you find is ordinary practice: it stays ordinary only if you say so. No held-out evaluation has been run on this build, so nothing currently bounds the selection bias, and its size is unknown. |
| One seed | No arm was run twice. The binomial standard error alone is 1.68 points on the release run, and between 1.21 and 1.85 across the actor table, so differences of a couple of points against any other figure are inside the noise. |
| The judge shares the release actor's model family | The official scorer uses gpt-4o-2024-08-06 to judge, and our release actor is GPT-4o. Family self-preference is a known effect in LLM judging and is inherited by any absolute number scored this way. In the actor table it cuts against the Claude rows rather than for them. It is not quantified. |
Four values in the engine were calibrated by measuring on this benchmark and cannot be shown to be general: the answer budgets (220 / 350 / 400 tokens), the read depth (top 5 files per question, top 8 above 200k source tokens), the extraction fact cap of 14, and the widened advice-detection phrasings. The rules that survived because they state a general principle, deduplicating restatements of one event, counting distinct real-world items rather than rows, calendar-inclusive windows, are a separate list and hold for any personal history.
One store, five answering models
Identical .dai stores (the same 19,195 ingested sessions, never re-ingested), one shared set of 500 rendered prompts, identical v4.4n retrieval. The only thing that changes between these rows is the model doing the answering. Every one is the full 500 questions scored by the official evaluate_qa.py with judge snapshot gpt-4o-2024-08-06. Measured 15 August 2026.
.dai accuracy with different answering models
| Answering model | Accuracy | Correct | ± 1 s.e. | Macro | Note |
|---|---|---|---|---|---|
| Claude Fable 5 | 92.00% | 460/500 | 1.21 | 91.94% | strongest actor |
| Claude Opus 5 | 91.00% | 455/500 | 1.28 | 90.94% | tied with Fable, not ranked below it: 5 questions is inside one s.e. |
| Claude Sonnet 5 | 85.60% | 428/500 | 1.57 | 86.75% | |
| GPT-4o the figure we publish | 83.00% | 415/500 | 1.68 | 84.02% | the benchmark's standard actor, held fixed for every comparison |
| Claude Haiku 4.5 | 78.00% | 390/500 | 1.85 | 77.66% | weakest actor, and the only one below the release run |
| GPT-4o, no memory system | 60.60% | 303/500 | 2.19 | not published | the benchmark authors' own full-context baseline: the same model with the whole history pasted in, their harness not ours. Same actor as our 83.00% row, 22.40 points apart |
This is the table that disproves the tidiest version of our own story. The format is model-portable: nothing failed to read it. Accuracy is not model-independent: the spread is 14.0 points on identical inputs, which is wider than the gap between most memory systems on the leaderboard. The spread is concentrated in one row: multi-session runs from 88.72% down to 69.92% across these actors, against much narrower spreads on every other question type. Since retrieval was identical, that is a difference in synthesising an answer out of many retrieved sessions, not a difference in finding them. Any memory-system score quoted without naming the answering model, ours included, is close to meaningless, which is why we hold the actor fixed at GPT-4o everywhere else.
The evidence that retrieval really was identical: the per-question diagnostics file (reading strategy, context tokens, files read, retrieval hit) is byte-identical across all five runs, sha256 95bcf659c42ff063…. Every actor was handed the same context, selected the same way, at the same token cost, at the same 94.6% retrieval hit rate.
How the Claude rows were run, and why they are not identical to the GPT-4o row. The release run called openai:gpt-4o over the API at temperature 0. The four Claude actors were not run over an API: they were run through a Claude subscription, with each rendered prompt written to a queue file, answered, and picked back off disk. Two differences travel with those four numbers. No sampling parameters were set, so there is no temperature 0 equivalent on that route, and the prompt arrived as a file read rather than as a user message: byte-identical content, different framing in the context window. The judge was unchanged and the same for every actor, which is what keeps the five numbers comparable to each other. Before any of them was accepted, the scoring tool was re-run against the release run's verdict file alone and reproduced 83.00%, 415/500, macro 84.02%, s.e. 1.68.
It gets better over time
Key versions of the engine, every one scored the same way: the official evaluate_qa.py, the same GPT-4o actor, the same observer, the same 500 questions. These are our internal development milestones rather than public releases, shown so the trajectory is visible instead of asserted. The gain from the first rung to the current engine is 9.0 points.
| Method version | Correct | Score |
|---|---|---|
| v4.4n (current) | 415/500 | 83.00% |
| v4.4d | 413/500 | 82.60% |
| v4.2 | 397/500 | 79.40% |
| v3.0 | 393/500 | 78.60% |
| v2.0-sem2 | 381/500 | 76.20% |
| v1.9.6 | 370/500 | 74.00% |
| Full-context GPT-4o no memory system | 303/500 | 60.60% |
Why this matters beyond bragging: because _raw/ keeps every original byte-exact, a new engine version re-reads the history you already have. The memories you saved last year get better when the reader improves, without you re-recording anything.
Sources and replication artifacts
Reproducing the numbers above takes three things, and none of them are ours to withhold:
| What a reproduction needs | Where it comes from |
|---|---|
| The dataset | LongMemEval-S, a published academic benchmark: 500 questions, abstentions included. Released by its authors (arXiv:2410.10813), not by us. |
| The engine | v4.4n (lib/methods/daidocs-v44n, ingest signature daidocs-4.4.1) under Apache 2.0 at github.com/Kerneta/daidocs, alongside the spec and the daidocs-mcp server. It is the engine that scored 415/500, not a lagging edition of it. |
| The scorer | the benchmark authors' own evaluate_qa.py, judge snapshot gpt-4o-2024-08-06. Their script and their judge, not ours. |
docs/REPLICATION.md | the full replication protocol, in the same repository |
The internal harness we drive our own runs with is not published. It saved us typing and nothing else: the dataset, the engine and the scorer are what a reproduction actually rests on, and a result you can only check by running our tooling would not be worth much anyway.