V3.1 data overview

DownloadAudit

v3.1/, from 2026-10-27T00: several files per UTC hour, listed by that hour's manifest, under v3.1/YYYY-MM-DD/HH/. Receive and exchange times are in milliseconds. The v3.1 preview sets out what v3.1 adds to v3; the notice above links the battle test and the guide to switching.

Published under CC BY-NC 4.0: free for non-commercial use with attribution. Commercial use needs a separate licence from us; ask through the contact links on the site.

Military Grade

Good enough to replay the orderbook. Several of our machines record the feed, and every row names the machines that saw it (witness_set). Each hour names one primary recorder, chosen once a day from the previous week's coverage (or that hour's best recorder when the primary did badly); keep is_primary_copy = true to see that one recorder's own view. In a two-outcome market a row for the second token is dropped only when it is an exact mirror of the first token's row, and it can be rebuilt. Where the hour's manifest lists one, its twin_check file gives counts and fingerprints of the rows as we captured them, so you can check your rebuild. Repeat book snapshots are dropped on purpose and cannot be rebuilt. Every stored row carries its receive time to the millisecond, rounded down (timestamp + recv_lag_ms); a rebuilt mirror row takes its own from the provenance file. Within a token, rows are ordered by venue timestamp, then sequence, then row_ordinal. Each hour is verified before it goes up, and the report of that check is published beside its manifest. The replay engine checks every file an hour lists against its sha256 before it replays it.

In short

• Start from the hour's <hour>.manifest.json and download exactly the files it lists. An hour is listed here only once its manifest is published, and the manifest is copied last.
• Every stored row carries its receive time to the millisecond, rounded down: timestamp + recv_lag_ms. A rebuilt mirror row takes its own from the provenance file.
• sequence is NOT unique within an hour; row_ordinal is. Within a token, rows are ordered by venue timestamp, then sequence, then row_ordinal.
• opening_state.parquet is keyed on the venue's time, not on when we received a row. For a start with no lookahead, order and filter by receive time instead.
• No per-hour coverage verdict is published for v3.1. The COVERAGE attestation covers v3 hours only.

How an hour is laid out

An hour is several files, so start from its manifest (for example 2026-10-11T01.manifest.json). Download the manifest first, then exactly the files it lists, plus the token map its canonical_token_map.object names.

The hour's rows are split by category, and busy sub_tags into their own units. Each unit is a lean file and a provenance file, or N such pairs when the unit is in parts. The lean file holds the event columns and each row's receive time, and the provenance file holds each row's sequence and which of our recorders saw it. The two are joined on row_ordinal. Our twin checks and replay engine check that each provenance file holds exactly its lean file's row_ordinal values. A file name reads left to right as <hour>.<category>[.<sub_tag>][.p<N>][.provenance].parquet.

A busy (category, sub_tag) gets its own pair, <hour>.<category>.<sub_tag>..., and the rest of the category stays in <hour>.<category>..., so one category can span several files. Which units are split is decided once a day from the busiest hour of each of the seven days ending two days earlier, and then held for at least 28 days. The manifest's layout_split.split names the units split out in that hour, and layout_split.parts gives the number of parts N of each unit. A unit in one file has no .p<k> in its name; a unit in parts is named .p1 to .p<N>, the first included.

Each row's category and sub_tag come from one category map, named in the manifest by category_map_sha256. Two category values do not come from a venue tag: uncategorised, a market with no venue tag our map turns into a category, and unknown, a market with no category in our metadata when the hour was built, mostly one not yet asked. The hour's audit gives how many. It appears only in hours built with our category gate or re-sorted later.

Other files of the hour, among them the verify report, twin_check, opening_state.parquet and the recorder logs, are under The hour's sidecars below.

Columns

Every column is nullable in the Parquet schema. In an hour that passes verify, the columns marked never null below never are. The lean file has 27 columns. The provenance file has 11. The provenance file has no timestamp_received column. The receive time is stored once, in the lean file. Each lean file's footer declares the sort category, sub_tag, market, asset_id, timestamp, row_ordinal, and each provenance file row_ordinal. The types below are Arrow types.

Lean file: <hour>.<category>[.<sub_tag>][.p<N>].parquet, 27 columns

Provenance file: <hour>.<category>[.<sub_tag>][.p<N>].provenance.parquet, 11 columns

The hour's sidecars

Beside its category files, an hour has a verify report and, when the manifest lists them, a twin_check file and an opening state.

Also published with some hours: each recorder's own connection log, <hour>.connection_events.<recorder>.parquet, when we have it. It is the recorder's file, copied byte for byte. Its columns are the recorder's, and its timestamp_received is in nanoseconds. A pair-4 manifest has a companions block for files that will come later. In pair-4 hours as first published, order_intent and chain_match are pending.

More than one machine is watching

Several of our recorders capture the feed. Every row names the recorders that saw it, in witness_set. The manifest's recorders block gives each recorder's summary and outages, and, since absent_fields is hour-wide, recorders.witnesses.<recorder>.status gives each recorder's own state for its connection log. Its three clock figures, clock_steps, clock_step_total_ns and clock_step_max_abs_ns, show how much a recorder's clock was corrected in the hour, not how far it was from true time. In pair-3, the absolute error of a recorder's wall clock against UTC is not measured.

The primary witness

Each hour names one primary recorder, primary_witness, with primary_reason saying why it was chosen. The choice is made once a day from the previous week's coverage, so it can be up to a day out of date. An hour where the primary recorder did badly uses that hour's best recorder instead (primary_reason = bad_hour). Rows with is_primary_copy = true carry the primary recorder's own receive times and sequence numbers. The other rows are what the other recorders added. Hours merged before this rule have primary_witness = null and is_primary_copy = false on every row.

The hour boundary

An hour holds rows stamped well before it. In 2026-10-11T01, 47.3% of book rows were stamped before the hour (median lag 7.4 h), and 0.0% of price_change, best_bid_ask, last_trade_price and tick_size_change rows. Hour 01 has 624,528 book rows stamped before 01:00. opening_state.parquet carries the previous hour's end state across hh:00. A reference-price window that ends exactly on the hour, for example 02:00:00, is in the next hour's file. A recorder's outage interval can reach outside the hour.

What is new in v3.1

• Built from v3. Each v3.1 hour is derived from a v3 source hour. The manifest embeds that hour's manifest as derived_from.source_manifest, and gives the v3 source file's sha256 as source_sha256.
• Several files per hour. Split by category into lean and provenance pairs, listed by the manifest.
• Mirror rows dropped, and rebuildable. In a two-outcome market, a row for the other token that is an exact mirror is dropped, and the canonical row it mirrors has mirror_dropped = true. Rebuilt with the provenance file, a row equals the v3 row exactly, receive time and arrival skew to the millisecond. The twin_check file lets you check your rebuild.
• Repeat books dropped on purpose. v3.1 drops repeat books on purpose, such as book snapshots with the same timestamp and levels as the previous book, received again later. There were 103,652 in hour 01, counted as book_rows_dropped in the manifest. They cannot be rebuilt.
• No spread column. Kept only where the venue differs, in spread_raw and spread_differs.
• Millisecond receive times, on every row. Kept to the millisecond, rounded down, as recv_lag_ms on the lean row.
• A unique row number. row_ordinal takes each value from 0 to row_count - 1 once across the hour.
• Per-hour files beside the data. The verify report, twin_check and opening_state.parquet (above).
• Re-issued hours. An hour can be re-issued as a new generation, when a companion is added or the hour is re-sorted, and the manifest it replaces is kept (below).
• No public lists of dropped rows. We check every hour against them before it goes up, then delete them.

Ordering

The manifest's notes give a token's row order as (timestamp, sequence, row_ordinal). Recipe 2, given the provenance file, returns each token's rows in that order. A rebuilt mirror row sorts by its own sequence. Stored rows that tie on timestamp and sequence come out in their published order, row_ordinal. Rows stamped in the same millisecond are ordered by sequence only if you read the provenance file, where sequence is.

row_ordinal is unique: it takes each value once across the hour's files. In an expansion with mirror rows rebuilt, a rebuilt row keeps its canonical row's row_ordinal, so there (row_ordinal, is_twin) is unique.

The order in the file is not the order anyone saw

Each lean file holds one run of rows per event type, in this order: best_bid_ask, book, last_trade_price, market_resolved, new_market, price_change, tick_size_change. A type with no rows in the file has no run. Within each run, rows are sorted by category, sub_tag, market, asset_id, timestamp and row_ordinal, nulls last. The footer declares this sort (sorting_columns) for each row group. row_ordinal is numbered the same way, by event type, then category, sub_tag, market, asset_id, timestamp and sequence. So one market's rows of one event type take one unbroken range of row_ordinal. An hour re-sorted under a new category map keeps every row's row_ordinal, so there it need not rise through a file.

timestamp is the venue's time. For an exact cut by arrival, use timestamp + to_milliseconds(recv_lag_ms), never timestamp. To see one recorder's own view, keep the rows with is_primary_copy = true. They may miss a token's first book of the hour, so rebuild books from all the rows, or start from opening_state.parquet.

Which machine recorded each row

The provenance columns answer it.

witness_set names the recorders that saw the row, written |a|b| with a bar at both ends.

source_witness names the recorder whose copy we kept.

is_primary_copy is true when that recorder is the hour's primary_witness. Keep those rows to see the primary recorder's own receive times and sequence numbers. Use that filter for timing and coverage questions, not to rebuild books.

arrival_skew, next to witness_set, shows how receive times differ between recorders, but it mixes the network path with clock error.

An hour is a container

Never glob for an hour's files: a missing or extra part gives wrong answers. Take the files from the manifest. quickstart.py picks up a category's split files for you. Its hour_files refuses a manifest whose lean and provenance files are not exactly the parts layout_split.parts gives. A companion in parts is all of its listed parts: read every one.

When an hour is re-sorted, its files keep their names, so the update is published in order: new files first, then the manifest, then any file no longer listed is removed. For a short window, someone holding the old manifest can download a file that has already been replaced.

An hour is also not a claim about when events happened. A book row's timestamp is the venue's time for the book, which can be hours before we received it. Hour 01 has 624,528 book rows stamped before 01:00. If you need arrival time, use timestamp + to_milliseconds(recv_lag_ms), not timestamp.

How to check what you downloaded

Three checks.

1. The manifest

Check <hour>.manifest.json against the archive's SHA256SUMS.txt, as the manifest's verification note says.

2. Every file the manifest lists

Each files entry gives the file's sha256.

sha256sum <hour>.<category>.parquet
# compare with that file's entry in <hour>.manifest.json

For a downloaded hour, quickstart.py checks the sha256 of each file it hands you: hour_files the lean and provenance files, sidecar_file the opening state or reference prices file, and check_twins the twin_check file. It refuses a mismatch and a listed file that is not there. The replay engine checks every file the manifest lists. quickstart.py does not check files read straight from the site, because that would mean downloading them whole. On a mismatch, download the hour again. The token map is not in files, so this check does not cover it.

3. The verify report

Hash <hour>.verify_report.json. The result must equal verify_check_id in the manifest's audit blocks. Until the private lists are deleted, verify_check_id is null and this check cannot be made. The report's manifest_sha256 must equal as_built_manifest_sha256 of the published manifest, and its ok must be true.

One category, without the whole hour

Every lean and provenance file has its own entry and sha256, so you can download and check one category's files on their own. check_twins takes a category, and checks the mirror rows of that category alone against the twin_check file.

What a checksum here does and does not prove

A file whose sha256 does not match its files entry is not the file that manifest lists. A mismatch is how you catch a file replaced when an hour is re-sorted, while you still hold the old manifest.

Our readers check only what they can see without the source an hour was built from. Damage that the manifest describes consistently, and that only a comparison with the source shows, is verify's to find. Our readers trust an hour whose verify report passed: the report's sha256 is the manifest's verify_check_id, its manifest_sha256 is the published manifest's as_built_manifest_sha256, and its ok is true. On any other hour, that kind of damage can change their answers without an error.

A twin_check match shows that your rebuild matches what we captured. We check that the right rows were dropped in the first place before we publish an hour.

Corrections and generations

An hour can be re-issued as a new generation under the same layout_version: when a companion file is added, or when the hour is re-sorted under a new category map. The manifest it replaces is kept beside the hour as <hour>.generation-<generation>.json, byte for byte. The new manifest's previous_generation gives the old generation and manifest_sha256.

Re-sorts. Polymarket sometimes tags a market after we have published its hour. We then re-sort the hour under the new category map instead of rebuilding it. Every row stays exactly as it was, with the same row_ordinal. Only category, sub_tag and the file a row sits in change. recategorised lists every market that moved, with its old and new category. A market takes a category only if our fetch of it from Polymarket succeeded.

The verify report is not replaced. The report we publish is the one for the hour's first generation. We publish no replacement of it and no other verify report. We check a re-sort against the hour's twin_check file and the kept previous manifest, and do not publish that report. <hour>.verify_report.<generation>.json is reserved for a later generation's report. No hour has one yet.

Keys and rules. Within a layout_version, a manifest's keys do not change in place. A new key arrives only with a new generation, and never changes the meaning of a key already there. A change that would is a new layout_version. Once hours are published under a layout_version, a change to a verify check, threshold or detail text raises n in verify_rules and leaves the layout_version as it is.