Working with the historical data

Practical notes on every era's files: what to expect and how to handle it. Points 1 to 3 are not our discoveries. Antonín Kříž found them in the v2 data, wrote them up, and rebuilt the collector so the same problems would not recur in what came after.

First, what a witness is

Everything under pmxt/ and third-party/ag6/ came from one collector each. Under v3/ several machines record the same feed at once, in different data centres, and the merge keeps the union of what they heard with the duplicates removed. Those machines are the witnesses, and a few columns exist only because of them.

Two consequences run through the notes below. More machines on an hour means better coverage, because a gap in one is covered by another, and the audit publishes how many heard each hour. And timestamps from different machines are not directly comparable, because each one stamps with its own clock; see time between two events, when different machines recorded them, under what this data cannot tell you.

The rest is on the v3 page: how the merge works, and what source_witness means.

0. The same event has different names in different eras

A filter written for one era returns nothing on another, and returns it silently. Nothing errors: you get zero rows, which reads exactly like a quiet period. The v1 era holds 1,283 hours of files, and it is the only era that uses these names, so a corpus-wide query written for the later ones drops every one of them without saying so.

The asset_id difference catches people joining v2 to v3. It is the same token either way, but v2 writes it as a decimal string and v3 writes the same number as 32 raw bytes, so a join on the column returns zero rows and reports nothing wrong. Convert one side first: the bytes are the integer big-endian, so str(int.from_bytes(v, "big")) gives you the v2 form.

Read the era's own page, v1, v2, AG6, v3, before writing a query that spans them, and check your row count against the hours you expected to cover. A cross-era count that matches one era's total exactly is usually a filter that matched nothing in the other.

1. Millisecond timestamps can tie

Timestamps in the historical (PMXT-era) files are millisecond resolution and can tie for the same asset. Where they tie, the original exchange ordering is not recoverable. Treat those files as Snapshot Grade: books, candles and volumes are fine, exact event replay is not.

2. Ordering within an hour is not guaranteed stable

The files are sorted by receive time, not by the exchange's own timestamp, and rows that tie on receive time are in no fixed order. Sort by (asset_id, timestamp) on read.

3. De-duplicate on read

Two separate things. Some repeated price updates were dropped by the collector's own de-duplication key: a live comparison saw about 4 dropped per 96 messages, and those cannot be recovered. And roughly 2.5% of trades appear twice; de-duplicate them on read by transaction_hash plus asset_id, price, size and side, as in the query below.

import duckdb

duckdb.sql("""
  SELECT * EXCLUDE (rn) FROM (
    SELECT *, row_number() OVER (
      PARTITION BY transaction_hash, asset_id, price, size, side
      ORDER BY timestamp_received
    ) AS rn
    FROM read_parquet('polymarket_orderbook_*.parquet')
  ) WHERE rn = 1
""")

4. From v3/ onward, de-duplication is exact, ordering is not

From the v3/ prefix onward the collector records an explicit sequence. It was unique across 2026-08-24T05 (84,148,495 rows, zero duplicates), so de-duplication needs no tie-breaking. It is not a global ordering: it is collector-local provenance, and where several hosts saw the same event the merged row carries one host's number. Sorting by it comes close to time order and is not guaranteed to reach it. Order by timestamp_received. See the collector's data model (DATA_MODEL.md on GitHub).

What this data cannot tell you

Things people try to compute from an archive like this that it cannot support. Each is here because a column in these files invites it.

An effective or realized spread, from averaged quotes

The spread that matters to a fill is the one prevailing at that fill. A daily, or hourly, mean of quoted spreads is a different quantity and cannot stand in for it. In v3 every best_bid_ask event carries its own best_bid and best_ask; in v2 and AG6 compute them from bids and asks, because the columns are often null there. Either way, join to the quote prevailing at your timestamp, not to a period average.

Depth, from fills

last_trade_price and price_change tell you what traded, not what was resting. Resting depth appears only in book snapshots, and those are periodic, not continuous: roughly every 2 to 3 minutes per market in v1, see each era's page.

Exact time between consecutive trades, in the historical files

Millisecond timestamps can tie and export order is not guaranteed, so consecutive events for one asset cannot be separated exactly. Inter-event gaps in the pmxt/ era are approximate. They are approximate in v3/ too: sequence is unique, so it de-duplicates exactly, but it is collector-local and does not order events across hosts. Microsecond timestamp_received makes v3 gaps far finer than v1 or v2. It does not make them exact.

Volume, if you aggregate across every asset without care

A market's outcomes are complements; in v3, new_market lists them in assets_ids. Before summing size across asset_id, check whether your aggregation counts one economic trade on both sides of the pair. Aggregate per asset, or per market with a stated rule, and say which you did.

Time between two events, when different machines recorded them

Rows with different source_witness values were stamped by different clocks on different machines. Subtracting one timestamp_received from another across that boundary measures the offset between two clocks as much as it measures anything that happened in the market, and it does so without any sign that it has.

Filter to a single source_witness before comparing arrival times, so every row you are subtracting shares one clock. The venue's own timestamp is unaffected: it comes from the exchange, not from us, so use it when you want event time rather than arrival time.

Filtering to one machine keeps most rows, if you pick the right machine for that hour. Which machine's copy dominates changes with the merge rule and the hour: under the older rule one machine carried 99.2% to 99.5% of all published rows, and about 99.8% of price_change, across 3 hours measured on 2026-08-30, with book snapshots the exception at about 84% for a different machine; under the newer rule the fastest machine wins each row. So do not hard-code one: read the hour's witness_stats.json, which counts rows per machine per event type, before choosing which machine to filter to.

And read the column for what it says. source_witness names the machine whose copy of a row was selected during the merge. Which copy that is depends on the hour: which copy the merge kept is a per-hour fact, read from the hour's merger stamp: fcbb2804 and later keep the earliest-received copy; earlier stamps keep the copy from the machine most recently added to the fleet. It does not mean that machine was the only one that heard the event, it is not a count of how many did, and shares computed from it measure the merge's configuration rather than the feed. For what each machine actually heard, use witness_set, which lists every machine that had the row rather than the one whose copy was kept.

And the rows here cannot tell you how fast each machine was. Every row carries the timestamp_received of the copy that was kept, chosen by that hour's kept-copy rule above, so a machine appears only for the rows the rule gave it. Grouping arrival times by machine over this file therefore describes the merge's selection, not the machines. Where an hour publishes a witness_stats.json file, that file carries the measured distribution per machine and per event type over every row that arrived, including the copies that were not kept, which are not in this file at all, and it is the only place that measurement survives. It therefore counts several times as many rows as the hour's Parquet, because each machine's copy is counted once.

A fill, from a price you touched

These files record what the book showed and what printed. They do not record what an order of yours would have received. A replay that fills every order the moment its price is touched overstates fills and understates the cost of the ones you get: at a touched price your order sits behind whatever size was already resting there, and it fills only after that size trades or cancels. When it does fill, the market has more often moved against you than for you, because the orders that get filled first are the ones on the wrong side of the next move.

So a market-making backtest needs two things this archive cannot supply: a queue-position model, which decides how much of the resting size ahead of you traded before your turn, and an adverse-selection model, which prices the fills you did get. Two readers of this archive put it plainly: "touching a price is not the same as getting filled" - @Polius20071, and "the touch-price execution assumption will systematically overestimate returns" - @3heeeeee. Treat any touch-fill result as an upper bound, and say so beside it.