v3.1 preview

Three things make it smaller. With the provenance files, every v3 row and column comes back, except book snapshots identical to the token's previous book, which add nothing.

• Mirrored rows kept once. In a two-outcome market every update to one token is mirrored on the other (price 1 - p, sides swapped). When the mirror is exact we keep one copy and set mirror_dropped = true. That was about 44% of the rows in these hours. Only exact mirrors are dropped: if anything differs, both rows stay.
• Repeat books dropped. A book snapshot identical to the token's previous book adds nothing.
• Bookkeeping moved out. Which machines saw each row, receive times and sequence numbers now live in separate provenance files, so you only download them if you want them. Event and provenance files together are about 70% of v3 (1.54 GB against 2.17 GB for hour 00), and most questions need only the event files.

Every Parquet file is split into row groups, and each group records the smallest and largest value of every column. A query engine reads those few bytes first and skips any group that cannot hold what you asked for. v3 was already sorted by market, token and time. v3.1 goes further in three ways: each category is its own file, so a whole category is one or two files; groups are about four times larger, so an engine makes fewer, bigger reads; and the file holds half the rows and only the columns most queries need, so there is less to read in the first place.

Group size is a trade-off. Too small and an engine makes thousands of tiny reads; too big and it reads more than it needs. Kuinox rewrote a v3 hour at sizes from 100 thousand to 10 million rows per group and timed DuckDB, Polars, DataFusion and chDB on each. Small groups were 5 to 7 times slower in chDB. We settled on about a million rows per group, cut only where the category changes. Timings on hour 01 in DuckDB:

Over the internet, read only the category you need: market_categories.parquet gives every market's category. Reading every file of an hour costs many requests and runs into the rate limit.

Your numbers will differ. Tell us what you see in your engine.

Pick the slice. Because of the layout above, a filter on category, market or token reads only the groups that hold it, and a time window inside one of those narrows it further. Crypto is 8.4 million of the 88.7 million rows in hour 01, so a Crypto-only query reads about a tenth of the file. Engines that read Parquet over HTTP, such as DuckDB and Polars, fetch only those parts, so you can query an hour without downloading all of it. For repeated work, download the hour once and query it locally.

Pick the detail. The event_type column sets the resolution. Rows in hour 01, by type:

Pick the depth and the interval. For the order book itself, the quickstart's book_snapshots gives a token's book at any interval you choose (every second, minute, 10 minutes) and to any depth (the top N levels or the full book), straight from these files with DuckDB. It matched our replay engine on every level we compared, and one token over a whole hour takes about 5 seconds at one-second snapshots ten levels deep.

Pick one side or both, carefully. In a two-outcome market almost every update to the second token (complement_token in markets.parquet) is stored as a mirrored row of the first. A plain filter on the second token's id sees about 1% of its rows. Query by market and rebuild the mirrored rows with the quickstart's expand_twins. With the provenance files passed in, it gives exactly the rows and columns v3 had for that market, except dropped repeat books.

Pick the extras. Receive times, sequence numbers and which machines saw each row sit in the provenance file, and the opening state and reference prices have their own files. Download them only if your question needs them.

opening_state.parquet gives every token's state at hh:00:00: the full book where we have one, best bid and ask, last trade and tick size. The books come from the same replay engine that rebuilds books from these files, carried forward from our own recording of the previous hours, and marked derived = true. Book times are the top of the hour for now; the last trade carries its own time.

To check it, we compare every real book that arrives during the hour with our carried book for that token at that moment. They matched 98.4% to 98.8% of the time across these hours. The misses are messages missed earlier, which stay wrong until the next real book corrects them. A token we have never seen a full book for gets best bid and ask and its last trade, with the book marked unknown (null); an empty list means the venue's book really was empty.

reference_prices.parquet holds the Chainlink (8 coins) and Binance (6 coins) prices Polymarket streams, about one a second. Since August, the 5 minute markets settle on a 30 second Chainlink TWAP, and the 15 minute and 4 hour markets on a 60 second one. With these you can see exactly what a market settled against, and backtest anything that trades into the close.

Polymarket does not publish the TWAP formula, so we worked it out from Chainlink's own TWAP prints. It is the plain average of the one-per-second Chainlink price over the window. Our rebuild matched Chainlink's value in every complete window we tested (3,456 of 3,456 at 30 seconds, 1,884 of 1,884 at 60). These three hours predate our recording of the TWAP stream, so their TWAP rows are rebuilt and marked derived. About 2% of seconds never arrive on the Chainlink stream, and the missing_seconds column counts them in each window.

One row is one order in a settlement transaction: the transaction hash (tx_hash), the fill's log index (fill_log_index), the order_hash, the order's wallet (maker), whether it was the taker (is_taker), the amounts, the fee, and the order's own timestamp client_timestamp_ms. Join per fill on tx_hash and fill_log_index, not on the hash alone: one transaction holds a taker and several makers. Fills on the newer combinatorial contract are listed from their logs only, without the order (about 0.5% of fills in hour 00). Each hour's order-intent.manifest.json gives the block range, the match rate and what every row kind means, and the README has more.

Three things to know before you join it to the event files:

• Join venue hour H against on-chain hours H and H+1. A fill lands in a block a few seconds after the venue's trade message, so trades near the end of an hour settle in the next hour's blocks.
• token_id is text, asset_id is bytes. Here token_id is the decimal uint256 as text; the event files' asset_id is the same number as 32 bytes, big-endian. In Python, bytes.fromhex(f"{int(token_id):064x}") converts it.
• client_timestamp_ms is the signer's, and 0 means unset. Whoever signs the order sets it, and it is not when the order was signed or sent. A 0 means the signer left it unset: about 1% of order rows in hour 00, from 12 wallets. Treat it as missing.

Every hour passed 19 checks before going up, with one more check not run in this preview. They cover row counts, ordering, nulls, schema and layout. They also confirm that every dropped mirror rebuilds exactly, and that the drops can be recomputed from the source independently. Each hour's verify_report.json lists them. We also ran 23 user-style tests on each hour against v3: every row comes back, four busy markets rebuild exactly on all 28 v3 columns, and books replayed minute by minute match. The scripts and results are in user-tests/, so you can run them yourself. How to expand the mirrored rows is in the README, and markets.parquet gives each market's two tokens.