Index of /v3.1/

Our own capture, v3.1 format. Live data from PendulumFlow.

Format, provenance and known issues for this data: V3.1 · live beta. What we measured about it, and where it falls short: the audit.

Status

Live beta. New hours are published every hour as they are built. The newest published hour is 2026-11-02 10:00 UTC, and the hours since then (2 so far) are still being caught up. This page reads SHA256SUMS.txt each time it is built, so it is never ahead of what you can download.

Coverage

From 2026-10-27 00:00 UTC to 2026-11-02 10:00 UTC: 155 hours. History from before then, the v3 hours rebuilt in the v3.1 layout, is coming.

What's in an hour

Each hour is one folder, v3.1/YYYY-MM-DD/HH/, and each file name starts with the hour, as in 2026-10-30T08.crypto.parquet. What the published hours hold:

• <category>.parquet: the hour's events for one Polymarket category (crypto, sports, politics and so on), sorted by sub-tag, market, token and time. Large categories come in numbered parts.
• <category>.provenance.parquet: which recorder each row came from and the mirrored twin rows each kept row stands for. Join it on to get back every row v3 had.
• opening_state.parquet: every token's state at the top of the hour: the book where we have one, best bid and ask, last trade and tick size, so a replay can start at this hour.
• twin_check.parquet: the check that every dropped twin row rebuilds exactly, so you can confirm it yourself instead of downloading the twins.
• verify_report.json: the checks the hour passed before it was published.
• manifest.json: every file in the hour with its row count, size and sha256, and how the hour was built.
• maps/: the category maps that assign each market its category, and the diffs between successive maps (197 files, listed in SHA256SUMS.txt). The manifest names the map its hour used.

How to download

Download each file once and check it against SHA256SUMS.txt, the list of every published file and its sha256. A file whose hash does not match is a bad download: fetch it again.

# the ledger, then one hour, then check what you got
curl -sO https://dl.pendulumflow.com/v3.1/SHA256SUMS.txt
grep 2026-11-02/10/ SHA256SUMS.txt | awk '{print $2}' | while read p; do
  mkdir -p "$(dirname "$p")" && curl -s -o "$p" https://dl.pendulumflow.com/v3.1/"$p"
done
grep 2026-11-02/10/ SHA256SUMS.txt | sha256sum -c -

Or query a file in place with DuckDB, without downloading it first:

INSTALL httpfs; LOAD httpfs;
SELECT * FROM read_parquet('https://dl.pendulumflow.com/v3.1/2026-11-02/10/2026-11-02T10.ai.parquet') LIMIT 10;

The three hours we shared for testing in October are still there as older preview samples.

The files

Checksums for this prefix: SHA256SUMS.txt, verify a download with sha256sum -c. It currently names 5694 hourly Parquet files and 342 sidecars (the manifests and capture-timing files), through 2026-11-02T10.

# every file URL under /v3.1/
curl -s https://archive.pendulumflow.com/v3.1/SHA256SUMS.txt | awk '{print $2}' \
  | sed 's|^|https://dl.pendulumflow.com/v3.1/|'

# fetch them, four at a time, keeping the directory layout
curl -s https://archive.pendulumflow.com/v3.1/SHA256SUMS.txt | awk '{print $2}' \
  | xargs -P4 -I{} curl -sf --create-dirs -o {} https://dl.pendulumflow.com/v3.1/{}

# then verify the lot, in the directory you downloaded into
curl -sO https://archive.pendulumflow.com/v3.1/SHA256SUMS.txt && sha256sum -c SHA256SUMS.txt

# fetched only PART of it? filter the ledger to MATCH WHAT YOU TOOK
grep <the prefix you fetched> SHA256SUMS.txt | sha256sum -c -

155 hours, 6032 objects.

Looking for data we don't publish yet? Tell us.