It's dangerous to go alone. Take this.
One hour of this archive is 1.26 GiB and about 119.3 million rows. Excel will not open it. Google Sheets will not open it. That is not a reason to leave. It is a reason to bring something that can.
You do not have to download anything. Parquet files are columnar, which means a query can reach into one over the network and pull back only the columns it needs. Every example below runs against the file where it sits, on an ordinary laptop, in seconds. The rest of the hour never moves.
And you do not have to know SQL. Each step gives you a sentence to paste into ChatGPT, Claude, Gemini or whatever you use, and then the query it should hand back. Read the two side by side for a few minutes and you will start recognising the shape. That is most of the skill.
Getting set up. Four steps, about five minutes.
Nothing here needs an account, and nothing you type can break your computer.
A Open a terminal
The terminal is just an app where you type commands instead of clicking things. You already have one.
- Windows: press the Windows key, type
powershell, press Enter. - Mac: press Command + Space, type
terminal, press Enter. - Linux: press Ctrl + Alt + T.
A window opens with a blinking cursor waiting for you. That is all it is.
B Install DuckDB
DuckDB is a free database that runs as one file on your own machine. Hit Copy on the line for your computer, paste it into that window, press Enter, and wait a minute.
winget install DuckDB.clibrew install duckdbWindows: when it finishes, close this window and open a new one. The installer says so in its own last line - "Path environment variable modified; restart your shell to use the new value" - and it means it. A terminal reads that setting once, when it starts, so the window you installed from cannot see DuckDB no matter what you type. Reopen PowerShell exactly as in step A.
On a Mac without Homebrew, or if either line is refused, the DuckDB download page has a plain installer for every system.
C Start it
Type this one word and press Enter.
duckdbThe prompt changes to a D. That means you are now typing into DuckDB
rather than into your terminal. When you want to leave, type .quit and press
Enter.
Got "the term 'duckdb' is not recognized"? That is the note in step B, not a broken install. Close the window, open a new one, and type it again.
D Set the clock to UTC
SET TimeZone = 'UTC';That is the only setting you need. Every timestamp in this archive is UTC, and without this line your database quietly re-labels them in your own timezone, so every hour you go on to cite is wrong by the size of your offset.
Nothing to install. Earlier versions of this page asked you to run INSTALL httpfs and LOAD httpfs first. You do not: DuckDB fetches what it needs the first time it sees a web address, on its own. Measured on DuckDB 1.5.5.
That is the whole setup. From here on, every box below works the same way: hit Copy, paste it into that same window, press Enter. Each query ends with a semicolon, which is how DuckDB knows you have finished typing, and each box is one paste.
Ctrl+V may do nothing. The old Windows console does not use it: right-click pastes instead. Windows Terminal takes Ctrl+Shift+V. On a Mac it is Command+V as usual. This catches almost everybody once.
Five things to try
All five use the same single hour, 2026-08-28T16, which the audit
records as complete: sixty minutes out of sixty, with capture reaching both edges.
1 See what an hour actually holds
An hour is not one kind of thing. It is quote updates, full book snapshots, trades, and the occasional market opening or resolving. Start by counting them.
Ask your assistant: Using DuckDB, count the rows by event_type in this Parquet file over HTTPS: https://archive.pendulumflow.com/v3/2026-08-28/16/2026-08-28T16.parquet
SELECT event_type, count(*) AS rows
FROM 'https://archive.pendulumflow.com/v3/2026-08-28/16/2026-08-28T16.parquet'
GROUP BY 1 ORDER BY rows DESC;About 16 seconds: this one has to look at every row, and the rest
are faster. price_change is 105 million of the 119, which is
what an orderbook mostly is - the price of something moving a little.
2 Find out what people were betting on
Markets that OPEN during an hour carry their own plain-English question and a slug you can paste onto the end of a Polymarket URL. This is the friendliest query on the page and it runs in under a second.
Ask your assistant: From that same file, show me the question and slug for every market with event_type = 'new_market'.
SELECT question, slug
FROM 'https://archive.pendulumflow.com/v3/2026-08-28/16/2026-08-28T16.parquet'
WHERE event_type = 'new_market' AND question IS NOT NULL
GROUP BY ALL
LIMIT 20;973 markets opened in this one hour. Only markets created here carry a name, though - most of the 4,720 that traded were opened earlier, so their rows have an id and no question. Step 5 fixes that.
3 Find the biggest bets
Every trade the exchange reported is here, with its price, its size and the second it happened. Sort by size and you are looking at the largest positions anyone took.
Ask your assistant: Show me the ten largest trades in that file by size, with the time, price, side, and the market id as hex with an 0x prefix.
SELECT strftime(timestamp_received, '%H:%M:%S') AS at,
round(price, 3) AS price,
round(size, 2) AS size,
side,
'0x' || lower(hex(market)) AS condition_id
FROM 'https://archive.pendulumflow.com/v3/2026-08-28/16/2026-08-28T16.parquet'
WHERE event_type = 'last_trade_price'
ORDER BY size DESC
LIMIT 10;The top row is a 411,127 unit buy at 0.48, at
16:38:02, and the second is a 403,000 sell at 0.999 -
someone taking near-certainty off the table. The hex() is not decoration:
market ids are stored as raw bytes, and printing one without it gives a column of numbers
instead of an id you can look up.
4 Watch the odds move
This is the one that makes an archive worth keeping. Pick a market and replay what the odds did, minute by minute, while it was happening.
Ask your assistant: For the market whose slug is ethereum-above-3000-on-september-4-2026, show the average best bid and best ask per minute, in order.
SELECT date_trunc('minute', timestamp_received) AS minute,
round(avg(best_bid), 3) AS bid,
round(avg(best_ask), 3) AS ask
FROM 'https://archive.pendulumflow.com/v3/2026-08-28/16/2026-08-28T16.parquet'
WHERE event_type = 'best_bid_ask'
AND market = (SELECT market FROM 'https://archive.pendulumflow.com/v3/2026-08-28/16/2026-08-28T16.parquet'
WHERE slug = 'ethereum-above-3000-on-september-4-2026' LIMIT 1)
GROUP BY 1 ORDER BY 1;Will the price of Ethereum be above $3,000 on September 4? The bid sits at 0.10 through 16:16, jumps to 0.485 at 16:33, and settles at 0.297 - the crowd going from "one in ten" to "coin flip" and back to "about three in ten" inside half an hour. That half hour is the reason an archive like this exists. Ask your assistant to plot it.
5 Give every market a name
Steps 2 and 3 leave you holding ids. The fix is to read the new_market
rows out of a whole day at once and keep them as your own lookup table. This is where you
stop poking at one hour and start doing research.
Ask your assistant: Read every hour of 2026-08-28 from this archive, keep only the new_market rows, and save market id, question and slug into a local table I can join against.
CREATE TABLE names AS
SELECT DISTINCT '0x' || lower(hex(market)) AS condition_id, question, slug
FROM read_parquet(list_transform(range(0, 24), lambda h:
printf('https://archive.pendulumflow.com/v3/2026-08-28/%02d/2026-08-28T%02d.parquet', h, h)))
WHERE event_type = 'new_market' AND question IS NOT NULL;
SELECT count(*) AS named FROM names;SELECT n.question, round(t.size, 2) AS size, round(t.price, 3) AS price
FROM 'https://archive.pendulumflow.com/v3/2026-08-28/16/2026-08-28T16.parquet' t
JOIN names n ON n.condition_id = '0x' || lower(hex(t.market))
WHERE t.event_type = 'last_trade_price'
ORDER BY t.size DESC
LIMIT 10;The first box builds the twenty-four hour URLs itself and reads them in one pass,
then prints what it found: 21,272 named markets, in about 13 seconds.
The count is there because CREATE TABLE says nothing on its own, and a
silent prompt is indistinguishable from a failure. It spells the list out
because a * wildcard does not work over plain HTTPS - there is no directory
for it to expand. The join then shows only markets NAMED that day, so the biggest trades
from older markets drop out of it; widen the range and they come back.
When you get stuck
You will, and it is almost always one of four things. Hand your assistant this page along with the error and it will usually spot it at once.
- "syntax error at or near" with nothing after the "at or near". An empty command reached the prompt, usually a stray blank line in a paste. The statement above it almost certainly ran; try the next box and see.
- Ctrl+V does nothing. Right-click pastes in the old Windows console; Windows Terminal uses Ctrl+Shift+V.
duckdbis not recognised, right after you installed it. On Windows the installer changes your PATH, and a terminal only reads that when it starts. Close the window and open a new one.- Timestamps are out by a few hours. You skipped
SET TimeZone = 'UTC'. - A market id prints as a list of numbers. It is raw bytes. Wrap it in
lower(hex(...)). - Everything is null. Columns belong to event types.
bidsandasksexist only onbookrows;priceandsizeonly on trades and price changes. Filter onevent_typefirst. - It is slow. Ask for fewer columns. A query naming three columns
downloads three columns;
SELECT *downloads all twenty-five.
What you are actually holding
Not a summary, and not a daily close. This is the order book itself, every quote and every trade, recorded as it happened by several machines at once and merged. You do not have to take our word for any of it: the audit publishes what we measured, including where we fall short, and the format notes describe every column in every era. The older V1 and V2 eras are third-party mirrors and are thinner, and the difference is measured rather than asserted.
It is free and it is CC BY 4.0. If you build something with it, we would love to see it.