gossip-observer: Bitcoin Lightning Network P2P traffic capture, timing data, and related metadata

This dataset is a capture of Bitcoin Lightning Network (BOLT 7) gossip messages as seen by a set of instrumented Lightning nodes ("collectors"), with microsecond timestamps for each send and receive. It is meant for studying how the P2P network functions: propagation delay, latency, fanout, redundancy, and how peers behave during sync.

Collection

Each 'collector' is a modified LDK node that captures the gossip messages it sends to or receives from a peer. Recorded types are channel_announcement, node_announcement and channel_update, plus a small number of pings. Captured messages and timing data are exported daily to Parquet files. Collection and infrastructure code: github.com/jharveyb/gossip_observer.

This version covers 21 collectors running on 2 hosts near NYC. 7 collectors each own one or two public channels (see collectors.csv). Peers are selected by splitting the network into communities with a hierarchical stochastic block-modelling algorithm, assigning a collector to each community, and then randomly selecting peers from that community. The infrastructure targets a total of 700 connections right now.

Files

One ZIP per table per month, named {table}_{YYYY-MM}.zip. The Parquet files inside the ZIP are zstd-compressed. Unzipping all of them into one directory gives a Hive-partitioned tree:

v2/table={table}/year={YYYY}/month={MM}/{table}_{START}_{END}.parquet
v2/table={table}/year={YYYY}/month={MM}/MANIFEST.csv

Each Parquet file covers the half-open UTC day [START, END) and carries the key-value metadata format_version=2. MANIFEST.csv lists the path, size and MD5 of each Parquet file in that month.

collectors.csv lists every collector with:

• its public key, a short id and its chain backend;
• which collectors share a host (colocated_group);
• the first and last timestamp it logged;
• one row per public channel it owns: SCID as a u64 and in BOLT 7 block x tx x output form, counterparty public key, and the channel_announcement first seen time.

Tables

Messages are identified by hash, a XXH3-64 hash of the message's unpadded base64url text, stored as a UBIGINT. All timestamps are UTC.

• timings (~90% of the data): one row per message, per sending peer, per direction.
  • net_timestamp (TIMESTAMPTZ, µs): time the collector logged the message.
  • hash, inner_hash (UBIGINT): message identifiers.
  • collector, peer (BLOB): 33-byte compressed node public keys.
  • dir: 1 = received from peer, 2 = sent to peer.
  • orig_timestamp (UINTEGER, Unix seconds): the BOLT 7 timestamp field, for node_announcement and channel_update only.
• messages: hash, raw and first_seen. raw is the message body without its 2-byte type prefix; the type is metadata.type. Prepend the type to decode it with a standard BOLT 7 parser.
• metadata:
  • hash, inner_hash;
  • type: 1 = channel_announcement, 2 = node_announcement, 3 = channel_update, 4 = ping, 5 = pong;
  • size: wire bytes;
  • orig_node: the announcing node's public key, for node_announcement only;
  • scid: short channel ID as a u64, for channel messages only;
  • first_seen.
• message_hashes: outer_hash → inner_hash mapping, plus first_seen.
• message_first_seen: hash and first_seen, the earliest time any collector saw the message. The messages, metadata and message_hashes files are split by day on this column. So a message is in the file for the day it was first seen, even when timings rows for it fall on later days.

inner_hash is an XXH3-64 hash of the message's unsigned contents, leaving out the timestamp field. A node that re-signs and rebroadcasts the same node_announcement or channel_update with only a newer timestamp therefore produces a new hash but the same inner_hash.

Interpretation notes

• dir=2 rows are mostly not relays or originations. Each row is one message the collector sent directly to one peer. Nearly all of them are the collector serving gossip during a peer's graph sync, so their timestamps are serve times, not creation times.
  • The collector's broadcasts are not recorded. That covers relaying others' gossip and the first broadcast of its own channel_updates and node_announcements.
  • A collector's own messages show up as dir=2 rows in two ways: when a channel_update is sent directly to that channel's counterparty, close to when it was created; and later, as sync serves to other peers.
  • To study origination, select a collector's own messages instead of using dir=2 timing. Join metadata.scid to collectors.csv scid for channel gossip, and metadata.orig_node to collectors.csv pubkey for node_announcements.
  • Anchor propagation measurements on message_first_seen.first_seen and dir=1 rows.
• dir=2 rows exist only for peers that have stayed connected for at least 10 minutes. The number of distinct dir=2 peers is therefore a count of stable peers, not total fanout.
• Peers drop a channel_update for a channel they don't yet know about. An update with no preceding announcement can therefore appear as dir=2 with no dir=1 receipts at other collectors.

Reading the data

Use one glob per table; the five tables have different schemas. For example, in DuckDB:

SELECT * FROM read_parquet('v2/table=timings/**/*.parquet', hive_partitioning = true);

A glob across tables gives no error for count(*); it silently adds up all five tables. With hive_partitioning = true, table, year and month become virtual columns. table is a reserved word, and month is a zero-padded string. Reference queries and loaders are in analysis/ in the repository.

A single day of timings is about 140 million rows. Multi-day queries that group by hash can need tens of GB of memory. An example Python script that analyzes the data using SQL and DuckDB can be found here: https://github.com/jharveyb/gossip_observer/blob/main/analysis/parquet_queries.py.