Wikipedia Compressed SQLite database

Starting point:

• static.wiki
• HN post

We downloaded the 40.3 GiB SQLite database of English Wikipedia and created a compressed version of it with sqlite_zstd_vfs, our read/write Zstandard compression layer for SQLite3. The compressed version is 10.4 GiB (26%) and remains randomly accessible over the web in the spirit of the original (although we don't yet have a WebAssembly build of the compression layer).

sqlite_zstd_vfs is a building block of our Genomics Extension for SQLite, which bundles it along with other domain-specific features. This demonstrates a general-purpose use case of the compression layer intended for large-scale genomics data storage.