The open source monthly Web crawl Common Crawl now “contains 3.08 billion+ web pages” and 250TB of uncompressed data. Back in September 2015, when this blog last took a look at Common Crawl, that was 145TB.

Advertisements