How much is out there?

Estimated number of webpages calculated based on Google (blue line) and Bing (brown line) data. Credit: WorldWideWebSize

The White web (tip of the iceberg) versus the Deep web and at the very bottom the Dark web. Credit: www.reddit.com

The world wide web has grown to such a size and changes so rapidly that it is impossible to make a web census.

At the turn of the century researchers at Berkeley made a study on How Much Information and repeated it in 2003. That study focussed on the production of information in a year, and one of the conclusion was that a big portion of that ended up in a digital form and was either published on the web or made available through Internet. At that time the study estimated 5 Exabyte of information produced, 92% in digital form.  The amount of information exchanged reached 18 Exabytes and 98% of that was carried over telecommunications lines (both fixed and wireless) This figure did include voice communications (telephony).

I remember reading and being impressed by that report.

There are today, June 2016, roughly 3.63 billion pages indexed by Google and probably close to 5 billion web pages in the "white" web and many, many more in the "deep" web. A web page on the average would require several A4 paper pages to be printed on and the estimate in A4 paper pages at the end of 2015 is 136 billion paper pages (that amount of pages would require 2% of the Amazon forest to produce!).

Also notice that you cannot print the whole web: most of it, in terms of data space, is actually related to video! In 2015, according to Cisco statistics 34 Exabytes of video were carried on the Internet and 14 Exabytes of file transfer. By the end of 2016, still based on Cisco estimate, the traffic over Internet will reach 1.2 Zettabyte and will reach 2 Zettabyte by 2019.  Just to clarify, 1 Zettabyte is equivalent to transporting 36,000 years of HD video.

As impressive as these numbers are, and you can get many more clicking on the links I provided, we have just started. 

As I see it, in the future we can well have one web page (with a complex structure) associated to each one of us, that is 7 billion pages to start with. Then we can have one web page associated to most of the objects that we are interacting with. And that would raise the number to the hundreds of billions.

The deep web will also keep growing at the same pace. Today it is estimated that the deep web (data contained in data bases that are not seen by search engines that are made visible, converted into information through processing of those data, more and more via APIs) contains some 500 times the amount of data present in the white web and accessed by search engines. Some go as far as estimating the deep web to be 5,000 times bigger than the white web.  

A more sinister part of the Deep web is the Dark web, and that can be accessed by special software like Tor. I do not have figures on the dark web size, it is "hidden", remember? But although it ought to be much smaller than the white web its implication are significant and there are some 3 million users of Tor services.

Author - Roberto Saracco

© 2010-2018 EIT Digital IVZW. All rights reserved. Legal notice