Since only Northern Light can provide an exact count of the size of their Web database on a given date, I used the number of hits reported by Northern Light as the starting point and then estimated the size of the other databases using that number times the percentage of total hits from the 25 searches used on the relative size analysis.

Both Northern Light and AltaVista have techniques that can be used for an at the moment count. On Northern Light, use the Power Search limited to the Web and search count or not count. On the AltaVista search, enter a +* in their Simple Search. AltaVista's reported count of 106,631,389 on March 5th proved to be relatively close to my estimate based on Northern Light's reported number.

So why these discrepancies between claimed size and these estimates? There are several factors to consider which may explain these results beyond the limit of basing the estimates on a small number of searches and on Northern Light's reported numbers.

The Inktomi-based search engines (HotBot, Snap, and MSN Web Search) are run on clusters of computers. According to Inktomi, at any point in time, some of the computers may be down for backup or other maintenance. Consequently, their entire database may not be searched at any point in time. My estimates thus reflect what was available to be searched at the time the searches were run.

AltaVista will time out on some searches and only deliver partial results. Since my numbers are based on actual number of hits found, that may cause AltaVista's size to be under-represented. On the other hand, if Inktomi and AltaVista do not have their full databases available to searchers, what is the use of that extra size if it is inaccessible? These estimates may well give a better sense of the size of the accessible portion of the search engine databases.

Lycos does not index every word on Web pages, so its results will be lower in my estimates which are based on the number of results. However, since the full page is not indexed in Lycos, my measure should be a useful guide for users trying to find unique words or phrases on the Web.

The other notes on the relative size page apply to these estimates as well.

See also the Jan. 1999 Total Size Estimate.

While decisions about which Web search engine to use should not be based on size alone, this information should be part of the decision. See also statistics on the lack of overlap between the search engines, unique hits, dead links, and the change over time.