Since only Northern Light can provide an exact count of the size of their Web database on a given date, I use the number of hits reported by Northern Light as the starting point and then estimated the size of the other databases using that number times the percentage of a search engines total hits from the 25 searches used on the relative size analysis as compared to the number found by Northern Light. While the terms used for the 25 searches are not completely chosen at random, they were chosen from a variety of subject areas and countries.

Northern Light has a technique that can be used for an at the moment count. On Northern Light, limit to the Web and search search or not search. The resulting number is the current size of their Web database. It should work with any term. The OR NOT operation finds every record which has the term as well as every record which does not have the term.

AltaVista used to have a similar technique, although it was far from accurate. With their changes on Oct. 25, 1999, it no longer works. However, just in case it starts working again, the trick was to enter an asterisk * in their Advance Search Boolean Box and check the Count Documents box. An early technique of +* on the Simple Search used to give a much higher number, but it no longer works either.

So why these discrepancies between claimed size and these estimates? There are several factors to consider which may explain these results beyond the limit of basing the estimates on a small number of searches and on Northern Light's reported numbers.

The Inktomi-based search engines (HotBot, Snap, and Yahoo!'s search engine component) are run on clusters of computers. According to Inktomi, at any point in time, some of the computers may be down for backup or other maintenance. Consequently, their entire database may not be searched at any point in time. My estimates thus reflect what was available to be searched at the time the searches were run.

AltaVista will time out on some searches and only deliver partial results. Since my numbers are based on actual number of hits found, that may cause AltaVista's size to be under-represented. On the other hand, if Inktomi and AltaVista do not have their full databases available to searchers, what is the use of that extra size if it is inaccessible? These estimates may well give a better sense of the size of the accessible portion of the search engine databases.

See also the previous Total Size Estimates:
Aug. 1999
May 1999
March 1999
Jan. 1999

While decisions about which Web search engine to use should not be based on size alone, this information is especially important when looking for very specific keywords, phrases, and areas of specialized interest. See also the following statistical analyses:

• Relative Size of the search engines
• Overlap between the search engines
• Unique Hits hits from each
• Dead Links comparison
• Change Over Time