Search Engine Showdown
[an error occurred while processing this directive]


Northern Light Tops Again
Search Engine Statistics: Database Relative Size
by Greg R. Notess

Data from search engine analysis run on May 5, 1999

New Fast Search ranks fourth!
Where's HotBot? See note 1 and use Inktomi/Anzwers figure for estimate.
See Total Size Estimate page for actual database size estimates.

This chart compares the size of the databases of the Web search engines. For this comparison, I used 25 single keyword and phrase searches that should have been processed identically by each search engine. Bar Chart Since Infoseek and Northern Light automatically recognize word variants and plurals, those terms that could be pluralized were OR'ed together in search engines that do not support automatic plural searching. This and other inconsistencies between the search engines may skew the results slightly.

This comparison is based on the reported number of hits from each database, verified by visiting the last page of results when possible. This is not a measure based on precision, recall, or relevance but only on the raw database size. As such, it provides an important measure of database coverage. For earlier comparisons see below:

Older Charts with Largest Three at that Time
March 1999:Northern Light, AltaVista, HotBot
January 1999:Northern Light, AltaVista, HotBot
August 1998:AltaVista, Northern Light, HotBot
May 1998:AltaVista, HotBot, Northern Light
February 1998: HotBot, AltaVista, Northern Light
October 1997:AltaVista, HotBot, Northern Light
September 1997:Northern Light, Excite, HotBot
June 1997:HotBot, AltaVista, Infoseek
October 1996:HotBot, Excite, AltaVista

While decisions about which Web search engine to use should not be based on size alone, this information is especially important when looking for very specific keywords, phrases, and areas of specialized interest. See also the following statistical analyses:

Note 1 *With HotBot now grouping hits under the site, it is nearly impossible to calculate a total number of hits. The number reported as of the date of this comparison is actually the number of sites, or an estimate thereof. After consulting with technical people at both HotBot and Inktomi, the numbers were obtained by running the same searches on Anzwers. According to Inktomi, both of these databases were running on the same cluster as HotBot. While there is no guarantee that the same will hold true on any other date, it should provide an accurate representation of the HotBot database for the date on which these comparisons were run.