Data from Feb. 21, 2000

Little overlap despite database growth!

Searches Used: Total Hits: Unique Hits:

5 small ones 795 298

This analysis compares the results of five small searches run on fourteen different search engines. The five searches found a total of 795 hits, 298 of which represented unique pages. Of those 298 hits, 110 were found by only one of the fourteen search engines while another 79 were found by only two. This analysis uses four new searches not used on past overlap analysis.

Several of the largest search engines have shown significant growth since Sept. 1999, when the overlap comparison was last run and four of the five searches were different then. Even so, the largest number of unique pages found were only found by one of the fourteen search engines. Over 76% were found by three search engines at most. Each pie slice in the pie chart represents the number of hits found by the given number of search engines. For example, the by 1 (110) slice represents the number of unique hits found by one (and only one) search engine. Of the 298 unique hits, 110 of them were found by just one of the fourteen search engines.

Even with six Inktomi-based databases (Anzwers, iWon, MSN Web Search, AOL, HotBot, and Yahoo!'s Inktomi database), there was not identical overlap. However, the Inktomi database have begun to look more similar than in the past. On one of these searches, all the Inktomi partners found exactly the same hits. On two others, they almost found identical hits (off by one). However, on the largest of the five searches, there were eight pages found by only one of these six Inktomi databases. Five were found on Anzwers and three by MSN Web Search.

Search Engines Analyzed:

• Fast
• Northern Light
• AltaVista
• Google!
• Excite
• Infoseek
• Lycos
• WebCrawler
• And the Inktomi crew:
  • MSN Search
  • iWon
  • Yahoo!
  • AOL
  • Anzwers
  • HotBot

See the more detailed analysis of unique hits to gain a sense of how the 110 pages found by only one search engine were distributed.

Previous Comparisons:

• Sept. 1999: Five searches on thirteen search engines. 326 hits, 140 unique pages. 66 found by only one search engine.
• May 1999: Five searches on eleven search engines. 267 hits, 122 unique pages. Over half found by only one search engine.
• March 1999: Four searches on ten search engines. 202 hits, 97 unique pages. None found by more than five search engines.
• Jan. 1999: Four searches on ten search engines. 176 hits, 83 unique pages. None found by more than six search engines.
• August 1998: Four searches on five search engines. 103 hits, 70 unique pages. None found by all five search engines.
• May 1998: Four searches on five search engines. 95 hits, 77 unique pages. None found by all five.
• Feb. 1998: Four searches on five search engines. 103 hits, 62 unique pages. Three found by all five search engines.
• October 1997: Four different searches on four search engines: 220 hits, 12 found by all four
• September 1997 and June 1997 found no pages in common among four small searches on the four largest search engines at those times. (No charts available.)

While decisions about which Web search engine to use should not be based on size alone, this information is especially important when looking for very specific keywords, phrases, and areas of specialized interest. See also the following statistical analyses:

• Relative Size of the search engines
• Total Size Estimate
• Unique Hits hits from each
• Dead Links comparison
• Change Over Time