The Network For IT Professionals
|
www.PlanetIT.com The Network For IT Professionals |
Search Engine Showdown by Greg R. Notess (07/12/99; 9:00 a.m. ET) URL: http://www.PlanetIT.com/docs/PIT19990714S0018 On the Web, search engines seem to be everywhere. The majority of the Web public use search engines and portals to find information at least weekly, if not daily. Webmasters check for how well their site is indexed, and they strive to find ever-better ways to increase their rankings or at least to be sure most of their site is included. IT managers can offer intranet- and extranet-specific search engines. The availability of free search engines that index words from millions of Web pages has been one of the driving forces of the Web. As with many Internet companies, the search engine industry seems to be in constant turmoil. Change and growth are the twin mantras, and no one knows for sure in which direction they are heading. Yet recent trends point out some interesting problems and potential among the search engines and portals.
Portals To Last? Certainly one trend among search engines has been to embrace the portal concept, and its successor, the destination/media site. The recipe has been fairly straightforward. Take one search engine or directory. Add a generous helping of supplemental databases such as e-mail lookup, white pages, yellow pages, maps, dictionaries, etc. Stir in personal services such as free e-mail, calendars, customized stock quotes, news, sports scores and horoscopes. Sprinkle some headlines and weather on the top page. Fold in more stickiness with chat and virtual communities. Top it off with free Web page space. It's a recipe designed to attract a Web audience and keep them. The larger the audience and the longer they stay, the greater the ad revenue possibilities. Nowadays, less than half of Yahoo visitors even use the search or directory functions. Can the search function be found? Are Lycos, Infoseek, and Excite still interested in their search engines? The companies all definitely state that search continues to remain an important part of their services. Yet, while the search engine was the carrot that first attracted Web wanderers to their sites, it is clear that the search engines are no longer the only focus. As it becomes a less significant portion of the overall media site that the portal is becoming, is the recipe perhaps half-baked? For all the chat and free e-mail users, and people searching for popular words such as cars, movies, or sports, there are the times a user needs to find a small company's site or a widget specification or an unusual snippet of JavaScript. For these kinds of specific questions, a large search engine is still needed. But could these engines be dying?
Merger Mania Another threat to the survival of the search engine comes from the rapid rate of change in the ownership, partnership and mergers in the search engine industry. Except for the Open Text Index, few search engines have folded. But many have new owners. Excite bought WebCrawler and Magellan years ago. WebCrawler is still going strong, but Magellan appears to have had less active development. Lycos acquired A2Z, Point, HotBot, and many other services. A2Z is gone. Point's Top 5% Sites still exists but is barely promoted. HotBot continues development and seems well-positioned to continue. Yet now CMGI, a major stockholder of Lycos, has bought a majority stake in AltaVista, which had moved from Digital Equipment to Compaq previously. Excite is with At Home. Snap merged with Xoom.com, and both are now part of NBC Internet. Infoseek has become a part of Walt Disney's huge Go Network. Many of these movements and mergers have brought new content or services to feed the portal emphasis. Yet as the search engine becomes a lesser part of an overall online media behemoth, the time could come when the search engine portion could be discarded, especially if it is seen as unprofitable or little-used. Admittedly, that is a remote possibility now, but with the rapid rates of change these days, especially in Internet companies, be prepared for anything.
Newcomers As the well-known search engines mutate into media and Web service companies, that leaves room for others to focus more on Internet searching. Newer search engines such as Google, Northern Light and Fast Search at www.alltheweb.com have little of the extra portal content of the more established search engines. Rather than building more content and more stickiness to keep people at their sites, the search specialists can focus on growing their databases and adding new search features. These sites build traffic by attracting searchers seeking a cleaner interface and a straight focus on searching. In addition, as people become aware of the unindexed portion of the Web and the lack of overlap between search engines, the search engine that offers more content or pages not indexed elsewhere should become even more popular. Sensing this trend, other companies may develop new competing search engines, better relevance ranking, more advanced searching features or other ways to differentiate themselves from the rest. Although the market may seem crowded even now, everyone is still seeking a faster, more relevant, larger or better focused search engine.
New Directions For Web site managers, especially of large sites, one of the frustrating aspects of search engines is the incomplete indexing of their sites. This can be especially frustrating when one day AltaVista may drop thousands of pages that were formerly indexed and the next day HotBot loses several hundred different ones. None of the search engines guarantee that they index every page on a Web site. One approach that's unpopular with users is that of paid placement in a search engine. GoTo uses this model, as does AltaVista to a limited degree. The paid listings are identified separately from the regular listings. AltaVista even uses a box to identify them. However, that approach is only being used at the site level and in conjunction with searches for specific words or phrases. Yahoo has a paid express consideration service. Other paid options could include payment for the indexing of, say, more than 5,000 pages per site. Web sites with constantly refreshed content could pay for more frequent crawling of their sites. Payment for inclusion in search engine databases is one possible direction. An alternative or complementary approach is to employ some type of cooperative indexing. For a large site, it just takes too long for the general search engines to index all the Web pages on the site. That is where a local Webmaster can set up an internal search engine that can be controlled locally to control which pages get indexed and how frequently. Eventually, search engines may even develop ways that locally indexed content can be imported into their general databases.
Open Challenge Beyond even the cooperative indexing solution lies the potential for open source solutions. Since the dawn of Netscape, Yahoo has been the more prominent Web directory. Looksmart and the directory portions of Lycos, Excite and Infoseek have all had their share of traffic but have not caught up with Yahoo. Enter NewHoo, a small volunteer effort at creating an alternative directory. Eventually bought out by Netscape and renamed the Open Directory, it continued to be developed as an open source directory. Now, with more than 13,000 volunteer editors and more than 715,000 sites divided among 100,000-plus categories, it is a directory to be reckoned with. Lycos dumped its own directory and now uses a modified Open Directory for both Lycos and HotBot. Netscape has moved its Netscape Search from an Excite-derived database to the Open Directory with follow-up searches on Google. With the big guns embracing an open source directory solution, perhaps similar endeavors will work for search engines as well. Demand for search engines will remain steady. As the portals, media sites, search specialists and newcomers all continue to mature, the future of search engines will unfold. The challenge for the IT professional is to keep up with the changes and to be sure that public Internet sites are indexed as fully as possible.
About The Author
|
 |
www.cmpnet.com The Technology Network |
Copyright 2000 CMP Media Inc.