Greg R. Notess Reference Librarian Montana State University

Web Wanderings Consumers' Revenge: Online Product Reviews and Ratings EContent, April 2000 Copyright © Online Inc.

Consumer evaluations have been an important information resource for years. Consumer Reports and similar publications have shown the demand for such comparisons, especially unbiased comparisons. Yet any market researcher will quickly point out the extreme difference in accuracy and quality between the evaluation techniques used by the Consumer's Union and that of most Internet-based product ratings and reviews.

First and oldest of all is the opinionated landscape of Usenet news.

Should we even consider Web reviews and ratings to be valuable resources? The Internet product evaluation sites certainly offer a large quantity of opinions and personal experience. They provide information on products, features, and services that may not be covered in the traditional evaluation sources. The value can be compared to asking friends and neighbors for their preferences and experiences. And the Web sites are instantaneous in

posting a review or rating. Understanding a bit of their background, biases, functions, methodologies, and practices can enable wise use and proper interpretation of the information they offer.

THE BEGINNINGS

From the early history of the Web, two examples demonstrate the trend toward the present state of product reviews. First and oldest of all is the opinionated landscape of Usenet news. Even before the widespread popularity of the Web, people were trading opinions, flames, facts, and fictions via Usenet about computer products, software, and all sorts of other topics. Usenet provided a way for people to express their opinions, relate negative and positive experiences, and provide feedback to others. Not only could users share their experiences and impressions of a particular product, but also they could help each other troubleshoot problems. This sharing of experiences worked well for software development and then expanded into many other areas. Usenet's common wisdom was first accumulated in Frequently Asked Questions files, then archived on CDs, and finally added to DejaNews, which brought the opinions and experience of Usenet participants to the Web.

The other early reviewer example is Amazon.com, an online bookstore with a difference. While business analysts can comment on the differences in business models, it is the consumer reviews that I will focus on here. Amazon provides an interactive feedback opportunity where shoppers, readers, and any other Web denizens can add their own reviews to specific products. Originally on Amazon, those products were books. Amazon even offered separate areas for publishers and authors to add comments. While books have a rich history of professional reviews in print, Amazon provided the opportunity for everyone to write a review and share their impressions of a book.

As other online commerce sites adopted Amazon's model and as DejaNews became Deja.com and added product reviews, the whole idea of consumer feedback, evaluation, and opinions took off. Another contributing factor has been the development of interactive quick polls. Many Web sites now offer these quick polls on a wide range of topics. Anyone visiting the site can cast their vote on the topic and instantly see it reflected in the results.

THE BIAS FACTOR

After submitting a rating, Deja sets a cookie that prevents that computer from adding another rating for the same product.

In general, one of the strengths of these Web reviews and ratings is that they bring a particular bias. Reviewers' biases are presumably determined by their own personal experience with the product. With almost all of the online reviews, polls, and evaluations, the participants are self-selected. Since it is certainly not a random sample of the population, and not even a random sample of the online population, the results should not be assumed to represent the views of the entire population, but only the views of the participants. The quick polls demonstrate this problem. For every 100 visitors to a Web site, perhaps only five choose to respond. Then, how many of those five repeat the poll from different computers, decide to answer less than honestly, are not even familiar with the product, or otherwise add an unbiased vote? How many may work for, invest in, or have been fired from the manufacturer's company? There is no way of knowing.

The Usenet and Amazon examples also demonstrate two different potential biases. In the realm of Usenet discussion, extreme opinions are more likely to be aired than moderate ones. In many cases, the vast majority of readers tend not to post, participate, or contribute their views. The Amazon approach has a different bias. After all, Amazon is in the business of selling. Positive reviews will sell more books than negative ones. But does that mean that Amazon's reviews have a positive bias? Or do authors and publishers ask friends to write positive reviews at Amazon? Certainly some reviewers have a stake in the reviews, but there is probably no foolproof way to always detect them.

Amazon and any other ecommerce outfit are likely to have a slight positive bias for another reason: liability due to slander or libel. No one is likely to sue over a dishonest or inaccurate positive evaluation. But given the current prominence of the Web, slanderous or libelous negative statements certainly have the potential for opening up the company to a lawsuit. To be fair, the bias factor often exists in traditional print resources as well. If a print publication supports itself primarily on advertising dollars, it has the same incentives toward positive reviews as does Amazon and other ecommerce sites. And the reader often has a similar difficulty determining what level of bias if any exists. While readers should be aware of the potential for bias, the reputation of all of these sites depends in part on the perception that the reviews are unbiased. For that reason, as will be seen later in this column, some sites have added a user-based rating of the reviewers themselves to help guard against any appearance of bias.

THE PLAYERS

Deja.com

Long a bastion of archived Usenet news postings, which are chock full of opinions, evaluations, reviews, problems, and fixes, Deja.com added a separate section for product reviews in 1999. It now claims over 1,000 categories, more than 40,000 products, and 900,000 reviews in that portion of its site. The product information contains both ratings and reviews. The product ratings are typically divided into several measures such as features, cost benefit, or safety, depending on the kind of product. The product ratings come from Deja visitors who chose to cast their votes in the online quick polls.

When you go to the Deja.com Web site, it is hard to avoid the constant requests to rate this or that product. Each visitor has an opportunity to take part in polls about various products. No registration is required to vote in the polls. After rating the specified categories for a product, each reviewer is also provided an opportunity to write a brief review. Deja has far more ratings than reviews, and the reviews are typically quite brief.

How do they prevent one individual from voting repeatedly on the same product? After submitting a rating, Deja sets a cookie that prevents that computer from adding another rating for the same product. However, deleting the cookie.txt file or switching to another browser or computer will still allow the same person to rate the same product again.

The reviews, though brief, can be informative, as long as they are seen as individual opinions and reports rather than representative or necessarily accurate. Take, for example, the reviews for the Fisher-Price Safe Embrace child car seat. One states that the "steel cage is so much better than plastic," while another claims that a "metal frame is stone-age technology and presents a danger to the child sitting next to it in the car during side-impact." The problem is that Deja. com has no editor to provide background, official safety reports, or expert opinions on these two conflicting reviews. The reviews are useful in pointing out a possible area of concern, but they are not very helpful in coming to a conclusion about the issue.

Be sure to look in the regular Deja Usenet archives (or messages as they are now known on the site), since you may well find more detailed information there. Yet even there, most of the information is only going to be anecdotal, consisting of individual opinions. The savvy consumer will often need to look beyond Deja.com to find authoritative comparisons and reviews.

Epinions

An interesting twist on the Deja. com approach can be seen at the Epinions.com Web site. At Epinions, members write and submit reviews. Then other members rate the reviewers. The best-rated reviewers get top billing as Trusted Members. In addition, reviewers receive Eroyalties for every member who reads one of their reviews. The idea is that by having a self-evaluating community, the best reviewers will rise to the top and that the community will be able to determine who are the experts in an area.

Epinions does have ratings, but the emphasis is on the reviews. They are much lengthier than the product reviews on Deja.com. In addition to the member reviews and the ability for members to rate other members' reviews, Epinions links to other sources of information. Some pages include the manufacturer's product information. Others include links to "professional reviews" which can be articles from media Web sites or subject-specific sites. Epinions still calls itself a "preview," at least as of the beginning of the year. The idea of reviewing the reviewers brings some semblance of balance to the reviews, and the length of the reviews provides a more detailed level of information than that available from the Deja.com product evaluation section. Even so, it still consists of self-selected reviewers expressing personal opinions.

RateItAll, ConsumerReview, and More

Moving back toward the review side of the evaluation business, there are the twin sites: ConsumerReview. com and ComputingReview.com. While these reviews may average a bit less in length than Epinions' reviews, they make up for this in their structured approach. The reviews all use a similar template with areas for strengths, weaknesses, and summary comments. The reviews on ConsumerReview. com also note the date of the review, the reviewer's self-reported length of experience using the product, a value rating, and an overall rating. Those on ComputingReview.com offer information on similar products used, a manufacturer's customer service, and ratings of value and performance. ComputingReview.com also has Web-based discussion forums designed to let consumers assist each other with questions and answers about specific products.

Yet another approach can be seen at BizRate.com. This site focuses on evaluating Web stores and providing online shoppers an opportunity to rate those stores. BizRate serves up their surveys after a visitor has bought something at one of their listed sites. BizRate members also can receive rebates for shopping at the listed stores. Since BizRate's own reviewers evaluate some online stores, their site offers a combination of user reviews and professional reviews.

Many other review and rating sites are available on the Web, and it seems to be yet another area of future growth. Generic subject sites and company Web sites are adding sections for ratings and reviews. Business-to-business sites are picking up the practice in the wholesale market. Reviews are becoming an increasing part of ecommerce at many levels.

REVIEWING THE REVIEW AND RATING SITES

How is the information professional to interpret such reviews and ratings? First of all, the ratings and scores that come from quick polls have only very limited value. They often have only a very small number of participants per product. What is more important, they do not represent a random sample since the participants are self-selected and can provide input several times. Thus, they are not statistically accurate. Do not interpret the results as representative of the general public. Instead, they can only be interpreted as the opinions of that small group, once again comparable to taking a quick informal poll of friends and family. On the other hand, the reviews and other evaluative comments on many of these sites can be quite valuable. Once again, they may only rarely represent the general population's views.

Yet the reports of an individual's experiences, difficulties, and successes can provide a level of detail not available in traditional consumer studies. They give a sense of real world applications. However, they should also be read as such. Is a user's detailed report of a frustrating attempt to install a software program simply a view of the user's incompetence or is it a general failing with the software? It could be either. In general, these consumers' revenge sites excel at offering a starting point for exploring issues with a specific product or service. They can substitute for asking the opinions of friends and coworkers, especially when no one in your circle of acquaintances has any experience with the item. They can alert a company to product problems and potential negative public relations. But in no way do they substitute for rigorous market research or full-fledged evaluative product comparisons.

Communications to the author should be addressed to Greg R. Notess, Montana State University Libraries, Bozeman, MT 59717-0332; 406/994-6563; greg@notess.com ; http://www.notess.com.