Ordinal Rankings were computed for each of the search engines on each test query. The highest ranked received a value of 8. The lowest received a value of 1. Search engines which failed the query received a value of 0. The highest achievable cumulative Ordinal Ranking for the 25 test queries is 8 X 25, or 200. No search engine tested achieved this score. It is clear from these results that by this measure of performance, based on a computed Absolute Ranking (see test methods), the three search engines AltaVista, HotBot and Infoseek are ranked almost coequally at the top of the engines tested. Surprising is the relatively poor performance of the Open Text Index search engine which fared considerably better in the 1996 rankings. Of note also is the relatively strong performance of the Excite! search engine which, although somewhat lower ranked than the top three by this measure, is clearly above the pack and close to the top three.

The figure above shows that the top ranked search engine by this measure is Infoseek. The Absolute Rankings from which these relative rankings derive were based on the total number of items retrieved, the location in the search output list of the first solid hit and the relevancy of the first 25 items listed in the search output. Clearly, strong performers are also AltaVista and HotBot, in the 90th percentile relative to Infoseek, and Excite!, in the mid 80th percentile relative to Infoseek. All the other search engines rank in the mid to high 40th percentile relative to Infoseek. Again, surprising is the relatively poor performance of the Open Text Index engine. Bear in mind that although these rankings are a good indication of the power of a search engine, they are no reflection of the ease of use of the search form interface. The nature of the interface is also an important criterion for selection of an appropriate search engine for end users. In this regard, the WebCrawler interface would win high honors for simplicity and appeal. Many of the other interfaces can be arcane to casual or novice users. If the user interface is difficult, the power of the search engine really doesn't matter. Trained researchers and intermediaries may wish to make power and flexibility the primary considerations in selecting a search engine as their default. Those with responsibility for public workstations with Internet access may wish to select engines with more intuitive interfaces for primary use and point-of-use training of casual and novice users.

The figures plotted above show the number of times each search engine was ranked in the top three; that is, received an Ordinal Ranking of 6 or higher, on each test query. The maximum achievable score is obviously 25. Infoseek is, again, the clear leader, having been ranked in the top three 17 times out of the 25 test queries. Close behind are AltaVista and the newcomer HotBot at 15 times. Also relatively strong compared to the lowest ranked engines are Excite!, and, perhaps not surprisingly, Yahoo! The Yahoo! search engine excelled on test queries which corresponded to one or more of the categories in its hierarchical subject directory structure. Disappointing is the poor performance of the Open Text Index relative to its 1996 ranking among the top three.

The Average Relevancy computed for the first 25 hits returned by the search engines on each test query is presented in the figure above. The search engines consistently ranked among the top four, Alta Vista, Excite!, HotBot and Infoseek, are the top ranked by this measure, as well. Interestingly, the performance of the HotBot engine slightly exceeds the consistently highest ranking Infoseek but these differences may not be statistically significant. What is more interesting is that the figures attained by the top ranked engines seem to cluster in the 60% to 70% relevancy range. Such a relevancy returns on almost any query would be considered high by trained search intermediaries.

The figure above shows the Average Relative Recall computed for each search engine over the 25 test queries. Bear in mind that recall is not necessarily a measure of good performance, especially with regard to the WWW search engines. Some consistently produce wildly inflated recall figures in the tens or even hundreds of thousands of items. This kind of return can be very daunting to novice and casual users and could be considered a search failure by some. Computing the average of relative recall within each query tends to even out such anomalies and might give greater credence to the figures reported above. Interestingly, the HotBot engine shows the highest recall. This relatively high recall linked with the demonstrated high relevancy among the first 25 hits makes HotBot a very strong search engine. These figures, alone, would merit a recommendation of HotBot as the search engine of choice for novice and casual users. Also at the high end in average recall is the Excite! engine. This is not unexpected since the Excite! engine uses fuzzy logic to interpret natural language queries. Again, a high ranking for relevancy makes Excite! another candidate for recommendation. All of the other search engines tested rank at the lower end with regard to recall. Remember that this is not necessarily a negative outcome. Low recall coupled with high relevancy can indicate strong performers. By these twin measures, and on the basis of their high relevancy, the Alta Vista and Infoseek engines would merit consideration for recommendation, as well.

BACK:   Results

NEXT:   Features

TOP:   Search Page