Dogpile: Search Engines Don’t Have Much in Common

Dogpile.com, owned by Infospace.com, announced it added results from MSN Search to its meta-search service.

Users now can select from links to search results from the top four search players: Google , Yahoo , Ask Jeeves and Microsoft’s MSN.

It backed up the announcement with the results of a study by researchers at the University of Pittsburgh and Pennsylvania State University showing a surprising lack of duplication in the top results of the major search engines.

When the researchers ran 12,570 different queries through search engines at Yahoo, Google, MSN and Ask Jeeves, they found that only 1.1 percent of the results appeared on all four engines, while 84.9 percent of the top results were unique to one engine. Only 2.6 percent of the results were shared by three search providers, and 11.4 percent were delivered by two search engines.

The researchers examined both paid and natural search results, but they tabulated only the results on the first page.

“The top four search engines are very viable sources of information,” said Brian Bowman, Infospace vice president of marketing and product management. “But they’re vastly different on page one. And most people never go beyond page one.”

Breaking those stats down by search engine, Google had the lowest percentage of unique results, at 66.4 percent. Yahoo, MSN and AskJeeves all were within 3.1 percent of each other, with Yahoo having the highest percentage of unique results at 71.2 percent.

However, only 7.0 percent of the top non-sponsored search results for any one query were the same across all four search engines. The top four completely disagreed on the top five non-sponsored results 19.2 percent of the time.

Because of this lack of overlap, the researchers wrote, “Searching only one Web search engine may impede [the] ability to find what is desired.”

The theory of meta-search, according to the study, is to “mitigate the innate differences of single-source search engines, thereby providing Web searchers with the best search results from the Web’s best search engines.” The result should be the best results in response to any particular query.

Bowman said Dogpile.com weights the different search engine results based on consumer clickthroughs in the past. The service has agreements in place with all the search engines it relies on. Its own algorithms also look at the overlap of results. “If there’s overlap in search results, we weight those more heavily. That’s a highly relevant result,” Bowman said.

Searchers can see the differences themselves on Dogpile.com. Combined paid and organic search results from all four search engines are intermingled on the first page of results. Users can click one or more buttons to see the results of a search of only that provider’s index; results unique to that providers are highlighted.

The search providers are willing to work with Dogpile.com, Bowman said, because it acts as a distribution partner for their search and pay-per-click advertising. Dogpile.com meta-search is distributed throughout the Infospace.com network of Internet white pages, yellow pages and directory searching. Dogpile.com gets a revenue share when users click on the search providers’ paid listings.

“We have a lot of very valuable traffic, and we throw off great cash to them,” he said. “Besides, if one pulled out, it would give a great competitive advantage to the others.”

The study turned up a similar lack of overlap in sponsored links. Yahoo and Google returned the same ads for a given query only 4.7 percent of the time. Google delivered no ad while Yahoo showed an ad in response to a query 15 percent of the time. In the reverse case, Yahoo did not return a sponsored link where Google did 14.5 percent of the time.

Dogpile.com also worked with Web traffic analytics firm comScore Media Metrix to determine searchers’ success, based on the number of times users clicked on a link. ComScore found that only slightly more than half of all Web searches resulted in clicks on a link on the first page. Dogpile.com did better, garnering first-page clicks 63 percent of the time.