Yahoo Content Aggregation
Web Design & Technology News, March 2, 2004

March 2, 2004

Yahoo on Tuesday will begin a systematic effort to draw more content into its searchable database of Web documents, its latest bid to win Web surfers from search rival Google.
By Stefanie Olsen

The Web portal, based in Sunnyvale, California, said that it will introduce a "content aggregation program" (CAP) designed to index the billions of documents contained in public databases but that are commonly inaccessible to search engines, or what's called the invisible or deep Web. To this end, it has aligned with the Library of Congress, UCLA, National Public Radio, the University of Michigan and Project Gutenberg, among others, to begin seeding its index with fresh, searchable material for Web surfers' queries.

"We aim to extend the way we discover content for free," said Tim Cadogan, Yahoo's vice president of search. "One of the challenges is that the interaction between content providers and search engines is lacking. So we said, let's reach out to the public domains and nonprofits, and try to get more of that content exposed."

There's also a commercial component to CAP. For a fee, Yahoo will let marketers pay to regularly feed it Web addresses for speedier indexing in the database, a practice in the industry called paid inclusion. Yahoo has essentially repackaged the paid-inclusion programs of its three recent acquisitions--Inktomi, AltaVista and Fast Web search--and produced Site Match. Marketers will now be able to buy into one indexing program, instead of three, to be included in Yahoo search results as well as those of search partners like MSN.

The move comes only weeks after Yahoo dumped Google as its search provider and began using its own technology platform. Yahoo had been gearing up for the switch for the past year, after buying search technology company Inktomi and commercial listings provider Overture Services for nearly $2 billion in a bid to recapture the search crown from Google.

The company has a vested interest in winning back the loyalty of Web searchers it lost to Google in recent years; financial analysts expect that search engine advertising will generate nearly $4 billion this year.

The move also comes weeks after Google boasted the expansion of its own searchable index to more than 6 billion documents. Google said at the time that growing its database serves to improve the overall comprehensiveness of its search results.

Yahoo's Cadogan would not say how many documents it searches--only that its index reaches into several billion. He said that about 99% of the company's search results draw on documents it has obtained freely from public Web sites. The other 1% is the result of the paid-inclusion program.

The CAP program will help it increase the comprehensiveness of the index by tapping the estimated 10 billion to 100 billion documents of the deep Web, he said. In one partnership, Yahoo will receive feeds of more than 17,000 pieces of related audio content from NPR, the nationally syndicated news radio program. It will also receive fresh feeds from NPR so that it can index new content within one to two days.

Cadogan said that Web sites like eBay can also pay to participate in site match to ensure that an auction listing is current when a Web surfer searches on "BMW 2002 green," for example.

Site match costs marketers $49 annually to list the first Web address and $29 for the second Web address thereafter, up to 10. After 10 addresses, the site would pay $10 per listing. The marketer then pays between 15 cents and 30 cents when a Web surfer clicks on its listing in search results.

Still, some industry watchers have expressed concern that paid-inclusion programs can adversely influence search results and confuse Web surfers who are looking for unbiased results.

Cadogan maintained that there is an "iron wall" between commercial and free search results, governed by quality standards and ranking algorithms that ensure that results are relevant to users.

"Everything goes through a quality process, including an example in the free crawl, and every piece of content is handled exactly the same way for ranking."