RealTime IT News

Private Search on Public Portal

Citizens trying to find information on the welter of federal and state government Web sites could soon have an easier time of it.

A deal with search technology provider Vivisimo and its partner, MSN Search, will bring those companies' algorithms to bear on the U.S. Government's official Web portal, FirstGov.gov, the U.S. General Services Administration (GSA) announced Thursday.

"Search on the site was working well, but we completed that contract in 2002," said MJ Pizzella, GSA associate administrator for Citizen Services and Communications. "Life has changed a lot."

The idea was to leverage the huge commercial Web indexes out there, add the government-specific documents, then customize the results to suit the needs of people searching for government-specific info.

The yearlong project will triple the amount of information available via search, as it will have added approximately 16 million pages by including local, tribal and territorial government Web sites to the mix. Phase one is expected to go live next spring, Pizzella said.

In a later iteration planned for next fall, FirstGov.gov will add searchable news and images to its content offerings.

For example, federal and state agencies have image libraries covering Hurricane Katrina. Today, citizens must search agency-by-agency. Vivisimo metasearch will do that for them, delivering results from the various governmental agencies, then ranking them according to relevance.

The portal has become the preferred method of finding government information among both government employees and the public. Last year, USA Services, which delivers information and services to citizens through Internet, e-mail, telephone and paper pamphlets, reported 242 million public contacts. Of those, 202 million were via FirstGov.gov.

Vivisimo CEO Raul Valdez-Perez said there were a few issues the GSA wanted to address. One was that crawling and indexing only pages on government servers misses valuable information for ranking results, such as outside linking.

Second, it wanted to include metasearch and clustering for improved usability. Finally, Vivisimo will institute "shortcuts." For example, typing a ZIP code in the query box will bring up information most pertinent to that area, including state and national representatives.

Vivisimo will supply selective crawling of some government sites, metasearch and selective metasearch, clustering of results according to topic, and the user interface. MSN will filter the results and apply its relevance algorithms.

Vivisimo's clustering and metasearch technologies, known as Clusty, will use artificial intelligence to automatically select which domains to search. Clusty metasearch will bridge government databases and Web results delivered by MSN Search.

"For example, if a user query is related to a job search, and if you can figure that out, you would selectively search usajobs.gov and bring in metasearch results from that," Valdez-Perez said.

Vivisimo doesn't crawl and index the Web; instead it partners with other companies and applies its clustering and relevance algorithms to their indexes.

"We needed to partner, so I telephoned Microsoft with an offer to partner," Valdes-Perez said. "They were very pleasant -- eager to serve the government's needs and quite flexible in how MSN search results would be used. They had no problem blending MSN results with results from calling a government search engine or from our own metasearch."

Justin Osmer, MSN search product manager, said that the partnership was a little different for MSN Search, adding, "We're just getting started in the field." MSN launched the first beta of its own search technology in November 2004; it now powers the main site.

"Down the road, you'll see us working with a variety of different technology providers to extend the reach of our technology and get more distribution and syndication deals out there," Osmer said.

The partnership with Vivisimo is the second in MSN Search's short life; in August, it began contributing search results to Dogpile.com's metasearch service.

The contract includes two special search services for kids and Spanish speakers, two important audiences that weren't served well. FirstGov for Kids on Kids.gov will be given a more robust "kids only" search engine focused on child-appropriate content, while Espanol.gov will offer the same search capabilities as FirstGov.gov, but in Spanish.

FirstGov said the new search strategy will be cheaper, as well. The Vivisimo/MSN contract is for $1.8 million annually, less than half the cost of the current search service on FirstGov.gov.

The contract is the first of several that may be awarded under a blanket purchase agreement that runs through 2005. Gigablast and Fast Search & Transfer, the former provider of FirstGov.gov search, also are on the approved vendor list, which the GSA has set up to acquire Web search services for the portal; the GSA could spend up to $18 million by the end of the purchasing period.

"This allows us a lot of flexibility," Pizzella said. "We're finding that with technology, you need as much flexibility as you can get since life changes so quickly. If we add new services, we're not locked in. We can move quicker."

This is something of a homecoming for the tiny search provider. Vivisimo is unusual in that its early funding came from the National Science Institute rather than venture capital. "We think it's a nice payback," said Valdes-Perez."