RealTime IT News

Wikia Goes The Way of Open Source Search

Look out, Google! Jimmy Wales is coming for you. The founder of Wikipedia is aggressively building out his Wikia effort, a commercial collection of Web sites anyone can edit. One of its key parts? Search.

Today Wikia announced the acquisition of the Grub distributed search project from LookSmart and released the technology under the GPL version 2 open source license. Financial terms were not disclosed.

Much like how Wikipedia and Wikia open the process of content contribution to the masses, Wikia Search will open up search to the masses.

Wales hired Jabber founder Jeremy Miller to lead the Wikia search project, which launched in December. One of the first things they did, Wales said, was look for technologies to help jump start the process.

"The interesting thing about Grub is that it's a distributed crawler," Wales explained. "So people download the client and the client does the crawl under the direction of a central server. It's very robust; they [Looksmart] used it in production for several years."

Though Wikia now has Grub, Wales was quick to note that it doesn't preclude it from using other tools, such as Nutch, which is an open source search technology built on top of Apache Lucene.

The Grub client is currently available for people to download and test. Beyond that Wales noted that it's too early to talk about how it will roll out.

"If we get a fair number of people using the client and are able to do a reasonable number of crawls by the end of the year, I'll be satisfied," Wales said.

Ultimately though the larger goal for Wales and his Wikia Search is nothing short of challenging the search giants themselves.

"Basically we're trying to build the complete search engine infrastructure, the whole package and Grub is one of the pieces of the puzzle," Wales said confidently. "Once we're done we do anticipate competing directly with Google and Yahoo."