San Francisco-based LookSmart has launching new “distributed crawling” technology its says has the potential to dramatically improve search.
The offering, dubbed Grub, is downloaded as a screensaver by a user or Webmaster from www.Grub.org. It taps idle processors on the user’s computer to crawl Web sites, including the users’ own.
Grub sends a compressed feed of changes back to the central registry for indexing. To minimize the impact of crawling Web masters may elect to only crawl content within their own LAN.
“About 1,000 people have downloaded the software,” said Grub founder Kord Campbell, who went to work for LookSmart after selling them the technology in January. “That lets us index anywhere from 100 million to 200 million pages per day.”
Given the mushrooming Web pages and other documents on the Web, Campbell says it will need tens of thousands of users to provide full coverage of the Web, something traditional search engines just can do with a finite number of servers.
LookSmart plans to introduce a Web application-programming interface
LookSmart, which also operates the search engine WiseNut, said the initiative helps Web masters reduce the bandwidth crunch caused by spiders indexing your entire site.
Distributed computing, also known as grid computing, has been popular in academic and research circles, aiming collective processing power at complex scientific equations.
Dr. Junghoo Cho, assistant professor of Computer Science at UCLA, believes the model will work for search.
“By sharing the task of crawling and updating the Web among many thousands of end users, distributed crawling may provide a solution to one of the main problems of current Internet search technologies,” Cho said.