LookSmart Search

Building a Bigger Search Engine [Wired] talks about a company called LookSmart which is trying to use the spare computing power of voluntary users to help search the web. This is similar to SETI@home and distributed.net but for web search instead of extraterrestrial intelligence search or encryption key search. One thing I like about the idea is the potential ablity to re-index pages every day instead of every month. The article states that google only refreshes every 30 day, however I think some sites are refreshed more frequently (Obviously with google news some sites are refreshed very frequently). Anyway, another thing I like is the open approach LookSmart says it will be taking. It mentions making the results of indexing are avaliable to the users. I’m not sure about other aspects but I think combining an open source search engine with voluntary distributed computing could be very cool.

Update: [April 20, 2003 1:25 AM] Well now that I actually have it installed and running, I realized something that didn’t occur to me while reading/skimming the article. Grub (the distributed web search client) doesn’t really take up CPU cycles as much as it takes up bandwidth. I see this as a bit of a problem. Most people don’t use close to full CPU power, and giving space cycles to search for new life or to crack some encryption doesn’t cause a noticable differenct to the user. Bandwidth however is a completely different. I don’t think as many people have as much excess bandwidth that is basically avaliable for free. I’ve recently lived with bandwidth restritions at university and have heard others talking about ISP’s charging for more bandwidth. Maybe this isn’t the norm, but I don’t see an excess of free bandwidth.