Have you seen YaCy? It's a distributed search engine project that's been running for several years. You can start using it to search and crawl right now.

http://yacy.net/en/