Yes, it is challenging to index and process billions of web pages on a regular basis. Our current architecture can be scaled to handle 500+ million web-pages. However, to increase the crawl frequency and introduce more services, we are building a distributed computing network - called Newton Network (newton.network). We hope this will give us enough processing power to power our ambitions ;)