1 million domains ranked by Alexa: http://s3.amazonaws.com/alexa-static/top-1m.c...

		anc84 on May 18, 2015 \| parent \| context \| favorite \| on: What one may find in robots.txt 1 million domains ranked by Alexa: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip For more try https://commoncrawl.org/