I'm not sure how often it's updated. It seems fairly complete from other sources I've been looking at (mainly robots.txt files for well-known publishers).