One problem with archive.org is that it retroactively obeys the robots.txt file, even if the files have been spidered and archived [0].
For example, consider the case when a new domain owner attempts to block all bots from spidering their site, by adding something like this to their robots.txt file
User-agent: *
Disallow: /
This is actually a fairly common case when domain resellers purchase expired domains.
Now when you try to visit the archived link, because the live robots.txt file disallows bots, you won't be able to access the archived site (which may have been owned by someone completely different).
This is true, however they don't purge any of the old archives. If the robots.txt is updated to allow bots again then all of the previous archives will become available again.
it seems weird to me to retroactively obey robots.txt for a _different_ version of the site.
It makes much more sense to archive the robots.txt along with the content, and use the robots.txt linked with that version of the site. Updating the current robots.txt shouldn't affect a past archive.
I believe the reasoning is that if I accidentally published a public page and didn't realize it but later fixed the problem, I'd like to make sure no one else could see it later. It's probably easier than dealing with all kinds of takedown requests.
For example, consider the case when a new domain owner attempts to block all bots from spidering their site, by adding something like this to their robots.txt file
This is actually a fairly common case when domain resellers purchase expired domains.Now when you try to visit the archived link, because the live robots.txt file disallows bots, you won't be able to access the archived site (which may have been owned by someone completely different).
[0] https://archive.org/post/406632/why-does-the-wayback-machine...