This already exists in the form of ‘if-modified-since’ http cache property. A requester can ask the server if the data they currently have is stale within the current spec.

In my experience, the problematic crawlers choose not to implement this feature.