> I feel the blame mostly falls on Anthropic not being considerate when they scrape other people's content.
That's certainly a fair assumption. I haven't looked into it in any detail. Is IFixit sending appropriate cache headers or is Anthropic just ignoring them? I don't know. But at some point, your caching strategy and your ability to respond to spikes in demand is your own problem. We live in reality, and in reality you can't really control who is hitting your website when. Sure, it would be nice if they respected your cache control policies, but what are you doing when you know they won't? I'll refer back to my previous statement. It's totally understandable when someone who didn't expect ever HN levels of attentions to be overwhelmed. But if you're a site as large and respected as IFixit, you've got to do better than the "unwashed masses".
That's certainly a fair assumption. I haven't looked into it in any detail. Is IFixit sending appropriate cache headers or is Anthropic just ignoring them? I don't know. But at some point, your caching strategy and your ability to respond to spikes in demand is your own problem. We live in reality, and in reality you can't really control who is hitting your website when. Sure, it would be nice if they respected your cache control policies, but what are you doing when you know they won't? I'll refer back to my previous statement. It's totally understandable when someone who didn't expect ever HN levels of attentions to be overwhelmed. But if you're a site as large and respected as IFixit, you've got to do better than the "unwashed masses".