Maybe I'm stating the obvious, but you write that:
So I scale back up to 100 follower requests for the next call. It goes through. Next 100, fails. I scale down again …
So, that sounds like a cache is being primed on the first request. If this is a consistent pattern, you should be able to issue a request for one record, followed by a request for 100 and get it through most of the time (E.g. unless you run into a garbage collection cycle). If you code against this assumption, you should be able to utilise 50% of the theoretical limit of a 100/hour.
sometimes, it's follows a clear pattern. request 100, fail. request same 100, success. repeat. --- that's likely because they load it in the cache once they have the results. it halves my success rate, basically.
but sometimes it's just random (depending on twitter overall traffic I'd assume).
plus, some records are faulty and can not be fetched. so this causes other issues as well and drops the api call.
So I scale back up to 100 follower requests for the next call. It goes through. Next 100, fails. I scale down again …
So, that sounds like a cache is being primed on the first request. If this is a consistent pattern, you should be able to issue a request for one record, followed by a request for 100 and get it through most of the time (E.g. unless you run into a garbage collection cycle). If you code against this assumption, you should be able to utilise 50% of the theoretical limit of a 100/hour.
Is there something I'm missing?