I had a similar problem when using SSL connections. I'm not sure about your data by in my case the data was pretty much public data and there was no harm in using simple HTTP connection. This significantly improved the speed and no more very frequent timeouts.
Also try enabling/disabling gzip compression for API calls.
the bottleneck is twitter's api limits, data-wise and http connection wise I have headroom, lots of.
simple http connections to parse/spider the follower records from public pages is a no-go since twitter blocks the IP then, and scaling this out will eventually not end in a good way.
Also try enabling/disabling gzip compression for API calls.