My HCI lecturer at uni used to call that our "monkey reflex". We have some really primitive reflexes around detecting movement in our peripheral vision that happen waaay before our conscious brain gets involved. I think some of them happen within the first few layers of visual processing.

You basically can't ever fully ignore movement, especially if it is irregular (so not a short loop), a part of your brain is always shouting about it, although your brain will tune out highly regular motion like a swinging pendulum.