As others have pointed out, if they’re in a stable orbit they won’t fall in (at least not for a very long time), because unlike luminous matter it won’t experience any force other than gravity. If you imagine a dust cloud of luminous matter around a black hole, it will tend to experience frictional heating the closer it gets, there is the chance of a collision or radioactive decay, and other forces acting to draw it in or send it far away. Dark matter won’t do that, it just couples to gravity. Our usual intuition about how a halo of matter behaves has a lot to do with interactions other than gravity. Clumping for example, aggregation and accretion pretty much work because of interactions other than gravity, until a body becomes massive enough.