The main opposition to pedestrian streets in the U.S. tends to be from merchants. Their view is that cars driving by is good advertising. Even if the cars don't actually stop, when they drive by a store daily, they're reminded it's there and might eventually visit, while if the traffic gets routed in a different way, fewer eyeballs will see the stores daily (and instead they'll see different stores). State Street in Chicago was pedestrianized from 1979-1996, for example, but merchant pressure eventually caused it to be reopened to traffic.
It depends a lot on the circumstances. Here's a recent situation where the decision to make some streets in Chinatown pedestrian-only and removing parking on others (if only for a day) was supported and well-received by merchants:
