I don’t see that distinction as arbitrary at all. It seems like, among no other good or obvious options, exactly the place to put the line between free speech and restricted action.
There is much “bad” free speech, but “bad” action is intolerable. The distinction between the results of bad speech and bad action are clear - even if it is unfortunately true that speech can inspire action.
It seems to me that there's an obvious alternative option. After all, if the reason we find the idea of free speech intuitively appealing is because we subscribe to some version of "speech can do no harm", why not just base everything upon the more fundamental harm principle ("one is free to V as long as V-ing causes no harm to others") which covers both the cases of speech and of act?
It's impossible to act in a world of limited resources without "harming" other people. The division between speech and action is practical under the circumstances.
There is much “bad” free speech, but “bad” action is intolerable. The distinction between the results of bad speech and bad action are clear - even if it is unfortunately true that speech can inspire action.