Basically: If you're fine with every program running on the system (including web browser in your case) having full, unfettered access to everything else on the system, then it's fine to disable the fixes.
In other words: Only do this on systems where you actually trust each running program not to be compromised in its day-to-day operations and turn against you. Anything that runs arbitrary code from an outside source (for example JS) is not safe.
And yet, with all of the pearl-clutching in this thread and countless others just like it, nobody seems to be able to point to any real-world exploits.
The threat is purely theoretical. The loss in performance is not.
As tsimionescu points out, it's only read access. But to go further, in most Unix systems, the default is already to give most users read access to almost everything: default masks in the filesystem tend to be 755, and when it comes to inspecting data from other users, there's an awful lot you can figure out by default. Leaving aside the fact that many home computers are single-user in practice anyways.
The JavaScript of shady adverts that sometimes pop through can also occur on no. Shady websites. So you are not entirely safe by only browsing safe sites.
That doesn't answer the question, of course; it's just a sales pitch for the proverbial tiger-proof rock. To reiterate: how long does s/he have to spend on a shady site before a successful SPECTRE exploit takes place?
If this were actually happening in the real world, maybe we'd know.
There are ready-made Spectre exploits that will attack your browser if it hasn’t been hardened yet; these kinds of exploits are fairly straightforward. Spectre v2 is harder to pull off, but can reach across processes and so you’re presumably vulnerable to that.
AFAIK, I (my personal workstation) would only be exposed via browser JS so if I do not spend too much time on shady sites, I should be good?