My suggestion is to get rid of down-voting. Griefers that want to abuse the system would need to do a lot more work if their only tool was up-voting. Accentuate the positive!
This also reduces the problem of people down-voting posts simply because they disagree with an opinion.
If the majority of a site has $opinion_x, chances are people with $opinion_y will earn jerk points every time they express themselves.