Simple. Don't have contribution agreements. Contributors maintain their copyright so as they can prevent relicensing. Mind, FSF requires copyright for their submissions (I believe), but they're, arguably, a "good actor" in this space.

But, if the code base becomes a patchwork of contributors, it can become difficult to relicense.