Yes, the drawback of unsafe is one single goof in just one unsafe block can blow your entire program wide open. The advantage is that your entire program isn't one gigantic unsafe block (like C).
Also, in my experience, the locality and unsafe api is better for testing purposes compared to unsafe language.
If I have an unsafe code that provides safe api with certain safety conditions.
1) I have a more ergonomic/precise/local contract to satisfy safety
2) Since this unsafe block is local, it is easier to set up its testing conditions for various scenarios. Otherwise, testing for bigger unsafe block (e.g. unsafe language) would also have to handle coupling between api from which ub originates and the rest of the code.
The magnitude matters.