I didn't see any hand-waving? It was a legitimate question and one which you offered no evidence for/against. Certainly, we know little to nothing about the total range of side-effects from accumulated plastic. Could be negligible, could cause major issues that we just don't understand yet.
It's not my job to offer evidence against every hypothetical claim of harm. There are infinite of those, and one of me.
It is my job (all scientists' jobs) to call out panic based on "science". More generally, the people who want you to believe that X hurts you (for whatever value of X) need to bring solid evidence, not just unproven hypotheses.