There was a WHO study published just last month[0] which did not find any dangers associated with microplastics in drinking water. One of the conclusions mirrors your reasoning:
"Although it is not possible to draw any firm conclusion on toxicity related to the physical hazard of plastic particles, particularly the nano size particles through drinking-water exposure, no reliable information suggests it is a concern. Humans have ingested microplastics and other particles in the environment for decades with no related indication of adverse health effects."
Sounds like PFAS (a byproduct from teflon production). Researchers struggle to find any humans who don't have it in their bloodstream, even in remote areas. It causes birth defects.
Fun Fact: the "compostable" paper bowls offered by many fast casual restaurants like Chipotle contain this chemical, and will leech into the soil if you try to compost them.
"The Devil We Know" is a pretty unnerving documentary on this subject.
"Although it is not possible to draw any firm conclusion on toxicity related to the physical hazard of plastic particles, particularly the nano size particles through drinking-water exposure, no reliable information suggests it is a concern. Humans have ingested microplastics and other particles in the environment for decades with no related indication of adverse health effects."
[0] https://apps.who.int/iris/bitstream/handle/10665/326499/9789...