While the conversion from points/pixels to em is not fixed, it is fixed for any particular font. As the pixel and point sizes change, so will the em - they're directly proportional except for any font hinting your browser might apply.
In other words: They are not a physical size measurement.
The definition of physical units is precisely that they are relative to only one unambiguous standard which relates it to all other dimensions of the universe. Distances, for example, are related to time. As such, one meter, as a unit of distance, is the distance that light in a vacuum travels in 1/299792458th of a second. A unit that does not have such a fixed relation to time is not a unit of distance by definition. So, unless increasing the font size in your browser also slows down time, the em is not a unit of physical size.
There are many more perfectly fine ways to specify how to derive a physical size relative to a given situation, but that does not make that specification a physical size itself.
OK, I'll grant your point. But Em is much more closely tied to the other physical units than intuition would suggest. It's one step removed rather than being completely independent.
