Power Performance Area. Area is how much wafer area each chip and the features within a chip use. Apple optimizes for the first two and sacrifices the third. Because they mark up their products a lot, are vertically integrated, and they move millions of devices they can sink a lot of $ into bigger chips made on bleeding edge lithography. More area means bigger caches, higher instruction level parallelism. Hiring lots of top chip designers gets better results too.
They're only large compared to other ARM cores, and Cortex-X4 is pretty close nowadays. AMD and especially Intel have a significantly larger per-core area budget; they have to in order to clock past 5 GHz.
