It seems to be a corruption of an older phrase comparing apples to oysters. Citation from 1670:

https://archive.org/details/bim_early-english-books-1641-170...

Apples and oysters are still things you can eat, but they're at least further apart than two fruits. The book also cites the similar expression "chalk and cheese".