Unless this theoretical market prevents individual investment, anyone who has opinions on any company being over- or under- valued would then be able to take advantage of the passive investors and make a lot of money buying and selling. And this would affect the price.
But then it's no longer your theoretical market that is 100% passively invested.
but even if you know a company was under valued, you should only invest in it if you think it'll be 'properly valued' in the future. for that to work, there needs to be a mechanism for price discovery on individual stocks. but if (nearly) nobody is buying individual stocks, how would it be possible?
and it's not even that the index fund is driving price discovery of the aggregate market because people using these funds just keep dumping money in on a schedule because they've internalized the message "time in market beats timing the market".
But then it's no longer your theoretical market that is 100% passively invested.