I think a lot of programmers still don't realize that this idiom comes from doing micro optimizations early.

Software does need to be architected for performance up front. If you aren't able to work on data with a lot of locality or have latency between fundamental operations, you won't get fast software until you address these issues.

If you are really starting from scratch you will barely understand the problem, but once you do, you can make sure your architecture will align with what you are doing. After that, optimization becomes much lower hanging fruit.