GDP going up does not absolutely improve people's quality of life (and there's quite a lot that can make GDP go up while making every participant in the economy miserable.) That's why optimizing for it makes no sense.
Yes, we can create a hypothetical world where that is true, but that's not the world we live in.
GDP is not just an abstract number. It's a measure of the value produced by the nation - value that people find, well, valuable.
We do not optimize only for GDP. It is merely a leading indicator we use that helps us predict things like recessions and unemployment which themselves are leading indicators for quality of life.
More money is strictly better; if you think rich countries are "unpleasant or unpractical" to live in, try living in a poor one.
You're right that GDP is far from the only thing that matters. But sacrificing growth for intangible benefits is a tradeoff that should be made very carefully indeed.
Money is just a token. The valuable thing is the actual output of products and services.
Companies are great at this, and taking resources away from them means they have to crank out less of it. The best and simplest thing is to tax the salaries and dividends paid by the companies, and stop taxing consumption (regressive tax that's hardest on the poor) or profits (reduces investment and output).
And those effects are unpleasant and make the country more impractical to live in.