The cost is coming down fast. You can get a $2000 desktop machine (AMD 395) that can run effectively chatGPT 3.5 levels of LLMs at over 100 tokens per second.