>documented problem in Home Assistant this evening. All it would take is 3-5 lines of YAML. The model failed miserably. I think we're all still safe.
This is mostly because HA changes so frequently and the documentation is sparse. To get around this and increase my correction rate, I give it access to the source code of the same version I'm running. Then instructions in CLAUDE.md on where to find source and it must use source code.
For this issue, additional Media Player storage locations, the configuration is actually quite old.
It does showcase that LLMs don't truly "think" when it's not even able to search for and find the things mentioned. But, even then this configuration has been stable for years and the training data should have plenty of mentions.
It's not really magic, in my project folder I will git clone the source code of whatever I'm working on. I will put something in the the local md file like:
Use ./home-assistant/core for the source code of home assistant, its the same version that I'm running. Always search and reference the source when debugging a problem.
I also have it frequently do deep dives into source code on a particular problem and write a detailed md file so it only needs to do that once.
"Deep dive into this code, find everything you can find about automations and then write a detailed analysis doc with working examples and source code, use the source code."
This is mostly because HA changes so frequently and the documentation is sparse. To get around this and increase my correction rate, I give it access to the source code of the same version I'm running. Then instructions in CLAUDE.md on where to find source and it must use source code.
This fixes 99% of my issues.