What bothers me is this: Claude & I work hard on a subtle issue; eventually (often after wiping Claude's memory clean and trying again) we collectively come to a solution that works.
But the insights gleaned from that battle are (for Claude) lost forever as soon as I start on a new task.
The way LLM's (fail to) handle memory and in-situ learning (beyond prompt engineering and working within the context window) is just clearly deficient compared to how human minds work.
And the thing is all these “memory features” don’t help either because the “memory” is too specific either to the task at hand and not generalizable to all things, or it is time bound and therefore won’t be useful later (eg: “user is searching for a new waterbed with flow master manifolds”). And rarely can you directly edit the memory so you are stuck with a bunch of potential nonsense polluting your context (with little control when or why the memory is presented).
The reason these tools haven't achieved greatness yet is because 99% of us are struggling at work with domain knowledge - how does this special project work in the frame of this company. If an AI tool is unable to "learn the ropes" at a specific company over time, they will never be better than a mid-senior developer on day 1 at the company. They NEED to be able to learn. They NEED to be able have long-term memory and to read entire codebases.
Yes, it's a common problem. There are 'memory' plugins that you can use to collect insights and feed it back to the LLM, but I tend just to update an AGENTS.md file (or equivalent).
But the insights gleaned from that battle are (for Claude) lost forever as soon as I start on a new task.
The way LLM's (fail to) handle memory and in-situ learning (beyond prompt engineering and working within the context window) is just clearly deficient compared to how human minds work.