> I can consistently reproduce this by asking Claude which code snippet it prefers: the one it generated in a different chat, or one that I refactored for my own needs and find more useful. It always picks its own :)
A better question to ask for each snippet is "Estimate the seniority and competence of the developer who wrote the following code, ignoring bugs that linters or LLMs can catch and focus only on structure, maintainability, logical layout and readability."
It almost always estimates the author of my code as above the author of it's own code.
> You're asking basically to ignore bugs and correctness.
Not ignore correctness, just bugs that will be caught by tooling.
> Can it be a useful comparison?
IME, yes. LLMs in an agent-loop are trivially able to write spaghetti code that will never do an off-by-one error or something else that is easily caught by tooling, which is not something humans can do.
Judging code on whether it has bugs easily caught by tooling is pointless - LLMs are running the tooling in a loop anyway, so no matter how bad or poor their code actually is, it never exhibits bugs that are caught by tooling.
> Bugs are bugs, if the instruction is "ignore bugs except for those that can be caught by you" then the instruction is basically "ignore bugs".
No, the instruction is "ignore bugs that can be caught by tooling", unless you are seriously complaining that missing a semi-colon should register the developer as a junior?
> And it implies "ignore correctness" because when program is incorrect we usually refer to it as a... you guessed it, "bug".
This ("Bugs are bugs" sentiment) is digressing from my original point, but I have some time to engage, so...
Now, this is a take (one that I used to hold, once upon a time), but it is incorrect.
There is no definite "correct" and "incorrect" states in non-trivial applications, because every non-trivial application has unspecified requirements that are understood by most parties involved (customer and developer) whilst not being written down anywhere.
For example, the "save file" specification for a cross-platform application does not specify the allowed/disallowed characters in a filename. The understanding by both the client and the dev is that the filename can be whatever the underlying OS and filesystem allows it to be but this is not written in the spec!
Is this a bug?
If the user saves a file to a filename with some odd characters in the name, then moves it to portable storage that truncates the filename/removes emojis/whatever, then attempts to upload it back to the system, the system can refuse because the metadata inside the file does not match the filename.
User is going to report it as a bug! The developer is going to reject it as a bug (there is no error in the code).
Sure, contrived example, but Line of Business applications have thousands of these unspecified but common-sense requirements baked in.
I'm looking at my employers triaging system right now, and even though this is a high-level business app (written mostly in SQL and C#), there is one category for bug (e.g. specific field not saved on form submission - defect in code), and another for deficiency (e.g. form field 'total' does not subtract non-tax costs - ambiguity in spec). The reason this is important is because clients aren't billed for bug fixes, but they are billed for disambiguating a spec + writing code.
Both those things were reported by the client as a "bug".
The reality is that we aren't dealing with what is "implied", only with what is there. There are defects in code and defects in specs. The code ones are the easy ones.
A better question to ask for each snippet is "Estimate the seniority and competence of the developer who wrote the following code, ignoring bugs that linters or LLMs can catch and focus only on structure, maintainability, logical layout and readability."
It almost always estimates the author of my code as above the author of it's own code.