I'm sorry if this makes me sound like a cranky old man but I find the idea of AI-generated PR description and reviews entirely misses the mark of what they're for.
The PR description should contain the context that made a change necessary, which is something that can not be contained in the change itself and thus can't reliably be inferred from it.
And if I request a PR review, I trust you, as my peer, to understand the problem my change is trying to solve and to judge whether it succeeds in doing so. If I wanted AI suggestions for it I can skip the middleman.
I guess the "chat with the codebase" feature could be useful, but the auto-review feature just gives me very uncanny vibes. If someone would "review" my code by blasting it through an LLM I'd be ticked off.
Having seen / used / etc similar things: yes I fully agree.
For people who already know the code well, it can catch things you didn't think about, and that's kinda nice. I suspect it has cost me/my team more time on wild goose chases than it has saved on real findings, but it might still be worth it.
For others, I've seen it constantly mislead people, and convince them to make changes that look reasonable but lead to the PR being incorrect. Over and over and over. And the "look reasonable" part is particularly dangerous during code reviews - reviewers are less careful about things that look reasonable, that's a relatively normal and unavoidable thing for humans even though it's incorrect.
When it's exposed to everyone, there's a pressure to follow and trust it, like existing linters. Private use is less bad, but still risky due to the plausible-change-review-negligence issue.
---
Also PR descriptions in particular are borderline useless when written from an LLM, because they just summarize the changes (or mislead summary-readers about the change, when incorrect). So you're just littering your commit history with the equivalent of auto-generated blogspam. No thanks.
We use this feature and it's actually been pretty handy. I still require my devs to write their own PR description (because of course they should), but it does give a few suggestions for things to look into as they review. Occasionally it's pointed out a few things I might have missed. But at least half of the suggestions are obviously off base, so I can't imagine trusting it to actually do the review.
In general, PR-Agent is designed to assist, not replace, human reviewers.
Many of the flows and processes we deployed are aimed at meeting this goal.
We try not to overclaim, or overpromise.
PR-Agent should not be the sole reviewer of a PR.
We believe it can definitely enhance and improve the reviewing process.
> The PR description should contain the context that made a change necessary, which is something that can not be contained in the change itself and thus can't reliably be inferred from it.
> And if I request a PR review, I trust you, as my peer, to understand the problem my change is trying to solve and to judge whether it succeeds in doing so. If I wanted AI suggestions for it I can skip the middleman.
Are you talking about keeping meaningful processes around? Because companies are trying to get rid of humans and people altogether. They don't really know why PR reviews exist - only that reviews can take time - and they don't want to spend time. But can't remove reviews anyway - so just paper over it.
I find AI summaries to be useful when writing PR's as a starting point. Having it generate a bulleted list summarizing the changes is a good way to make sure you've covered your entire PR (and it's easy to trim down unnecessary things). You can add context as needed.
Yeah, thars the big issue. Really only takes one unengaged or lazy reviewer to not take the time to revise that AI output and it will ruin the entire Org. Honestly it's a theme for all LLMs: they are great as a guide but people inevitably use them as if they are a replacement. The companies marking it as such don't help either.
As is, getting in revised LLMs responses seems like it'll be Gen Alpha's "let me Google that for you" swipe. Except at least Googling means you still need to put in leg work.
your statement is clear without the first half sentence .. another way to characterize this is discussing the role of human people as team members versus robot support services.. whose role? whose benefit? from the human side.. what accuracy ? what costs (both hidden and overt) ? from the toolset side
My last company had codium on our internal codebases for PR review/summary. I found it neat, but I never saw it make any suggestions that surpassed linting. But as a lazy dev who didn't want to rewrite all the details I put in my commit messages when making a PR, I felt it did a pretty good job summarizing.
Funny story: One time I jokingly asked it to rate the skill level of the programmer who wrote a certain PR (which was created by me). It was very complimentary of my skills, which I shared in the humor chat about how it was a very smart AI. The very next day I was laid off (unrelated, it was along with most of my team).
Wow there really is nothing that people won't shoehorn AI into these days eh? What is the point of this? What is the net business benefit, other than "but, but it's got AI"?
The PR description should contain the context that made a change necessary, which is something that can not be contained in the change itself and thus can't reliably be inferred from it.
And if I request a PR review, I trust you, as my peer, to understand the problem my change is trying to solve and to judge whether it succeeds in doing so. If I wanted AI suggestions for it I can skip the middleman.
I guess the "chat with the codebase" feature could be useful, but the auto-review feature just gives me very uncanny vibes. If someone would "review" my code by blasting it through an LLM I'd be ticked off.