This is really cool, but it's clear that the person requesting the SQL has to know whether the generated SQL is correct for it to be of use.
If I'm a non-technical user and I ask a plain-language question and the generated SQL is incorrect, it's likely going to give the wrong answer -- but unless it's terribly wrong ("Syntax error", truly implausible values) the user may not know that it's wrong.
So I see this as more of a tool to speed up development than a tool that can power end users' plain-language queries. But who knows? Maybe GPT-4 will clear that hurdle.
One of the authors here. You're exactly right. We're no where near being able to automate a 100 line `SELECT` statement with CTE's etc., but it does a decent job of starting you off.
If I'm a non-technical user and I ask a plain-language question and the generated SQL is incorrect, it's likely going to give the wrong answer -- but unless it's terribly wrong ("Syntax error", truly implausible values) the user may not know that it's wrong.
So I see this as more of a tool to speed up development than a tool that can power end users' plain-language queries. But who knows? Maybe GPT-4 will clear that hurdle.