Here's an idea: make the AIs consistent at doing things computers are good at. Here's an anecdote from a friend who's living in Japan:
> i used chatgpt for the first time today and have some lite rage if you wanna hear it. tldr it wasnt correct. i thought of one simple task that it should be good at and it couldnt do that.
> (The kangxi radicals are neatly in order in unicode so you can just ++ thru em. The cjks are not. I couldnt see any clear mapping so i asked gpt to do it. Big mess i had to untangle manually anyway it woulda been faster to look them up by hand (theres 214))
> The big kicker was like, it gave me 213. And i was like, "why is one missing?" Then i put it back in and said count how many numbers are here and it said 214, and there just werent. Like come on you SHOULD be able to count.
If you can make the language models actually interface with what we've been able to do with computers for decades, i imagine many paths open up.
This needs to be generalized however. For example, if you present an AI with a drawing of some directed graph (a state diagram, for example), it should be able to answer questions based on the precise set of all possible paths in that graph, without someone having to write tooling for diagram or graph processing and traversal. Or, given a photo of a dropped box of matches, an AI should be able to precisely count the matches, as far as they are individually visible (which a human could do by keeping a tally while coloring the matches). There are probably better examples, these are off the cuff.
There’s an infinite repertoire of such tasks that combine AI capabilities with traditional computer algorithms, and I don’t think we have a generic way of having AI autonomously outsource whatever parts require precision in a reliable way.
What you're describing sounds like agentic tool usage. Have you kept up with the latest developments on that? it's already solved depending on how strict you define your criteria above
My understanding is that you need to provide and configure task-specific tools. You can’t combine the AI with just a general-purpose computer and have the AI figure out on its own how to make use of it to achieve with reliability and precision whatever task it is given. In other words, the current tool usage isn’t general-purpose in the way the LLM itself is, and also the LLM doesn’t reason about its own capabilities in order to decide how to incorporate computer use to compensate for its own weaknesses. Instead you have to tell the LLM what it should apply the tooling for.
> i used chatgpt for the first time today and have some lite rage if you wanna hear it. tldr it wasnt correct. i thought of one simple task that it should be good at and it couldnt do that.
> (The kangxi radicals are neatly in order in unicode so you can just ++ thru em. The cjks are not. I couldnt see any clear mapping so i asked gpt to do it. Big mess i had to untangle manually anyway it woulda been faster to look them up by hand (theres 214))
> The big kicker was like, it gave me 213. And i was like, "why is one missing?" Then i put it back in and said count how many numbers are here and it said 214, and there just werent. Like come on you SHOULD be able to count.
If you can make the language models actually interface with what we've been able to do with computers for decades, i imagine many paths open up.