> I have to imagine if you give it positions in text it'd be pretty great

Not at all? LLMs are a terrible match for the kind of analysis a chess engine does (scaled deep search, deeply trained position evaluations). It's just not that kind of tool.