Thanks, that's a much better example of what I was trying to explain. Again, I'm projecting humanity onto a machine, but it's not a big stretch to imagine that the interaction of attention heads over high-dimensional vectors produces "decisions" that can't be articulated in the moment, only described after the fact in text form.
As I naively understand it, that's when we do or say something then narrate ourselves why we decided to do so. We think non-verbally, then verbalize a plausible rationale for it, post hoc.
I'm not sure that applies to discursive writing, when we essentially use rules of logic to decide on the course of the narrative. Non-verbal heuristics still applies, of course, but we constrain it, so it's probably not entirely post hoc.
https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r...
https://www.researchgate.net/publication/316045349_Post_Hoc_...