I see that you use euclidean distance of perceptual color and named your color function bad_color_distance. My only input here would be to try something like the CIEDE2000 color difference formula to see if the result can be further improved.
Nice! I made a doom renderer with emojis some time ago, but I decided to keep the emojis bigger, so you could identify them. But I like your approach, will definitely try in my doom renderer too: https://github.com/CrociDB/doomoji