Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

That is not the absolute minimum. Unicode is complex beast and oversimplification is dangerous.

When these absolute minimum intros talk just about encoding it misleads people to think that that's enough. I can't count the number of people who have read Joel's article and have the misconception that all user perceived characters are mapped to code points. I was one of those people. Just because ASCII and Latin-1 character sets can be mapped to code points does not mean that's how Unicode works.

At minimum every software developer must know four different levels:

* bytes,

* code points,

* combining character sequence

* grapheme clusters, extended grapheme clusters

Joel stops at the second level. He never gets into point where he explains how encode user perceived characters, how to detect grapheme cluster boundaries in the Unicode encoding.

examples: 각 , नी , நி



Maybe the perfect is enemy of the good?

If a developer knows and understands the concept of character encoding ("It does not make sense to have a string without knowing what encoding it uses"), at least they will know how to read a string from one system and move it to a different system that expects a different encoding. They'll know that they need to call the relevant conversion routine in a specialized library that knows how to handle the conversion.

With this, maybe they won't be able to correctly build or modify their own strings directly. But being able to handle strings from an external system that produces them, and passing them to another system that consumes them, without breaking them in the process, IMHO does qualify as "the absolute minimum" they should know.


Thanks, but I'm happy to be ignorant about levels three and four. I know they exist, and that's enough for me. Something is really wrong if programmers of most applications domains have to care about that complexity.


Knowing that their exist is the minimum you must know. Knowing what you don't know is already knowledge.

Joel gives the impression that he don't know that he don't know.

Knowing that you can't break a Unicode text string or insert text into the middle of Unicode string unless you know what language it uses is usually enough. They are just binary blocks you can't modify unless you have some extra info or uses specific libraries.


You have to know they exist, and know when you should worry about them (and call in to the appropriate APIs)

https://manishearth.github.io/blog/2017/01/15/breaking-our-l... gives an overview of most of the different things scripts do. Being aware of those helps a lot. It also gives a brief idea of how to deal with this stuff (usually it's just calling an API)

https://manishearth.github.io/blog/2017/01/14/stop-ascribing... gives an idea of how grapheme clusters work. You don't need to know the algorithm, just the stuff around it.


What Joel covers is just fine for software developers who a) work in a language written in Latin script and b) aren't specifically responsible for internationalization of their product. Deeper issues can be left to specialists.


At minimum every software developer must also know about normalization, and must know what pre-composed and decomposed forms are (especially normal forms, NFC and NFD).

These things are things one can mostly ignore, until one can't.

Most input modes produce pre-composed (but not necessarily NFC) output. But some things will decompose (e.g., HFS+ will decompose filenames). So... if you cut-n-paste non-ASCII Unicode from an HFS+ file picker UI... you'll get into trouble if the software you paste into is unaware of these things.

Ultimately, every software developer needs:

- UTF validator (at least for UTF-8)

- UTF converters (unless only supporting UTF-8)

- case mapping (probably)

- normalization (almost certainly)

- collation (probably)

That's... not too bad.

Networking software )may_ also need:

- IDNA2008 implementation

- UTS#46 implementation

Word processing / typesetting software also absolutely needs to know about grapheme clusters in order to determine the size of each grapheme. Also: modern fonts.


So you're just basically listing everything you know about Unicode and then state that every software developer MUST know the same. That's bullshit of course.


WAT? I gave a breakdown of when you might need support for specific aspects of Unicode. That is not an exhaustive list, just a list that will get 95% of developers covered in most cases. I didn't mention bi-di, for example, though I probably should have.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: