Now this is UX. Wonderful stuff.

<img alt="Screenshot of the page showing me 20 mouse cursors moving across the page" src="https://i.vgy.me/LQLNTN.png">

I love the comparison of string length of the same UTF-8 string in four programming languages (only the last one is correct, by the way):

Python 3:

len(“🤦🏼‍♂️”)

5

JavaScript / Java / C#:

“🤦🏼‍♂️”.length

7

Rust:

println!(“{}”, “🤦🏼‍♂️”.len());

17

Swift:

print(“🤦🏼‍♂️”.count)

1

I’m still sour about text having color. Yeah I know little icons peppered forums. That’s why people liked reddit! It got rid of that shit! Now it’s part of the universal standard? Not just the ability to draw a turd on someone’s monitor, but to have it be colored-in brown? The hell with that. You wanna have animated GIFs next? Let someone put their username in marquee? Or like right-alignment, make rainbow signatures a free gimmick that text engines have to live with.

Meanwhile the alphabet of upside-down or small-caps letters are still incomplete.

They believed 65,536 characters would be enough for all human languages.

Gotta love these kind of misjudgements. Obviously, they were pushing against pretty hard size restrictions back then, but at the same time, they did have the explicit goal of fitting in all languages and if you just look at the Asian languages, it should be pretty clear that it’s not a lot at all…

Holy Jesus, what a color scheme.

Was actually a great read. I didn’t realize there were so many ways to encode the same character. TIL.

Unicode is thoroughly underrated.

UTF-8, doubly so. One of the amazing/clever things they did was to build off of ASCII as a subset by taking advantage of the extra bit to stay backwards compatible, which is a lesson we should all learn when evolving systems with users (your chances of success are much better if you extend than to rewrite).

On the other hand, having dealt with UTF-7 (a very “special” email encoding), it takes a certain kind of nerd to really appreciate the nuances of encodings.

Man, Unicode is one of those things that is both brilliant and absolutely absurd. There is so much complexity to language and making one system to rule them all ends up involving so many compromises. Unicode has metadata for each character and algorithms dealing with normalization and capitalization and sorting. With human language being as varied as it is, these algorithms can have really wacky results. Another good article on it is eev.ee/blog/2015/09/12/dark-corners-of-unicode/

And if you want to RENDER text, oh boy. Look at this: faultlore.com/blah/text-hates-you/

The mouse pointer background is kinda a dick move. Good article. but the background is annoying for tired old eyes - which I assume are a target demographic for that article.

wss://tonsky.me
http://tonsky.me/pointers/
https://tonsky.me/pointers/

I do understant why old unicode versions re-used “i” and “I” for turkish lowercase dotted i and turkish uppercase dotless I, but I don’t understand why more recent version have not introduce two new characters that looks exactly the same but who don’t require locale-dependant knowlege to do something as basic as “to lowercase”.

The article sure mentions 💩a lot.

If you go to the page without the trailing slash, the images don’t load

currency symbols other than the $ (kind of tells you who invented computers, doesn’t it?)

Who wants to tell the author that not everything was invented in the US? (And computers certainly weren’t)

The only modern language that gets it right is Swift:

print("🤦🏼‍♂️".count)
// => 1

Minor, but I’m not sure this is as unambiguous as the article claims. It’s true that for someone “that isn’t burdened with computer internals” that this is the most obvious “length” of the string, but programmers are by definition burdened with computer internals. That’s not to say the length shouldn’t be 1 though, it’s more that the “length” field/property has a terrible name, and asking for the length of a string is a very ambiguous question to begin with.

Instead, I think a better solution is to be clear what length you’re actually referring to. For example, with Rust, the .len() method documents itself as the number of bytes in the string and warns that it may not be what you’re interested in. Similarly, .chars() clarifies that it iterates over Unicode Scalar Values, and not grapheme clusters (and that grapheme clusters are unfortunately not handled by the standard library).

For most high level applications, I think you generally do want to work with grapheme clusters, and what Swift does makes sense (assuming you can also iterate over the individual bytes somehow for low level operations). As long as it is clearly documented what your “length” refers to, and assuming the other lengths can be calculated, I think any reasonably useful length is valid.

The article they link in that section does cover a lot of the nuances between them, and is a great read for more discussion around what the length should be.

Edit: I should also add that Korean, for example, adds some additional complexity to it. For example, what’s the string length of 각? Is it 1, because it visually consumes a single “space”? Or is it 3 because it’s 3 letters (ㄱ, ㅏ, ㄱ)? Swift says the length is 1.

I’m personally waiting for utf-64 and for unicode to go back to fixed encoding and forgetting about merging code points into complex characters. Just keep a zeptillion code points for absolutely everything.

Just give me plain UTF32 with ~@4 billion code points, that really should be enough for any symbol ee can come up with. Give everything it’s own code point, no bullshit with combined glyphs that make text processing a nightmare. I need to be able to do a strlen either on byte length or amount of characters without the CPU spendings minute to count each individual character.

I think Unicode started as a great idea and the kind of blubbered into aimless “everybody kinda does what everyone wants” territory. Unicode is for humans, sure, but we shouldn’t forget that computers actually have to do the work

Because strings are such a huge problem nowadays, every single software developer needs to know the internals of them. I can’t even stress it enough, strings are such a burden nowadays that if you don’t know how to encode and decode one, you’re beyond fucked. It’ll make programming so difficult - no even worse, nigh impossible! Only those who know about unicode will be able to write any meaningful code.