Can gzip be a language model? (nathan.rs)
from cm0002@mander.xyz to programming@programming.dev on 18 Jun 14:04
https://mander.xyz/post/53820216

#programming

threaded - newest

phoenixz@lemmy.ca on 18 Jun 15:35 next collapse

Very interesting, thanks!

litchralee@sh.itjust.works on 18 Jun 16:10 next collapse

For an explainer of the theory behind the “language modeling is compression” paper, this video by 3Blue1Brown is especially relevant: youtube.com/watch?v=l6DKRf-fAAM

AnAmericanPotato@programming.dev on 18 Jun 16:13 collapse

The coolest, and often most confusing thing about computer science, information theory, and perhaps reality in general, is how everything becomes more or less equivalent if you boil it down and twist it around a little.

Everything is sorting. Everything is compression. Everything is geometry. Everything is language. Everything is music. Everything is, like, waves, man. *puff*

Or more accurately, everything can be expressed in any of those other things’ terms.

These are not new ideas, but computers have made them provably and demonstrably true in many contexts, and I think that’s super cool.