26
all 10 comments
sorted by: hot top new old
[-] explodes@lemmy.world 23 points 2 years ago

If you're storing your data as json on disk, you probably never cared about efficient storage anyway.

[-] Quetzalcutlass@lemmy.world 9 points 2 years ago

They thought "data lake" was a physical description.

[-] wewbull@feddit.uk 8 points 2 years ago

Rust was the important factor in this result. That's why it's in the headline. It wasn't the hugely inefficient way of storing the data initially. Not at all.

FFS, you could just have run gzip on it probably.

[-] FizzyOrange@programming.dev 4 points 2 years ago

Hmm I think just using SQLite or DuckDB with normalised data would probably get you 99% of the way there...

[-] chaospatterns@lemmy.world 2 points 2 years ago* (last edited 2 years ago)

It all depends on how it's represented on disk though and how the query is executed. Sqlite only supports numbers and strings, and if you keep using a VARCHAR, a read of those rows are going to have materialize a string into memory inside the sqlite library. DuckDB has more types, but if you're using varchars everywhere, something has to read that string into memory unless you can push down logic into a query that doesn't actually have to read the actual value, such as one that can use indices.

The best way is to change the representation on disk, such as converting low-cardinality columns like the station into a numeric id. A standard int being four bytes is a lot more efficient than an n-byte string + a header and it can be compared by value.

This is where file formats, like Parquet, shine. They're oriented more towards parsing by systems. JSON is geared towards human parsing.

this post was submitted on 06 Mar 2025
26 points (90.6% liked)

Programming

28322 readers
691 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 3 years ago
MODERATORS