Parsing a CSV file really really fast in Python (datapythonista.me)

submitted 2 years ago by rimu@piefed.social to c/python@programming.dev

5 comments fedilink hide all child comments

If you care about performance, you may want to avoid CSV files. But since our data sources are often like our family, we can't make a choice, we'll see in this blog post how to process a CSV file as fast as possible.

you are viewing a single comment's thread
view the rest of the comments

[-] stevedidwhat_infosec@infosec.pub 4 points 2 years ago* (last edited 2 years ago)

Okay so would it be faster to convert it to something better and then do something faster with this better format?

Edit: I guess looking at the numbers, they’re already pretty low there. Idk how much faster it’d really be and whether not it’d be worth doing

What’s even the “gold standard” for logging stuff I guess?

[-] sugar_in_your_tea@sh.itjust.works 2 points 2 years ago

That really depends on how much of it you're doing. If you're just handing a few times at a time, the difference between 0.1s and 3s isn't that big of a deal. If you're handling thousands or even millions in a day, it can be an order of magnitude cost savings to make it more efficient.

We use a CSVs at work, but it's not a common thing so we just use the built-in csv library. If we did more with it, pandas would be the way to go (or maybe we'd rewrite that service in Rust).