84
Parsing a CSV file really really fast in Python (datapythonista.me)
submitted 2 years ago by rimu@piefed.social to c/python@programming.dev
5 comments fedilink hide all child comments

If you care about performance, you may want to avoid CSV files. But since our data sources are often like our family, we can't make a choice, we'll see in this blog post how to process a CSV file as fast as possible.

you are viewing a single comment's thread
view the rest of the comments
[-] sugar_in_your_tea@sh.itjust.works 2 points 2 years ago

That really depends on how much of it you're doing. If you're just handing a few times at a time, the difference between 0.1s and 3s isn't that big of a deal. If you're handling thousands or even millions in a day, it can be an order of magnitude cost savings to make it more efficient.

We use a CSVs at work, but it's not a common thing so we just use the built-in csv library. If we did more with it, pandas would be the way to go (or maybe we'd rewrite that service in Rust).

this post was submitted on 18 Mar 2024
84 points (100.0% liked)

Python

8022 readers
2 users here now

Welcome to the Python community on the programming.dev Lemmy instance!

📅 Events

PastNovember 2023

October 2023

July 2023

August 2023

September 2023

🐍 Python project:
💓 Python Community:
✨ Python Ecosystem:
🌌 Fediverse
Communities
Projects
Feeds

founded 3 years ago
MODERATORS
erlingur@programming.dev
jnovinger@programming.dev
norambna@programming.dev