84
Parsing a CSV file really really fast in Python (datapythonista.me)
submitted 2 years ago by rimu@piefed.social to c/python@programming.dev
5 comments fedilink hide all child comments

If you care about performance, you may want to avoid CSV files. But since our data sources are often like our family, we can't make a choice, we'll see in this blog post how to process a CSV file as fast as possible.

you are viewing a single comment's thread
view the rest of the comments
[-] fartsparkles@sh.itjust.works 27 points 2 years ago

Holy shit, switching to PyArrow is going to make me seem a mystical wizard when I merge in the morning. I’ve easily halved the execution time of a horrible but unavoidable job (yay crappy vendor “API” that returns a huge CSV).

[-] AlecSadler@sh.itjust.works 3 points 2 years ago

You and me both. I've been parsing around 10-100 million row CSVs lately and...this will hopefully help.

this post was submitted on 18 Mar 2024
84 points (100.0% liked)

Python

8022 readers
2 users here now

Welcome to the Python community on the programming.dev Lemmy instance!

📅 Events

PastNovember 2023

October 2023

July 2023

August 2023

September 2023

🐍 Python project:
💓 Python Community:
✨ Python Ecosystem:
🌌 Fediverse
Communities
Projects
Feeds

founded 3 years ago
MODERATORS
erlingur@programming.dev
jnovinger@programming.dev
norambna@programming.dev