I've had a brief look into duckdb, and not too sure if I'm interpreting it's use case correctly, but does it basically allow you to use SQL within your Python to query your large datasets that you have locally?

[–] catsdoingcatstuff 2 points 11 months ago (1 children)

That's right. You can read in structured files and query them locally without having to load into a database. It's nice in the case where you would rather write analytics sql, or want to convert between sql and pandas. It's very quick to load and run files. It can connect to databases, too.

[–] NoRamyunForYou 2 points 11 months ago (1 children)

Oooh that sounds pretty promising - I've been struggling with how to handle quite large datasets when they don't live within a Database.

Thank you for enlightening me! :) - I might have to send you some messages or the like later if I have any questions if that's okay with you?

[–] catsdoingcatstuff 2 points 11 months ago (1 children)

Sure thing!

[–] NoRamyunForYou 1 points 11 months ago

Thank you :)