fossilesque@mander.xyzM to Science Memes@mander.xyz • 5 months agoPandasexternal-linkmessage-square8 fedilinkarrow-up1112
arrow-up1112external-linkPandasfossilesque@mander.xyzM to Science Memes@mander.xyz • 5 months agomessage-square8 Commentsfedilink
minus-squareKausta@lemm.eehexbear24·5 months agoYou havent seen anything until you need to put a 4.2gb gzipped csv into a pandas dataframe, which works without any issues I should note. linkfedilink
minus-squareQuizzaciousOtter@lemm.eehexbear5·5 months agoI really don't think that's a lot either. Nowadays we routinely process terabytes of data. linkfedilink
minus-squareKausta@lemm.eehexbear2·5 months agoYeah, it was just a simple example. Although using just pandas (without something like dask) for loading terabytes of data at once into a single dataframe may not be the best idea, even with enough memory. linkfedilink
You havent seen anything until you need to put a 4.2gb gzipped csv into a pandas dataframe, which works without any issues I should note.
I really don't think that's a lot either. Nowadays we routinely process terabytes of data.
Yeah, it was just a simple example. Although using just pandas (without something like dask) for loading terabytes of data at once into a single dataframe may not be the best idea, even with enough memory.