Muntaha Pasha (May 17 2021 at 21:42): Muntaha Pasha (May 17 2021 at 21:42):I'm encountering a consistent error with a Kernel restarting on a file I've run in this past just fine. Capture.PNG
Basically when I try to load in a big data file into a dataframe, the Kernel always seems to die on me. Anyone know why this is? I just recently got handed updated data files with wind records from 2021, and since I copied those files into my repo, disregarded the old ones, and ran the new data, my Kernel always seems to just reset. It also always happens in the exact same spot- while loading data into the dF.
Is this because maybe something went wrong while copying the data? Or because of the kernel itself? Any help would be great, thanks!
Matt Long (May 17 2021 at 21:43):Could be that you are blowing memory. How big is the file? How much memory do you have? (asked rhetorically)
Muntaha Pasha (May 17 2021 at 21:52):Im running this on my glade/u/home, and when I ran the gladequota command it seems its 41.5% used (21G/50G). The file size im ingesting itself is around 5-6 GB. That shouldn't be an issue right?
Matt Long (May 17 2021 at 21:55):When you are loading the data, it is the memory limitation you need to consider, not disk space. If you are running on a login node, reading a 5 GB file could possibly blow memory.
Try spawing a server on Casper PBS Batch, for example, and specify 25 GB as the memory request.
Max Grover (May 17 2021 at 21:56):For reference, see this thread which also suggested switching from the login node to Casper PBS Batch https://zulip2.cloud.ucar.edu/#narrow/stream/16-jupyter/topic/can't.20start.20new.20thread.20error/near/31342
Muntaha Pasha (May 17 2021 at 22:01):Ah, got it! Thanks, I'll try that out now and see if it fixes the issue!
Last updated: May 16 2025 at 17:14 UTC