Miles Alan Moore (Jul 20 2024 at 21:44): Miles Alan Moore (Jul 20 2024 at 21:44):Hello all,
I am a graduate student working with Will Wieder (TSS/CGD) at the Mesa Lab. I am looking to use NASA Western Land Data Assimilation System data in an analysis we are working on. To my knowledge, it is not already in any campaign directories or on glade. Even if I clip to my AOI, the data are still several Terabytes. I wonder if anyone has experience compiling data of this size onto glade or otherwise?
The WLDAS documentation provides good documentation for downloading the data (using curl, wget), but the download will likely take a couple days using the wget/curl methods. I'm afraid that would be a pretty careless and wasteful use of our project's compute resources. Any advice here? Thank you for your time!
Cheers,
Miles
Katelyn FitzGerald (Jul 22 2024 at 19:28):@Miles Alan Moore it looks like this is a fairly recent product. It's possible someone's grabbed this (especially LSM / hydrology folks), but I'd guess you're right that it's not archived in a shared location. @Negin Sobhani or someone else in the support group (also reachable here: https://ithelp.ucar.edu/plugins/servlet/desk/site/rc) might know better and/or have some advice.
They'd likely have better recommendations for how to do data transfers as well. I know there are some dedicated resources for supporting certain types of data transfers.
Miles Alan Moore (Jul 22 2024 at 21:42):@Katelyn FitzGerald Thank you so much for the kind and helpful response! Someone had mentioned to me that it was fine to run a disowned processing running (like wget via a screen ) in a casper login node for a while to download data of this sort into my glade scratch/ space. This way I am not hogging any compute resources that others could be using. So this is what I've done for now! If anyone thinks this is actually a bad idea / party foul, I'm happy to do it differently.
Thank you, again, for your time! :grinning_face_with_smiling_eyes:
Katelyn FitzGerald (Jul 22 2024 at 21:47):Glad you got this sorted!
Notification Bot (Jul 22 2024 at 21:47):Katelyn FitzGerald has marked this topic as resolved.
Negin Sobhani (Jul 23 2024 at 18:28):Hello @Miles Alan Moore , There should not be an issue with downloading files via wget as long as you have sufficient storage space, but I would suggest NOT using casper login nodes as they are usually very busy and high usage will cause slowness for all users. Instead, I suggest submitting jobs to the queue to download the files. If you prefer using login nodes, I recommend using Derecho rather than Casper. Please avoid running large wget processes on Casper login nodes to ensure optimal performance for everyone.
If you have any further questions or need assistance, feel free to reach out at rchelp@ucar.edu.
Miles Alan Moore (Jul 23 2024 at 18:37):@Negin Sobhani Thank you for your response! I'll stop my wget process on the Casper Login node and resubmit as a job.
Last updated: May 16 2025 at 17:14 UTC