Max Grover (Jun 04 2021 at 13:53): Max Grover (Jun 04 2021 at 13:53):For anyone interested in learning more about python environments, this is a really good write up https://whiteboxml.com/blog/the-definitive-guide-to-python-virtual-environments-with-conda
Gabriela Negrete Garcia (Aug 01 2023 at 15:08):What python environments are people using that include dask clients and mapping tools like cartopy, matplotlib and Basemap? I tried using the NPL 2023b, and it supports the dask clients, but doesn’t have these mapping packages.
Katie Dagon (Aug 01 2023 at 15:31):Hi @Gabriela Negrete Garcia, I took a quick look at the packages in npl-2023b using conda list and it looks like cartopy and matplotlib are in there but basemap is not. To request packages be added to the npl environment, you can either submit a help ticket (help.ucar.edu) or open an issue in this github repo.
Katie Dagon (Aug 01 2023 at 15:31):Another option that I use a lot is creating your own environment with the packages you want in there.
Kevin Sampson (Aug 02 2023 at 16:27):I also cannot use these environments because they typically lack some of the geospatial libraries I need (OGR, GDAL, etc.). I use conda or mamba to create new environments that will contain these libraries and allow me to add new ones.
Rachel McCrary (Aug 08 2023 at 16:11):Hi! Ok so I think I'm having an issue with the NLP environment on Cheyenne/Casper. I'm trying to run some automated scripts to restart model runs - these scripts are un on the command line and I do the following:
module load conda
conda activate npl
But when I run my script - I get the following :
import numpy as np
ModuleNotFoundError: No module named 'numpy'
in another script I get
import xarray as xr
ModuleNotFoundError: No module named 'xarray'
Other import calls are working - (e.g. import glob,os,re,sys, fnmatch
from math import floor).
I'm curious what might be going on - I fully acknowledge I might be doing something wrong.
Scripts I'm working with are here: /glade/work/rmccrary/icar/cmip/MP_runs/CMIP5/CanESM2_WSM3_hist_1979-2010/scripts/ setup_next_run.py or aggregate_parallel_files.py
I have no issues with this on jupyter hub - but for some reason nothing seems to be working on the command line. Am I loading the npl environment incorrectly?
Katie Dagon (Aug 08 2023 at 16:35):Hi @Rachel McCrary I am also curious about this. I would have thought that activating NPL would point to the right python libraries so you should be able to use those modules, but I tested this and got the same import error with numpy/xarray. Tagging @Negin Sobhani and @Brian Vanderwende for any insight here.
Rachel McCrary (Aug 08 2023 at 16:47):@Katie Dagon ok so its not just me? I fumble around with Python so I always assume I'm doing something wrong - but this is breaking all my model runs! (I haven't run anything in about 6 months, but this system worked just fine in January)
Brian Vanderwende (Aug 08 2023 at 16:48):Thanks for the ping! @Katie Dagon - when you tested this, did you use Rachel's script, or did you try a simplified example? I am unable to reproduce it right now, so I'm guessing we are running into some difference in either how we are testing or our environments.
@Rachel McCrary - Do you use this on the command line on a login node or in a PBS job? If you type which python at the command line after activating the NPL, what do you get?
Rachel McCrary (Aug 08 2023 at 16:51):@Brian Vanderwende - I've tried in both a PBS job and a command line on Cheyenne. When I type which python I get
/glade/u/apps/opt/conda/bin/python
I thought it might be something up with the script I was running, but when I just open an interactive python (just by typing python) and trying to import xarray I get the same error ...
Katie Dagon (Aug 08 2023 at 16:52):Yes I did the same, just testing python and then import numpy as np after module load conda and conda activate npl. I also got the same path from which python.
Brian Vanderwende (Aug 08 2023 at 16:53):Ah, that explains why it doesn't work, though not why it is failing. It looks like it is giving you the Python in the base environment, and not the NPL. Thanks for sharing this - I need to figure out why the activation is not working for you two (whereas it is working for me).
Rachel McCrary (Aug 08 2023 at 16:55):It does work when I use a jupyter notebook :)
Katie Dagon (Aug 08 2023 at 17:02):Re: the activation, I have a separate miniconda install and I'm wondering if it doesn't get fully overriden by the module load conda. But conda env list does indicate that I have npl activated, and which conda points to /glade/u/apps/opt/conda/bin/conda
Brian Vanderwende (Aug 08 2023 at 17:05):It seems plausible, though my miniconda install does get overridden. I suppose it could be a version difference. @Katie Dagon I could try initializing your miniconda if you provide me the path and I can see if I get the same result.
Katie Dagon (Aug 08 2023 at 17:06):/glade/work/kdagon/miniconda
Katie Dagon (Aug 08 2023 at 17:06):maybe also something in PATH?
Rachel McCrary (Aug 08 2023 at 17:08):I also have a miniconda setup and something in my PATH - however I never used to have this problem, so it is a new issue! Setting all of this up was a bit ad-hoc and asking people what they did, rather than actually knowing what to do :). I haven't used my own environments in a long time.
Brian Vanderwende (Aug 08 2023 at 17:09):Hmm... okay so I do get the issue using your miniconda, Katie, though not with mine. I'll keep digging.
Katie Dagon (Aug 08 2023 at 17:12):My conda version in there is very old :grimacing: . But as Rachel said, strange that this worked previously for her and it's a new issue.
Rachel McCrary (Aug 08 2023 at 17:14):"new" is probably since January for me - I almost exclusively use JupyterHub now ...
Brian Vanderwende (Aug 08 2023 at 17:18):We do 2x a year updates to conda itself in the module. It was updated about 1 month ago, so perhaps this broke compatibility with older conda versions in the way we do it in the module. That seems like a reasonable explanation at this point, but looking to test it out myself with an older conda version.
Rachel McCrary (Aug 08 2023 at 17:20):Ok - and then let me know how I might fix this :)
Brian Vanderwende (Aug 08 2023 at 17:22):The easy fix would be to unload the conda module and run conda update conda, which should bring your miniconda to the latest (and greatest?) version of conda. I'm going to try to make this work again for older miniconda versions, but not sure yet how easy that will be!
Rachel McCrary (Aug 08 2023 at 17:25):Ok I have this installed in my home directory - which was a bad decision and I was never sure how to move it to my work directory ... can I just move it or do I need to start over?
Brian Vanderwende (Aug 08 2023 at 17:29):In my experience, trying to relocate (mv) conda environments does not work without a bunch of extra steps. Are your environments in your home too, or just miniconda itself?
Rachel McCrary (Aug 08 2023 at 19:14):@Brian Vanderwende - I think my environments are in my home too - and honestly I don't really use them I've just used npl! I have an office hours with someone from the ESDS to discuss what the heck to do with the mess I've made :)
Rachel McCrary (Aug 08 2023 at 19:33):Ok doing conda update conda did work! So thanks for that :).
Brian Vanderwende (Aug 08 2023 at 19:36):Excellent. Please let us know if you run into any other issues after the virtual help. Thanks again for reporting the issue.
Katie Dagon (Aug 08 2023 at 22:36):Thanks for your help with this Brian!
Brian Vanderwende (Aug 10 2023 at 15:07):Hey all - I've made some changes to the conda module that should make this work better. Now, if you have a conda initialized into your environment, the module will only add our settings, but will leave your conda as the active one, even if the module is loaded. This should avoid these old-new conflicts that Rachel and Katie were seeing. If you do not have conda initialized in your shell, the module will provide you with our conda.
Let me know if you run into any issues or otherwise have any feedback. Thanks again for reporting these issues to us!
Last updated: May 16 2025 at 17:14 UTC