conda on derecho · python-questions

This is probably a question for @Brian Vanderwende -- CESM uses python scripts to set up an appropriate environment and call make; I believe Jim Edwards talked to you yesterday about how using qcmd to call the case.build script on a derecho compute node failed if the conda module was loaded, and now the script runs successfully... but I'm still seeing

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

$ conda init <SHELL_NAME>

Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

To be clear, I used to see that message and then case.build would abort, now I see the message but case.build continues to run and succesfully builds the model... so there's definite improvement, but I was wondering if there is still something not quite right about loading the conda module on the derecho compute nodes.

I only see this message when using qcmd; using qinteractive appears to load the module correctly.

Brian Vanderwende (Aug 15 2023 at 17:01):

Sam Rabin (Aug 15 2023 at 19:07):

When I've had this issue before, I've solved it by replacing conda activate envname; python scriptname with conda run -n envname python scriptname.

Sam Rabin (Aug 15 2023 at 19:11):

As noted here, conda activate is intended for interactive shells only. qcmd is sort of a gray area, but may be affected for this reason.

Michael Levy (Aug 15 2023 at 19:19):

Huh, interesting. I think the issue is that I have a conda activate call in my .bashrc, which I thought was necessary to standardize which python I was using. On further investigation, qcmd appears to be passing my entire environment, so the conda activate is causing the error I quoted above but the system is already using the conda environment that was active when I called qcmd. So I guess the answer is to only have .bashrc call conda activate for interactive shells?

Brian Vanderwende (Aug 15 2023 at 19:21):

Yeah, that would explain why qcmd usage gives you that error. qcmd will pass along environment variables, but shell functions (which conda activate) will use do not get propagated, so the init aspects get broken to some degree. I'll see if I can make this a bit more robust, but it shouldn't prevent you from using an environment you have activated before running the qcmd (since all of the modifications conda would do to activate an env are stored in variables).

Michael Levy (Aug 15 2023 at 19:27):

I was able to fix this in my .bashrc by wrapping the conda activate in an if statement:

if [ -t 0 ]; then
  conda activate
fi

I've verified that this activates conda as expected when I log in to the machine, but does not try to run conda activate when I execute a command via qcmd. ~~I haven't tested it on cheyenne, yet~~ looks like it works on cheyenne, as well (I don't remember ever seeing this error with a cheyenne qcmd call, but qcmd -- which python returns my conda python)

Brian Vanderwende (Aug 15 2023 at 21:01):

After doing some testing - I think this is working as expected. If you were using miniconda, things would be initialized in your .bashrc by the init commands it injects. With our module, it is similar. @Michael Levy your workaround should be fine to keep if it meets all of your needs. If you would like conda to work in batch shells to, you could add module load conda to your .bashrc before the conda activate, instead of having it loaded by a default module set or your .profile.

This does not happen on Cheyenne because some module reloading tricks are happening on Derecho to bring in CESM-specific modules during the build. These tricks seem to interact with the conda module in ways I didn't anticipate, but hopefully now are harmless.

Michael Levy (Aug 15 2023 at 21:06):

yeah, I'll just stick with what I have since it seems to be working :) Thanks for all the details about why things are working the way they are!

Stream: python-questions

Topic: conda on derecho

Michael Levy (Aug 15 2023 at 15:46):