Stream: announce

Topic: New conda tools and tracking repo for managed environments

Brian Vanderwende (Jul 05 2023 at 16:59):

I want to share a few things we've put together that may make creating and sharing Conda environments easier. First, I've put together two helper utilities which are now in the default environment on Cheyenne and Casper:

• create_conda_kernel - you can use this script to quickly create a personal Jupyter kernel from any accessible conda environment. Let's say a colleague has an environment they are using to run a notebook, and you wish to do the same. Simply run create_conda_kernel /glade/work/colleague/conda-envs/their_env and it will create a kernel you can use that points to their environment. Keep in mind that any changes they make to their environment will impact your usage too.
• compare_env_yaml - you can use this script to check the status of packages in an environment relative to a specified YAML file. This could be useful if you've deployed an environment from a requirements file but then added packages using conda/mamba, and you want to see the differences between the two. Example: compare_env_yaml -y /path/to/requirements.yml conda-env-name

Finally, we are now tracking our environment YAML files in a github repo: https://github.com/NCAR/ncar-conda. Any changes we make to our deployments can be tracked there. You can clone this repo elsewhere if you have reason to use our environments on another system. And feel free to post GitHub issues there too if you prefer - though we still consider support tickets to be our primary means of support for these environments (our NPL and R environments).

Katie Dagon (Jul 05 2023 at 20:44):

These are great resources, thanks @Brian Vanderwende. I was going to suggest we organize an ESDS forum around conda, and then I saw you are already signed up for August 7 :smile: . Looking forward to it! We could also think about turning that discussion into a blog post or FAQ for the ESDS website, which I think would be broadly useful.

Brian Vanderwende (Jul 05 2023 at 21:20):

Thanks @Katie Dagon, the blog post sounds like a great idea. Once I have a decent draft of material, I'll reach out to you to coordinate that part of it.

Daniel Howard (Jul 06 2023 at 17:17):

I also encourage, with approval already provded from Brian, to submit recommended changes or library additions to npl via a GitHub issue at https://github.com/NCAR/ncar-conda/issues. With any issue, best practice will be to include a record of successful installation of that package to an NPL clone and any important changes, downgrades, dependencies, etc brought in with said install.

Also of note, here are useful commands conda list --revisions and conda install --revision=REVNUM for investigating changes to an environment over time and reverting changes.

Deepak Cherian (Jul 12 2023 at 15:10):

Thanks @Brian Vanderwende et al. conda-store looks like a useful tool for this problem, based on conversations at SciPy.

Brian Vanderwende (Jul 12 2023 at 15:56):

Thanks for passing that along @Deepak Cherian. At first glance it looks intriguing, though I'll have to play with it some to get a full picture.

Last updated: May 16 2025 at 17:14 UTC