Stream: jupyterlab-hub

Topic: jupyterhub not showing new conda environments

Keith Lindsay (Jan 21 2022 at 15:34):

I'm starting a new suite of notebooks and have created a new conda environment that I intend to use for them.
However, when I create a new notebook with jupyterhub, and go to change the kernel, the new conda environment is not listed.
I have this problem with conda and/or mamba created environments.
I've tried new jhub sessions on casper login and batch nodes, and cheyenne login nodes and still don't see the new environments.
I haven't created new conda environments in quite awhile.
Is there something I need to do these days to get jupyterhub to recognize new conda environments as potential kernels?

I did have problems with inconsistencies in my base conda environment and re-installed a handful of packages. It's possible that I inadvertently changed a setting related to jhub recognizing conda environments. I don't know how this recognization works, so I don't know if my actions broke it.

Matt Long (Jan 21 2022 at 15:36):

@Keith Lindsay, I think you need to include the nb_conda_kernels package in your kernel environments. I usually just include the jupyterlab package in my environment.yml file, which has also includes this package.

Keith Lindsay (Jan 21 2022 at 15:51):

Thanks @Matt Long, adding jupyterlab to the environment enabled jhub recognization (though it did not add nb_conda_kernels).

I was previously thinking, but hadn't mentioned it, that an ESDS blog on minimal packages for conda environments would be useful.
This gotcha reinforces that thought enough for me to mention it.

Matt Long (Jan 21 2022 at 15:52):

Great idea on the ESDS blog post.

Matt Long (Jan 21 2022 at 15:52):

cc @Max Grover

Max Grover (Jan 21 2022 at 16:07):

Typically, the package required to use your environment using the hub is ipykernel, which is mentioned within the NCAR HPC documention, and within the video tutorial of how to setup your custom conda environment on NCAR HPC resources displayed on the ESDS FAQ page... perhaps including this requirement a bit more explicitly within the text on the FAQ would be better? Open to suggestions.

Michael Levy (Jan 21 2022 at 16:26):

@Max Grover -- Keith and I had a conversation yesterday about best-practices for spinning up a new environment. It seemed like there were a few options, each with pros and cons:

1. Start with an environment.yaml file that contains a large list of recommended packages, and is likely to contain every package you might need (and then some). It's easy, since you probably only need to run conda once (to create the environment)... but the install will likely take a while (especially if you haven't started using mamba yet)
2. Start with a barebones environment.yaml file where you set up packages you know you will need (ipykernel, probably xarray, numpy, etc), and then add more packages as the need arises. This will get you up and running faster, but increases the chance of user error (oh, I need cartopy so I'll just run conda install cartopy but oops, I forgot update the YAML file). Also, install new packages piecemeal might change versions of other packages depending on when new releases are made available.
3. Create the environment by hand, and then generate environment.yaml at the end. As far as I know, conda env export doesn't always play nicely between operating systems, though?

There's also the question of "should I pin versions of packages?" It seems useful, especially given the propensity of packages (cough cough xarray cough cough) to break backwards compatibility with new releases, but on the other hand most of us tend to reuse environment files from project to project and I imagine we don't want users to be creating environments using xarray 0.15.0 today just because that's what they used in their first project two years ago.

So I think a blog post that encompasses best practices regarding (a) when / how to create your environment.yaml file, and (b) whether to pin releases of all / some / none of the packages listed in the yaml file would be extremely useful. There should definitely be a group conversation, either among ESDS or maybe just @xdev, to form a consensus... and I suspect the answers to all of the questions will be "it depends" but maybe we can provide rough guidance on how to know which process is right for a specific task?

Keith Lindsay (Jan 21 2022 at 17:10):

@Max Grover , thank you for pointing out where this is documented.

As a user, I find it challenging that documentation/advice on the python/analysis stack is scattered across so many disparate locations. I don't know what I don't know, and I don't know where to discover it. This is probably a reflection of the stack actually being composed of disparate pieces, so I don't see an easy solution to the disjointed documentation. But it makes it hard to find info on tidbits that span the pieces, like this gotcha that stumped me for a couple of hours.

Deepak Cherian (Jan 21 2022 at 17:46):

A variant on (2) in Mike's post is to add the package to environment.yaml and then call conda env update -f environment.yaml (IIRC). That way the environment and the file stay in sync, it requires some discipline though :smile:

Michael Levy (Jan 21 2022 at 17:55):

Deepak Cherian said:

A variant on (2) in Mike's post is to add the package to environment.yaml and then call conda env update -f environment.yaml (IIRC). That way the environment and the file stay in sync, it requires some discipline though :)

I create environments so infrequently that when I try to manage everything strictly through the yaml file I invariably slip up and use conda install for at least one package. It would be nice if conda had a per-environment option to require using the environment file to update (especially if that flag could be set in the environment file itself). I don't feel too bad about it, though, because I've also been on the other side of the issue and been pointed to code with an incomplete environment file. So for now I don't think many of us have the discipline required but at least it's usually an easy problem to fix :)

Max Grover (Jan 21 2022 at 18:39):

For now, I updated the FAQ page to reflect this requirement of having ipykernel installed see the section here, and I agree that a more detailed blog post would also help... @Michael Levy we are always open to contributors to the blog :grinning_face_with_smiling_eyes: seems like you have a good idea of the structure!

Last updated: May 16 2025 at 17:14 UTC