commands hanging but they execute when I press stop · jupyterlab-hub

Hello,
I've been experiencing some weird behavior on jupyterhub since the updates. I've been finding that even fairly basic commands are hanging for ages, perhaps forever - I haven't let it get complete. When I press the stop button, they finish executing and they appear to do it successfully. For example, I will plot a plot, and it'll just hang there looking like it's working away for way longer than it should. I press stop and then the plot appears. Is anyone experiencing similar issues?
Thanks,
Isla

Isla Simpson (Jan 15 2024 at 16:51):

Just adding another issue that I'm finding since the updates. Often when I execute cells the jupyterhub interface jumps around to some other cell such that you lose track of where you are in your code. I know that I'm not the only one who is experiencing this particular problem since the updates. But I'm not sure if others are experiencing the same problem as I am encountering above.

Anna-Lena Deppenmeier (Jan 16 2024 at 17:25):

Adrianna Foster (Jan 16 2024 at 17:36):

Isla Simpson (Jan 16 2024 at 17:43):

Some more weird behavior to add to the list is that I don't seem to be able to undo anything.

Anna-Lena Deppenmeier (Jan 16 2024 at 19:50):

and when I try to stop a cell that would have executed two steps it seems that it only stops the first step and then goes through with the second in that cell.

Anna-Lena Deppenmeier (Jan 16 2024 at 19:52):

and it might be executing more than one cell though it does not show that it has finished executing the first cell. e.g. i am currenlty trying to load some data, I have four cells that each load one specific field. I started executing all of them, I saw the apporpriate workers on the dask panel. now it seems to be loading the data from the second cell that I am trying to execute, but it does not indicate that it is done with the first one.

Brian Vanderwende (Jan 16 2024 at 20:19):

Thanks all - and thanks to Isla for pinging me in the NHUG meeting on these issues. I think we'll work these as a ticket for HPCD tracking, but I'll forward the reports to Jared immediately to get eyes on it.

Yassir Eddebbar (Jan 16 2024 at 23:05):

David Ahijevych (Jan 17 2024 at 18:03):

I noticed similar behavior before the update. The cell would hang, the kernel activity/progress circle would indicate it was busy. It would slowly empty the progress circle on its own, but if I pressed the stop button, it would empty faster--one little pie piece at a time. It seemed like the more CPUs I used, the more times I had to press the stop button.

Daniel Kennedy (Jan 17 2024 at 20:36):

another (more minor) degradation, tab completion for file paths (as strings) used to work really nicely for me and now is not very useful

Katie Dagon (Jan 17 2024 at 23:30):

I'm was having similar, but not exactly the same issues with very slow dask/xarray commands after the JHub updates. For example a notebook that I ran before the update would now take 10x as long to execute, and the dask dashboard seemed to indicate tasks been executed repeatedly (or not "lazy"). This appears to be resolved with a fresh environment install, so I suspect for me it might have been the version of dask/xarray (or something else) in my old environment didn't play nicely with something about the updated JHub.

Brian Vanderwende (Jan 17 2024 at 23:46):

Thanks for that data point Katie. We're still looking into things and trying to find patterns, so that type of information is very useful.

@Daniel Kennedy - that might be tied to some lab extension that has changed. We'll look into it, but at lower priority than the slowdowns and other strange cell behavior. Thanks for the heads up!

Holly Olivarez (Jan 17 2024 at 23:50):

@Katie Dagon are you saying you created a fresh environment? I am experiencing problems similar to what you have described. So far, I've updated all packages and shut down/restart the server and my laptop, but to no avail.

Katie Dagon (Jan 18 2024 at 16:27):

@Holly Olivarez yes a fresh environment. I also used the CISL supported versions of conda/mamba this time instead of my personal miniconda install (not sure if that made a difference, but just in case!)

Adrianna Foster (Jan 18 2024 at 17:06):

I also am using a fresh environment but it doesn't seem to help. how did you use the CISL supported version of conda/mamba? I tried installing the env using mamba but it looks like it still was placed in my miniconda

Katie Dagon (Jan 18 2024 at 17:12):

@Adrianna Foster try module load conda and then check which conda before creating the environment. it should not point to your personal miniconda

Brian Vanderwende (Jan 19 2024 at 16:29):

Hey all - it's still unclear to me why this only happens intermittently, but I have come to suspect that a particular JupyterLab extension may be causing the strange behavior with Dask when using Xarray objects. The "variableinspector" extension allows you to look at variable values in the GUI. This extension was installed before the Hub upgrade too, but we are using a newer version that added support for Xarray.

Variable inspectors have been known to break lazy evaluation (e.g., PyCharm can do this...). So far in my testing turning this extension off has eliminated the problem for me, but I'd love to get some more data points before we make any global changes.

So if you are seeing issues with Dask notebooks, I encourage you to turn this extension off and report back your findings. You can turn it off by starting a terminal in a Hub session and running:

jupyter labextension disable --level=user @lckr/jupyterlab_variableinspector

Extensions don't turn on/off without reloading JupyterLab, so you will then need to terminate your running server and start a new server before the extension is turned off.

Isla Simpson (Jan 19 2024 at 16:42):

Isla Simpson (Jan 19 2024 at 17:35):

I tried after doing this and I'm still having the issue that it won't execute the cell until I press stop. However, I'm unsure whether the labextension disable command has been executed properly because of the issue I mentioned above.

Adrianna Foster (Jan 19 2024 at 20:31):

I am about to try this - but I also wanted to note that I keep getting this error message (and it's happening over and over again in the cell), i'm not sure if this is related or not or just something weird I am doing...

Task exception was never retrieved
future: <Task finished name='Task-434389' coro=<Client._gather.<locals>.wait() done, defined at /glade/work/afoster/conda-envs/ml_analysis/lib/python3.11/site-packages/distributed/client.py:2208> exception=AllExit()>
Traceback (most recent call last):
  File "/glade/work/afoster/conda-envs/ml_analysis/lib/python3.11/site-packages/distributed/client.py", line 2217, in wait
    raise AllExit()
distributed.client.AllExit
Task exception was never retrieved
future: <Task finished name='Task-413222' coro=<Client._gather.<locals>.wait() done, defined at /glade/work/afoster/conda-envs/ml_analysis/lib/python3.11/site-packages/distributed/client.py:2208> exception=AllExit()>
Traceback (most recent call last):
  File "/glade/work/afoster/conda-envs/ml_analysis/lib/python3.11/site-packages/distributed/client.py", line 2217, in wait
    raise AllExit()
distributed.client.AllExit

Brian Vanderwende (Jan 19 2024 at 20:40):

@Adrianna Foster - I'm not sure that the error is caused by this, but the fact that it happens repeatedly might be triggered by the extension. Let's see how things work for you once the extension is off and if the error remains, hopefully it will be easier to debug.

Adrianna Foster (Jan 19 2024 at 22:13):

@Brian Vanderwende it didn't seem to stop that error message unfortunately, and I'm still getting the weird behavior

Adrianna Foster (Jan 19 2024 at 22:14):

I'm actually noticing now that the whole page seems to freeze and/or crash when stuff like this happens

Stephen Yeager (Jan 22 2024 at 20:27):

@Katie Dagon I'm wondering if you pinned any packages in your environment update? I tried to update my environment and am finding that dask is either extremely slow or unresponsive with v2024.1.0 (for both xarray and dask).

Katie Dagon (Jan 22 2024 at 20:42):

@Stephen Yeager I didn't specify any package versions in my new environment except asking for python=3.11. It looks like it installed dask v2024.1.0. I didn't have any issues initially, but given @Brian Vanderwende's comment above it might be worth checking that labextension if the issues persist.

Katie Dagon (Jan 22 2024 at 20:44):

However I just noticed the xarray version I have installed is 2023.12.0. It looks like I just missed the latest version v2024.01.0 which came out on Jan 17, so that could be a difference.

Negin Sobhani (Jan 22 2024 at 21:28):

You can check jupyter labextension list to check if the extension is on or off.

Brian Vanderwende (Jan 23 2024 at 18:02):

Quick update - we've disabled the variableinspector extension for all users. You can still optionally enable it if you like it, but since it impacts Dask behavior when using Xarray, it wasn't tenable to keep it on by default.

We've also had a few users ask about disabling the LSP/linter that underlines code and adds other IDE-type behaviors. This one is more subjective so it's not clear whether it makes sense to set it off by default, but if you wish to disable it entirely, you can do so by running:

jupyter labextension disable --level=user @jupyter-lsp/jupyterlab-lsp

This will again require a restart of your JupyterHub server for it to take effect.

Stream: jupyterlab-hub

Topic: commands hanging but they execute when I press stop

Isla Simpson (Jan 13 2024 at 20:27):