Stream: jupyter
Topic: ssh-tunnel to Casper
Brian Bonnlander (Mar 12 2020 at 17:01):
Hi,
Brian Bonnlander (Mar 12 2020 at 17:02):
I'm unable to start JupyterHub on Casper today through my web browser. Can someone point me to the instructions for SSH tunneling to Casper, in order to start a Jupyter session?
Brian Bonnlander (Mar 12 2020 at 17:05):
Google points me to this page: https://www2.cisl.ucar.edu/resources/computational-systems/cheyenne/software/jupyter-and-ipython
Michael Levy (Mar 12 2020 at 17:05):
@Brian Bonnlander Matt Long posted instructions in the #python-questions channel - see the "jupyterlab on Casper, no Hub" topic: https://zulip.cloud.ucar.edu/#narrow/stream/11-python-questions/topic/jupyterlab.20on.20Casper.2C.20no.20Hub
Brian Bonnlander (Mar 12 2020 at 17:06):
Thanks Michael, I knew I had seen something somewhere.
Brian Bonnlander (Mar 12 2020 at 17:20):
Perhaps I made a mistake, but I'm unable to connect to my Jupyter session in a local browser window. In one terminal, I did this:
$ ssh -Y bonnland@casper.ucar.edu Token_Response: Warning: No xauth data; using fake authentication data for X11 forwarding. Last login: Thu Mar 12 10:42:09 2020 from 128.117.9.129 ****************************************************************************** * Welcome to Casper - March 12, 2020 ****************************************************************************** Today in the Daily Bulletin (dailyb.cisl.ucar.edu) - Advance Computing for Social Change institute oppertunities for students Quick Start: www2.cisl.ucar.edu/resources/cheyenne/quick-start-cheyenne User environment: www2.cisl.ucar.edu/resources/cheyenne/user-environment Key module commands: module list, module avail, module spider, module help CISL Help: support.ucar.edu -- 303-497-2400 ------------------------------------------------------------------------------------ Resetting modules to system default (base) casper26:~$ jupyter lab --no-browser --ip=$(hostname) [I 11:10:13.194 LabApp] [nb_conda_kernels] enabled, 7 kernels found [I 11:10:16.001 LabApp] The port 8888 is already in use, trying another port. [I 11:10:16.002 LabApp] The port 8889 is already in use, trying another port. [I 11:10:16.002 LabApp] The port 8890 is already in use, trying another port. [I 11:10:16.002 LabApp] The port 8891 is already in use, trying another port. [I 11:10:16.002 LabApp] The port 8892 is already in use, trying another port. [I 11:10:17.643 LabApp] [jupyter_nbextensions_configurator] enabled 0.4.1 [I 11:10:17.717 LabApp] JupyterLab extension loaded from /glade/u/home/bonnland/miniconda3/lib/python3.7/site-packages/jupyterlab [I 11:10:17.717 LabApp] JupyterLab application directory is /glade/u/home/bonnland/miniconda3/share/jupyter/lab [I 11:10:17.719 LabApp] Serving notebooks from local directory: /glade/u/home/bonnland [I 11:10:17.719 LabApp] The Jupyter Notebook is running at: [I 11:10:17.719 LabApp] http://casper26:8930/?token=dd7e9771449d71cee8ce5d225d208a4198b3db49092da386 [I 11:10:17.719 LabApp] or http://127.0.0.1:8930/?token=dd7e9771449d71cee8ce5d225d208a4198b3db49092da386 [I 11:10:17.719 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 11:10:17.797 LabApp] To access the notebook, open this file in a browser: file:///glade/u/home/bonnland/.local/share/jupyter/runtime/nbserver-36686-open.html Or copy and paste one of these URLs: http://casper26:8930/?token=dd7e9771449d71cee8ce5d225d208a4198b3db49092da386 or http://127.0.0.1:8930/?token=dd7e9771449d71cee8ce5d225d208a4198b3db49092da386
And then in another window, I got this:
$ ssh -N -L 8930:localhost:8888 casper26.ucar.edu The authenticity of host 'casper26.ucar.edu (128.117.181.206)' can't be established. ECDSA key fingerprint is SHA256:zH2MWNf39A459aI/iHdTszsWqhmRcGtgD3krCrGHT7s. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'casper26.ucar.edu' (ECDSA) to the list of known hosts. Token_Response: channel 2: open failed: connect failed: Connection refused channel 3: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused
Has anyone seen this before, or did I do anything wrong?
Brian Bonnlander (Mar 12 2020 at 17:24):
I've also tried this URL in my browser, and it hangs with the message "Connecting....": http://casper26.ucar.edu:8930/?token=dd7e9771449d71cee8ce5d225d208a4198b3db49092da386
Michael Levy (Mar 12 2020 at 17:29):
The second window should be
$ ssh -N -L 8930:casper26:8888 casper26.ucar.edu
(replace localhost with casper26 between the port numbers)
Brian Bonnlander (Mar 12 2020 at 17:35):
OK, thanks for that! I am supposing that the result of the "ssh -N" command is that you are asked for a Token_Response, and then it doesn't return to the command prompt, and there is no further output. This is what I'm seeing. I still can't connect to localhost:8888 or casper26:8888 in my browser, however.
Michael Levy (Mar 12 2020 at 17:37):
there shouldn't be any output from the ssh tunnel command, but I would expect http://localhost:8888 to work. Is the first window showing that jupyterlab is still running?
Michael Levy (Mar 12 2020 at 17:40):
Oh, swap the port numbers:
$ ssh -N -L 8888:casper26:8930 casper26.ucar.edu
Michael Levy (Mar 12 2020 at 17:41):
(I always get those reversed so I typically just use the same port locally)
Brian Bonnlander (Mar 12 2020 at 17:44):
OK, that works! Thank you Michael!
Matt Long (Mar 12 2020 at 19:42):
FYI, Casper was being hammered this morning by a renegade user. I couldn't get a session all morning. The help desk has dealt with the issue.
Brian Bonnlander (Mar 12 2020 at 19:49):
What a thankless job. They need a way of preventing users from blindly firing off identical processes. On my casper node, I'm seeing this for Jupyter lab sessions:
(base) casper26:~$ ps aux | grep jupyter vgarza 18304 0.0 0.0 113148 1588 ? S Mar06 0:00 /bin/bash /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/start-jupyter vgarza 18308 0.0 0.0 353008 57460 ? S Mar06 0:00 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/jupyter-lab --no-browser --ip=casper26 vgarza 18592 0.0 0.0 113148 1592 ? S Mar06 0:00 /bin/bash /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/start-jupyter vgarza 18596 0.0 0.0 966740 64004 ? Sl Mar06 2:07 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/jupyter-lab --no-browser --ip=casper26 vgarza 18914 0.0 0.0 1133832 185200 ? Ssl Mar06 0:21 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-de0ab2a1-43a8-40b7-beea-dd8a1337fba0.json vgarza 18917 0.0 0.0 1416876 202428 ? Ssl Mar06 0:28 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-28dfa765-7906-41bb-b6db-143d5729c872.json vgarza 18920 0.0 0.0 737476 43508 ? Ssl Mar06 0:19 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-8e552a3a-b80a-4b90-ba15-ffe246580fec.json vgarza 18923 0.0 0.0 737476 46912 ? Ssl Mar06 0:20 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-34f06f3f-8e8c-4138-b515-d369603b5e75.json bryan 21528 0.0 0.0 611908 57884 ? Ssl 10:40 0:01 /glade/work/bryan/miniconda3/envs/analysis/bin/python -m ipykernel_launcher -f /glade/u/home/bryan/.local/share/jupyter/runtime/kernel-2d7a6f62-e10c-4a84-82a7-6c893dd5921c.json bryan 23085 0.0 0.0 611896 55016 ? Ssl 10:44 0:01 /glade/work/bryan/miniconda3/envs/analysis/bin/python -m ipykernel_launcher -f /glade/u/home/bryan/.local/share/jupyter/runtime/kernel-6c42044f-6fdc-45b4-a866-b6983d346b37.json vgarza 27955 0.0 0.0 113148 1592 ? S Mar05 0:00 /bin/bash /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/start-jupyter vgarza 27959 0.0 0.0 968376 63256 ? Sl Mar05 2:11 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/jupyter-lab --no-browser --ip=casper26 vgarza 28428 0.0 0.0 1044340 89856 ? Ssl Mar05 0:23 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-2b19e44d-5386-4d23-9c69-b4e201374ea7.json vgarza 28431 0.0 0.0 737476 44572 ? Ssl Mar05 0:22 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-c041311a-bfbe-4bfc-9124-efc518925aec.json vgarza 28434 0.0 0.0 737476 44552 ? Ssl Mar05 0:22 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-d4d2b099-fdc1-4874-bcad-7c516232f943.json vgarza 28437 0.0 0.0 1273584 134344 ? Ssl Mar05 0:23 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-32dfeaba-4541-42ba-a470-2318c78ac851.json vgarza 29172 0.0 0.0 113148 1592 ? S Mar06 0:00 /bin/bash /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/start-jupyter vgarza 29176 0.0 0.0 353004 54464 ? S Mar06 0:00 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/jupyter-lab --no-browser --ip=casper26 vgarza 30228 0.0 0.0 113148 1588 ? S Mar06 0:00 /bin/bash /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/start-jupyter vgarza 30232 0.0 0.0 353004 54456 ? S Mar06 0:00 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/jupyter-lab --no-browser --ip=casper26 mkbren 31551 0.0 0.0 113272 1660 pts/85 S+ 10:58 0:00 bash start_jupyter.sh mkbren 31598 0.0 0.0 113152 1640 pts/85 S+ 10:58 0:00 /bin/bash /glade/work/mkbren/my_npl_clone/bin/start-jupyter mkbren 31602 0.0 0.0 574588 64220 pts/85 Sl+ 10:58 0:09 /glade/work/mkbren/my_npl_clone/bin/python3.6 /glade/work/mkbren/my_npl_clone/bin/jupyter-lab --no-browser --ip=casper26 vgarza 31612 0.0 0.0 113148 1584 ? S Mar06 0:00 /bin/bash /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/start-jupyter vgarza 31616 0.0 0.0 1035504 66384 ? Sl Mar06 2:05 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/jupyter-lab --no-browser --ip=casper26 mkbren 31839 10.8 0.6 6456208 2344264 ? Ssl 10:59 18:00 /glade/work/mkbren/my_npl_clone/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/mkbren/.local/share/jupyter/runtime/kernel-e91a106f-fa63-4595-a75f-dc968683f912.json vgarza 32006 0.0 0.0 737476 45700 ? Ssl Mar06 0:19 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-ef8afca0-a6f5-4518-942b-80dda108aa2e.json vgarza 32009 0.0 0.1 1800732 584604 ? Ssl Mar06 1:18 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-8a47d30d-5842-45f0-948e-c5fe6c17b0d1.json vgarza 32012 0.1 0.3 7418124 1397100 ? Ssl Mar06 16:03 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-82961952-9747-421f-a8a9-68c83b63e875.json vgarza 32015 0.0 0.0 737476 43648 ? Ssl Mar06 0:18 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/vgarza/.local/share/jupyter/runtime/kernel-22f10abf-5985-4484-906c-1f50b4bd77f8.json vgarza 36070 0.0 0.0 113148 1124 ? S Feb28 0:00 /bin/bash /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/start-jupyter vgarza 36074 0.0 0.0 353004 2208 ? S Feb28 0:00 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/python3.6 /gpfs/u/apps/dav/opt/python/3.6.8/gnu/7.3.0/pkg-library/20190723/bin/jupyter-lab --no-browser --ip=casper26 bonnland 36686 0.0 0.0 604824 71708 pts/90 Sl 11:10 0:07 /glade/u/home/bonnland/miniconda3/bin/python3.7 /glade/u/home/bonnland/miniconda3/bin/jupyter-lab --no-browser --ip=casper26 bryan 38085 0.0 0.0 611896 52760 ? Ssl 11:13 0:01 /glade/work/bryan/miniconda3/envs/analysis/bin/python -m ipykernel_launcher -f /glade/u/home/bryan/.local/share/jupyter/runtime/kernel-f082343c-bcee-475c-b299-c61929037222.json bonnland 51662 0.0 0.0 578280 42164 ? Ssl 11:44 0:00 /glade/u/home/bonnland/miniconda3/envs/analysis/bin/python -m ipykernel_launcher -f /glade/u/home/bonnland/.local/share/jupyter/runtime/kernel-5113a544-266b-45eb-a7eb-a63f4b6d6687.json bonnland 51666 0.0 0.0 1021812 234988 ? Ssl 11:44 0:04 /glade/u/home/bonnland/miniconda3/envs/analysis/bin/python -m ipykernel_launcher -f /glade/u/home/bonnland/.local/share/jupyter/runtime/kernel-20508bd8-4627-4bdf-a21b-b2e843af6a96.json bryan 54133 0.1 0.1 1111512 393148 ? Ssl 11:48 0:13 /glade/work/bryan/miniconda3/envs/analysis/bin/python -m ipykernel_launcher -f /glade/u/home/bryan/.local/share/jupyter/runtime/kernel-f37c2161-9240-4085-92d4-ba119a9760fe.json zephyrs 81981 0.1 0.0 1022384 80516 pts/32 Sl+ 12:25 0:08 /glade/work/zephyrs/miniconda3/bin/python3.7 /glade/work/zephyrs/miniconda3/bin/jupyter-lab --no-browser --ip=casper26 wkrantz 83077 0.0 0.0 113148 1592 pts/101 S+ 12:27 0:00 /bin/bash /glade/work/wkrantz/wk_npl_casper/bin/start-jupyter wkrantz 83081 0.2 0.0 1251496 71728 pts/101 Sl+ 12:27 0:10 /glade/work/wkrantz/wk_npl_casper/bin/python3.6 /glade/work/wkrantz/wk_npl_casper/bin/jupyter-lab --no-browser --ip=casper26 wkrantz 84533 0.0 0.0 744652 43908 ? Ssl 12:29 0:00 /glade/work/wkrantz/wk_npl_casper/bin/python -m ipykernel_launcher -f /glade/u/home/wkrantz/.local/share/jupyter/runtime/kernel-d11cc587-4e87-4263-a19a-b03943e7e26a.json wkrantz 84536 0.0 0.0 744652 43900 ? Ssl 12:29 0:00 /glade/work/wkrantz/wk_npl_casper/bin/python -m ipykernel_launcher -f /glade/u/home/wkrantz/.local/share/jupyter/runtime/kernel-1b3cca21-03ba-40a2-b9bc-2c4bd8b4b376.json wkrantz 84539 0.0 0.0 744652 43904 ? Ssl 12:29 0:00 /glade/work/wkrantz/wk_npl_casper/bin/python3.6 -m ipykernel_launcher -f /glade/u/home/wkrantz/.local/share/jupyter/runtime/kernel-a963694d-7f4f-4d35-a3be-fe589734ea69.json
Matt Long (Mar 12 2020 at 19:53):
I think it's acceptable to run multiple instance of JLab; I sometimes do that when I am multi-tasking...but generally not on the same node.
Michael Levy (Mar 12 2020 at 19:59):
I think the best way to keep the login nodes free is to use execdav. I couldn't get the command to work in a single line, but did the following:
[localhost] $ ssh casper.ucar.edu [casper01] $ execdav [casper02] $ jupyter lab --no-browser --ip=$(hostname)
Then in a separate window
[localhost] $ ssh -N -L 8888:casper02:$PORT casper01
where $PORT is whatever port on casper02 got assigned to my lab. In the example above, I really did end up logged in to casper01 and then dropped on casper02 by execdav, but you may see different node names there.
Haiying Xu (Jul 13 2020 at 17:50):
I typed "ssh -N -L 8847:localhost:8889 haiyingx@casper21.ucar.edu", and typed the password, and click duo-button. But it is just hanging there and cannot log in. Do you have this problem?
Brian Bonnlander (Jul 13 2020 at 17:51):
The Token Authentication server was down for maintenance earlier today; it should be working again now.
Michael Levy (Jul 13 2020 at 18:17):
I typed "ssh -N -L 8847:localhost:8889 haiyingx@casper21.ucar.edu", and typed the password, and click duo-button. But it is just hanging there and cannot log in. Do you have this problem?
@Haiying Xu I think you want to replace localhost with the casper host: ssh -N -L 8847:casper21:8889 haiyingx@casper21.ucar.edu
Michael Levy (Jul 13 2020 at 18:19):
although I'm confused about why you would be asked for your duo confirmation - what machine did you issue the ssh call from?
Haiying Xu (Jul 13 2020 at 18:28):
Still hang. And I want to open from my local laptop, why shall I replace localhost with casper21?
Michael Levy (Jul 13 2020 at 18:34):
The way we are using the -L option, it wants an argument of the form [bind_address:]port:host:hostport; we're omitting bind_address, but that's where you would put localhost.
Also, I think "hanging" is what we expect -- the ssh command shouldn't be putting anything to stdout it's just staying open to forward the port. What happens if you open https://localhost:8847 in your browser? (edit to add the part @Kevin Paul recommends :)
Kevin Paul (Jul 13 2020 at 18:36):
...in your browser, that is.
Haiying Xu (Jul 13 2020 at 19:29):
Thanks lot, Mike. I saw your explanation and switch 8847 and 8889, and it worked.
Haiying Xu (Jul 13 2020 at 19:53):
@Michael Levy , I issued the ssh from my laptop to casper.
Last updated: Jan 30 2022 at 12:01 UTC