...

First, from the head node (gc-prd-hpclogin1), request an interactive session on a compute node. The command below requests 1 CPU-core with 4 GB of memory for 1 hour:

qsub -I -q workq  -l select=1:ncpus=1:mem=4gb,walltime=1:00:00

Once the node has been allocated, run the hostname command to get the name of the node (e.g

gc-prd-hpcn002) 

.You should not be in the login node if you are inside the job.
On the compute node:

module load anaconda3/2021.11

source activate myenv  #e.g source activate s123456-tf-cpu

jupyter-notebook --no-browser --port=8889 --ip=0.0.0.0

or
jupyter-lab --no-browser --port=8889 --ip=0.0.0.0
#note the last line of the output which will be something like
http://127.0.0.1:8889/?token=61f8a2aa8ad5e469d14d6a1f59baac05a8d9577916bd7eb0
# leave the session running

Next, start a second terminal session on your local machine (e.g., laptop) and setup the tunnel as follows:

ssh -N -f -L 8889:gc-prd-hpcn002:8889 snumber@gc-prd-hpclogin1.rcs.griffith.edu.au

In the command above, be sure to replace gc-prd-hpcn002 with the hostname of the node that pbs assigned to you.

Note that we selected the Linux port 8889 to connect to the notebook. If you don't specify the port, it will default to port 8888 but sometimes this port can be already in use either on the remote machine or the local one (i.e., your laptop). If the port you selected is unavailable, you will get an error message, in which case you should just pick another one. It is best to keep it greater than 1024. Consider starting with 8888 and increment by 1 if it fails, e.g., try 8888, 8889, 8890 and so on. If you are running on a different port then substitute your port number for 8889.

Lastly, open a web browser and copy and paste the URL from the previous output:

http://127.0.0.1:8889/?token=61f8a2aa8ad5e469d14d6a1f59baac05a8d9577916bd7eb0

Choose "New" then "Python 3" to launch a new notebook. Note that Jupyter may use a port that is different than the one you specified. This is why it is import to copy and paste the URL.  When you are done, terminate the ssh tunnel on your local machine by running lsof -i tcp:8889 to get the PID and then kill -9 <PID> (e.g., kill -9 6010).

Aside on ssh

Looking at the man page for ssh, the relevant flags are:

-N  Do not execute a remote command. This is useful for just forwarding ports.

-f  Requests ssh to go to background just before command execution. This is useful if ssh is
going to ask for passwords or passphrases, but the user wants it in the background.

-L  Specifies that the given port on the local (client) host is to be forwarded to the given
host and port on the remote side

Aside on Open Ports

Jupyter will automatically find an open port if you happen to specify one that is occupied. If you wish to do the scanning yourself then run the command below:

netstat -antp | grep :88 | sort

Reference

1. https://jupyter.org/try
2. https://researchcomputing.princeton.edu/support/knowledge-base/jupyter