It looks like the WebSocket connection between GCP Dataproc JupyterLab and LSP server (python-lsp-server) is blocked. Is there any way to customise initialisation actions on Dataproc cluster to forward this WebSocket connection to the gateway URL (or any other way to make the connection work)?
JupyterLab WSS connection failure
My Dataproc cluster is running fine using 2.0.45-debian10 image with Jupyter as an additional component installed. The cluster automatically gives me the Component Gateway interface for JupyterLab, and the access is working fine with all kernels working.
I am now trying to activate the autocomplete/hinting on the JupyterLab notebook, with softwareConfig.properties['dataproc:pip.packages']: 'jupyterlab-lsp==3.9.3,python-lsp-server==1.5.0' properties added to initialise the cluster with the relevant lsp packages. Packages are working fine as well and the Dataproc correctly builds JupyterLab with the lsp activated.
But when I am trying to use the hinting in my JupyterLab notebook through component gateway, LSP server pane hangs at initializing and showing that the notebook is not connected to pyls/pylsp server. Chrome dev console shows that it is trying to connect to wss://.../gateway/default/jupyter/lsp/ws/pyls and connection drops.
I suspect this is because the lsp WebSocket connection is not forwarded through the component gateway - when I use SOCKS5 proxy to connect to the notebook using cluster web interface instruction, this works fine.
Any clue how this could work without me/team needing to make a manual SSH connection into the cluster?
