Tip for troubleshooting rserver startup problems

[agordon]

Hello,

Thanks for the awesome work on jupyter-resession-proxy.

After many hours of troubleshooting and debugging, perhaps I can save some time for future users:

-1-
The rserver/rsession process requires access to some system-wide directories.
If using a JupyterHub configuration with SystemdSpawner, and isolating paths - that could prove to be problematic.

Specifically, if your jupyterhub_config.py has something like this:

c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
c.SystemdSpawner.isolate_tmp = True
c.SystemdSpawner.isolate_devices = True
c.SystemdSpawner.disable_user_sudo = True
c.SystemdSpawner.readonly_paths = ['/']
c.SystemdSpawner.readwrite_paths = ['/home/{USERNAME}','/var/run']
c.SystemdSpawner.dynamic_users = False
c.SystemdSpawner.cpu_limit = 1

The readonly_paths =['/'] will cause the rserver/rsession process to fail to start.
Similarly, it seems the rserver/rsession needs read access to /proc and read/write access to /var/run and /var/log
Any attempts to hide/block these folders will again result in rserver/rsession process to fail.
For example, the following will cause it to fail:

c.SystemdSpawner.unit_extra_properties = {
        'InaccessiblePaths' : [
             '/proc/',
             '/var/log',
             '/root/',
             '/sys/',
             '/var/www'
             ]
        }

(If you wonder why would anyone use such restrictive settings, it is because they add one small additional layer of protection on a shared system, and most jupyter kernels work just fine with them).

-2-
When encountering jupyterhub+rstudio start up problems (the dreaded 404 or "timeout errors),
the first place to look is in the logs. If starting jupyterhub from systemd, then journalctl -xe will show the error messages.
In my case it was completely unhelpful rserver[12354]: ... Error 13: permission denied but without showing which file access is denied.

If you need to escalate the troubleshooting and see exactly what is being access/denied, you can replace the rserver binary with a wrapper script, like so:

cd /usr/lib/rstudio-server/bin
mv rserver rserver.bin
cat<<'EOF'>rserver
#!/bin/sh
log=$(mktemp /tmp/rserver.strace.XXXXXXX.log)
echo "=== RSERVER DEBUG, strace log file: $log ==="
echo "args: $@"
echo "=== env: "
env
echo ===

strace -f -e trace=file -o "$log" /usr/lib/rstudio-server/bin/rserver.bin "$@"
EOF

Then, the journalctl log will show the parameters used to start rserver, and the name of the temporary log file.
In that log file (/tmp/rserver.strace.XXXXXX.log), grep for EPERM and EACCES to find the blocked files.
NOTE: if the strace log file is stored in /tmp, ensure that c.SystemdSpawner.isolate_tmp is set to False in the jupyterhub config file during the troubleshooting session - otherwise it will disappear.

This is certainly a low-level technical method, but if all else fails, it'll provide some hints as to what the problem is.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[ryanlovett]

Thanks @agordon ! I've definitely debugged rserver the way you did in the second section. :) Instead of moving the executables, you can put the wrapper script somewhere earlier in PATH than rserver.

I feel like this should be preserved in documentation somehow. Perhaps the first section should be put in README.md and the second section should be put in CONTRIBUTING.md? If so, care to submit a PR?