Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP 403: Forbidden error when starting jupyterhub #1377

Closed
fmilano1975 opened this issue Aug 29, 2017 · 15 comments · Fixed by #1996
Closed

HTTP 403: Forbidden error when starting jupyterhub #1377

fmilano1975 opened this issue Aug 29, 2017 · 15 comments · Fixed by #1996

Comments

@fmilano1975
Copy link

fmilano1975 commented Aug 29, 2017

Hi,
I have been trying to use jupyterhub for a while now but I am struggling with a 403 error which I cannot sort out.

I am using jupyterhub version 0.8.0.b2 on fedora 25. I have installed all dependencies as described in the the installation instructions and configured jupyterhub to run on port 443. Apart from the IP of my server and the user admin name, all other options are set to the default ones in the jupyterhub_config.py file.

I can connect to the server through ssh and there is a webpage working at port 80. I have set selinux as permissive and all relevant ports should be properly set up in the firewall.

Now, whenever I run jupyterhub I get the following messages:

[root@psat jupyterhub]# jupyterhub -f jupyterhub_config.py --debug
[D 2017-08-29 17:27:42.508 JupyterHub application:555] Looking for jupyterhub_config in /etc/jupyterhub
[D 2017-08-29 17:27:42.509 JupyterHub application:577] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2017-08-29 17:27:42.521 JupyterHub app:843] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[D 2017-08-29 17:27:42.521 JupyterHub app:894] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2017-08-29 17:27:42.542 JupyterHub orm:503] database schema version found: 3ec6993fe20c
[D 2017-08-29 17:27:42.600 JupyterHub app:1216] Loading state for fmilano from db
[D 2017-08-29 17:27:42.605 JupyterHub app:1244] Loaded users: 
     fmilano admin
[I 2017-08-29 17:27:42.611 JupyterHub app:1531] Hub API listening on http://127.0.0.1:8081/hub/
[I 2017-08-29 17:27:42.612 JupyterHub proxy:457] Starting proxy @ https://137.43.92.162:443/
[D 2017-08-29 17:27:42.612 JupyterHub proxy:458] Proxy cmd: ['configurable-http-proxy', '--ip', '137.43.92.162', '--port', '443', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error', '--ssl-key', '/etc/letsencrypt/live/psat.ucd.ie/privkey.pem', '--ssl-cert', '/etc/letsencrypt/live/psat.ucd.ie/fullchain.pem']
17:27:42.891 - info: [ConfigProxy] Proxying https://137.43.92.162:443 to (no default)
17:27:42.894 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[D 2017-08-29 17:27:43.001 JupyterHub proxy:490] Proxy started and appears to be up
[D 2017-08-29 17:27:43.003 JupyterHub proxy:551] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
17:27:43.041 - warn: [ConfigProxy] 403 GET /api/routes 
[E 2017-08-29 17:27:43.040 JupyterHub app:1626]
    Traceback (most recent call last):
      File "/home/fmilano/jupyterhub/jupyterhub/app.py", line 1624, in launch_instance_async
        yield self.start()
      File "/home/fmilano/jupyterhub/jupyterhub/app.py", line 1572, in start
        yield self.proxy.check_routes(self.users, self._service_map)
      File "/home/fmilano/jupyterhub/jupyterhub/proxy.py", line 293, in check_routes
        routes = yield self.get_all_routes()
      File "/home/fmilano/jupyterhub/jupyterhub/proxy.py", line 588, in get_all_routes
        resp = yield self.api_request('', client=client)
    tornado.httpclient.HTTPError: HTTP 403: Forbidden

I have been looking for a solution for a week now (I have tried different ports, different configuration of apache, ssh, and the firewall), but I am always getting the same message and the jupyterhub exits as indicated above.

Any hint?

Thanks in advance for your help!

@willingc
Copy link
Contributor

Hi @fmilano1975, There is now a 0.8.0b3 release. Would you mind giving that version a try? Thanks.

cc/ @minrk Do you see anything with the 443 port usage or permissions/SSL cert that would prevent connection? Or possibly an apache setting?

@minrk
Copy link
Member

minrk commented Aug 31, 2017

This probably means that there is a configurable-http-proxy instance still running with an older API token. Try shutting down the hub and check if there are any:

ps aux | grep configurable-http-proxy

If there are, shut them down and try launching jupyterhub again.

@bottydim
Copy link

It would be nice if the service was doing that automatically on shutdown.

@akjems
Copy link

akjems commented May 18, 2018

Is there a permanent solution for this? Our Jupyterhub restarts every night and fails to start again.

@titansmc
Copy link

titansmc commented May 29, 2018

Hi,
Since I run Jupyterhub through systemd, I have a

ExecStartPre=/var/local/anaconda3/jupyter/home/kill-configurable-http-proxy.sh

that will kill any remaining process:

#!/bin/bash
ps aux | grep "/var/local/anaconda3/bin/configurable-http-proxy" |grep -v grep|awk '{print "kill -9 " $2}'|sh

It is just a dirty fix , but it works for me.

@willingc
Copy link
Contributor

willingc commented Jun 8, 2018

I ran up against a stale configurable-http-proxy when running tests yesterday. @minrk, is there a straightforward way to do the cleanup automatically or should I document the solution you mention above to check for stale processes?

@minrk
Copy link
Member

minrk commented Jun 14, 2018

If jupyterhub wrote a pid file for the proxy, it could find the previous process and clean it up on startup.

@Adam-Origamiiris how is your jupyterhub restarting every night? Do you have logs? Even if JupyterHub crashes it should shutdown CHP unless it's dying of a pretty severe error

@Koshmaar
Copy link

I was getting something similar

[W 2018-10-26 13:37:16.209 JupyterHub app:1173] No admin users, admin interface will be unavailable.
[W 2018-10-26 13:37:16.210 JupyterHub app:1174] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2018-10-26 13:37:16.210 JupyterHub app:1201] Not using whitelist. Any authenticated user will be allowed.
[D 2018-10-26 13:37:16.264 JupyterHub app:1489] Loaded users:
    
[I 2018-10-26 13:37:16.281 JupyterHub app:1855] Hub API listening on http://0.0.0.0:8081/hub/
[I 2018-10-26 13:37:16.281 JupyterHub app:1857] Private Hub API connect url http://10.100.240.93:8081/hub/
[I 2018-10-26 13:37:16.281 JupyterHub app:1870] Not starting proxy
[D 2018-10-26 13:37:16.281 JupyterHub proxy:296] Fetching routes to check
[D 2018-10-26 13:37:16.323 JupyterHub proxy:686] Proxy: Fetching GET http://10.100.231.253:8001/api/routes
[E 2018-10-26 13:37:16.595 JupyterHub app:1958]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1956, in launch_instance_async
        await self.start()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1901, in start
        await self.proxy.check_routes(self.users, self._service_map)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 57, in locked_method
        return await method(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 297, in check_routes
        routes = await self.get_all_routes()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 733, in get_all_routes
        resp = await self.api_request('', client=client)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 694, in api_request
        result = await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden

I was following zero-to-jupyterhub and didn't replace secretToken='generate with openssl rand -hex 32' in config file with the output of that command.....

@makarov20211221
Copy link

The configurable-http-proxy is Restarting all the time. And the Pid is changing all the time.
Anyone got a solution ?

@ChandlerBang
Copy link

The configurable-http-proxy is Restarting all the time. And the Pid is changing all the time.
Anyone got a solution ?

Same problem here. Anyone can help?

@michaelaye
Copy link

As this issue is closed, someone needs to make a new issue out of it.

@2yongbum
Copy link

In my case, server is behind corporate proxy and firewall and jupyterhub tries connect http://127.0.0.1:8001/api/routes using proxy setting.
Unset the proxy enviromnet and it works.

unset http_proxy

@engineerali2009
Copy link

Kindly use
jupyter notebook --ip 127.0.0.1

@kingdavid72
Copy link

This probably means that there is a configurable-http-proxy instance still running with an older API token. Try shutting down the hub and check if there are any:

ps aux | grep configurable-http-proxy

If there are, shut them down and try launching jupyterhub again.

Thanks kill ^^ solved the problem

@wordbaosun
Copy link

hello,maybe ,u should reinstall configurable-http-proxy

sudo npm install -g configurable-http-proxy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.