HTTP 403: Forbidden error when starting jupyterhub #1377

fmilano1975 · 2017-08-29T15:01:21Z

Hi,
I have been trying to use jupyterhub for a while now but I am struggling with a 403 error which I cannot sort out.

I am using jupyterhub version 0.8.0.b2 on fedora 25. I have installed all dependencies as described in the the installation instructions and configured jupyterhub to run on port 443. Apart from the IP of my server and the user admin name, all other options are set to the default ones in the jupyterhub_config.py file.

I can connect to the server through ssh and there is a webpage working at port 80. I have set selinux as permissive and all relevant ports should be properly set up in the firewall.

Now, whenever I run jupyterhub I get the following messages:

[root@psat jupyterhub]# jupyterhub -f jupyterhub_config.py --debug
[D 2017-08-29 17:27:42.508 JupyterHub application:555] Looking for jupyterhub_config in /etc/jupyterhub
[D 2017-08-29 17:27:42.509 JupyterHub application:577] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2017-08-29 17:27:42.521 JupyterHub app:843] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[D 2017-08-29 17:27:42.521 JupyterHub app:894] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2017-08-29 17:27:42.542 JupyterHub orm:503] database schema version found: 3ec6993fe20c
[D 2017-08-29 17:27:42.600 JupyterHub app:1216] Loading state for fmilano from db
[D 2017-08-29 17:27:42.605 JupyterHub app:1244] Loaded users: 
     fmilano admin
[I 2017-08-29 17:27:42.611 JupyterHub app:1531] Hub API listening on http://127.0.0.1:8081/hub/
[I 2017-08-29 17:27:42.612 JupyterHub proxy:457] Starting proxy @ https://137.43.92.162:443/
[D 2017-08-29 17:27:42.612 JupyterHub proxy:458] Proxy cmd: ['configurable-http-proxy', '--ip', '137.43.92.162', '--port', '443', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error', '--ssl-key', '/etc/letsencrypt/live/psat.ucd.ie/privkey.pem', '--ssl-cert', '/etc/letsencrypt/live/psat.ucd.ie/fullchain.pem']
17:27:42.891 - info: [ConfigProxy] Proxying https://137.43.92.162:443 to (no default)
17:27:42.894 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[D 2017-08-29 17:27:43.001 JupyterHub proxy:490] Proxy started and appears to be up
[D 2017-08-29 17:27:43.003 JupyterHub proxy:551] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
17:27:43.041 - warn: [ConfigProxy] 403 GET /api/routes 
[E 2017-08-29 17:27:43.040 JupyterHub app:1626]
    Traceback (most recent call last):
      File "/home/fmilano/jupyterhub/jupyterhub/app.py", line 1624, in launch_instance_async
        yield self.start()
      File "/home/fmilano/jupyterhub/jupyterhub/app.py", line 1572, in start
        yield self.proxy.check_routes(self.users, self._service_map)
      File "/home/fmilano/jupyterhub/jupyterhub/proxy.py", line 293, in check_routes
        routes = yield self.get_all_routes()
      File "/home/fmilano/jupyterhub/jupyterhub/proxy.py", line 588, in get_all_routes
        resp = yield self.api_request('', client=client)
    tornado.httpclient.HTTPError: HTTP 403: Forbidden

I have been looking for a solution for a week now (I have tried different ports, different configuration of apache, ssh, and the firewall), but I am always getting the same message and the jupyterhub exits as indicated above.

Any hint?

Thanks in advance for your help!

The text was updated successfully, but these errors were encountered:

willingc · 2017-08-30T12:18:34Z

Hi @fmilano1975, There is now a 0.8.0b3 release. Would you mind giving that version a try? Thanks.

cc/ @minrk Do you see anything with the 443 port usage or permissions/SSL cert that would prevent connection? Or possibly an apache setting?

minrk · 2017-08-31T21:55:55Z

This probably means that there is a configurable-http-proxy instance still running with an older API token. Try shutting down the hub and check if there are any:

ps aux | grep configurable-http-proxy

If there are, shut them down and try launching jupyterhub again.

bottydim · 2017-12-13T09:05:06Z

It would be nice if the service was doing that automatically on shutdown.

a-bangk · 2018-05-18T11:29:10Z

Is there a permanent solution for this? Our Jupyterhub restarts every night and fails to start again.

titansmc · 2018-05-29T10:06:39Z

Hi,
Since I run Jupyterhub through systemd, I have a

ExecStartPre=/var/local/anaconda3/jupyter/home/kill-configurable-http-proxy.sh

that will kill any remaining process:

#!/bin/bash
ps aux | grep "/var/local/anaconda3/bin/configurable-http-proxy" |grep -v grep|awk '{print "kill -9 " $2}'|sh

It is just a dirty fix , but it works for me.

willingc · 2018-06-08T12:17:36Z

I ran up against a stale configurable-http-proxy when running tests yesterday. @minrk, is there a straightforward way to do the cleanup automatically or should I document the solution you mention above to check for stale processes?

minrk · 2018-06-14T15:03:43Z

If jupyterhub wrote a pid file for the proxy, it could find the previous process and clean it up on startup.

@Adam-Origamiiris how is your jupyterhub restarting every night? Do you have logs? Even if JupyterHub crashes it should shutdown CHP unless it's dying of a pretty severe error

Koshmaar · 2018-10-29T14:40:32Z

I was getting something similar

[W 2018-10-26 13:37:16.209 JupyterHub app:1173] No admin users, admin interface will be unavailable.
[W 2018-10-26 13:37:16.210 JupyterHub app:1174] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2018-10-26 13:37:16.210 JupyterHub app:1201] Not using whitelist. Any authenticated user will be allowed.
[D 2018-10-26 13:37:16.264 JupyterHub app:1489] Loaded users:
    
[I 2018-10-26 13:37:16.281 JupyterHub app:1855] Hub API listening on http://0.0.0.0:8081/hub/
[I 2018-10-26 13:37:16.281 JupyterHub app:1857] Private Hub API connect url http://10.100.240.93:8081/hub/
[I 2018-10-26 13:37:16.281 JupyterHub app:1870] Not starting proxy
[D 2018-10-26 13:37:16.281 JupyterHub proxy:296] Fetching routes to check
[D 2018-10-26 13:37:16.323 JupyterHub proxy:686] Proxy: Fetching GET http://10.100.231.253:8001/api/routes
[E 2018-10-26 13:37:16.595 JupyterHub app:1958]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1956, in launch_instance_async
        await self.start()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1901, in start
        await self.proxy.check_routes(self.users, self._service_map)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 57, in locked_method
        return await method(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 297, in check_routes
        routes = await self.get_all_routes()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 733, in get_all_routes
        resp = await self.api_request('', client=client)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 694, in api_request
        result = await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden

I was following zero-to-jupyterhub and didn't replace secretToken='generate with openssl rand -hex 32' in config file with the output of that command.....

makarov20211221 · 2019-05-31T06:53:48Z

The configurable-http-proxy is Restarting all the time. And the Pid is changing all the time.
Anyone got a solution ?

ChandlerBang · 2019-08-07T01:43:46Z

The configurable-http-proxy is Restarting all the time. And the Pid is changing all the time.
Anyone got a solution ?

Same problem here. Anyone can help?

michaelaye · 2020-03-19T00:23:19Z

As this issue is closed, someone needs to make a new issue out of it.

2yongbum · 2021-06-17T11:27:02Z

In my case, server is behind corporate proxy and firewall and jupyterhub tries connect http://127.0.0.1:8001/api/routes using proxy setting.
Unset the proxy enviromnet and it works.

unset http_proxy

engineerali2009 · 2021-11-10T19:28:37Z

Kindly use
jupyter notebook --ip 127.0.0.1

kingdavid72 · 2021-11-26T09:30:40Z

This probably means that there is a configurable-http-proxy instance still running with an older API token. Try shutting down the hub and check if there are any:
ps aux | grep configurable-http-proxy
If there are, shut them down and try launching jupyterhub again.

Thanks kill ^^ solved the problem

wordbaosun · 2023-03-20T12:45:50Z

hello，maybe ,u should reinstall configurable-http-proxy

sudo npm install -g configurable-http-proxy

willingc added configuration question labels Aug 30, 2017

willingc added needs: more user information and removed question labels Sep 1, 2017

sudheer0553 mentioned this issue Oct 30, 2017

JupyterHub LDAPAuthenticator with AD is failing #1510

Closed

willingc added enhancement and removed needs: more user information labels Jun 8, 2018

minrk mentioned this issue Jun 18, 2018

use pid file to check for previous proxy instances #1996

Merged

willingc closed this as completed in #1996 Jun 26, 2018

SamirJabari mentioned this issue Sep 26, 2018

tornado.httpclient.HTTPClientError: HTTP 403: Forbidden FAU-DLM/GPU-Jupyterhub#4

Closed

whlteXbread mentioned this issue Nov 4, 2018

tornado.httpclient.HTTPClientError: HTTP 403: Forbidden #2173

Closed

rajaprerak mentioned this issue Sep 30, 2021

HTTP 403: Forbidden error when opening courses tab #3633

Closed

tinaok mentioned this issue Oct 4, 2022

CLIVER workshop : EGI checkin setup and use of foss4g jupyter-hub instance . pangeo-data/pangeo-eosc#24

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HTTP 403: Forbidden error when starting jupyterhub #1377

HTTP 403: Forbidden error when starting jupyterhub #1377

fmilano1975 commented Aug 29, 2017 •

edited by willingc

Loading

willingc commented Aug 30, 2017

Uh oh!

minrk commented Aug 31, 2017

Uh oh!

bottydim commented Dec 13, 2017

Uh oh!

a-bangk commented May 18, 2018

Uh oh!

titansmc commented May 29, 2018 •

edited

Loading

Uh oh!

willingc commented Jun 8, 2018

Uh oh!

minrk commented Jun 14, 2018

Uh oh!

Koshmaar commented Oct 29, 2018

Uh oh!

makarov20211221 commented May 31, 2019

Uh oh!

ChandlerBang commented Aug 7, 2019

Uh oh!

michaelaye commented Mar 19, 2020

Uh oh!

2yongbum commented Jun 17, 2021

Uh oh!

engineerali2009 commented Nov 10, 2021

Uh oh!

kingdavid72 commented Nov 26, 2021

Uh oh!

wordbaosun commented Mar 20, 2023

Uh oh!

HTTP 403: Forbidden error when starting jupyterhub #1377

HTTP 403: Forbidden error when starting jupyterhub #1377

Comments

fmilano1975 commented Aug 29, 2017 • edited by willingc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

willingc commented Aug 30, 2017

Uh oh!

minrk commented Aug 31, 2017

Uh oh!

bottydim commented Dec 13, 2017

Uh oh!

a-bangk commented May 18, 2018

Uh oh!

titansmc commented May 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

willingc commented Jun 8, 2018

Uh oh!

minrk commented Jun 14, 2018

Uh oh!

Koshmaar commented Oct 29, 2018

Uh oh!

makarov20211221 commented May 31, 2019

Uh oh!

ChandlerBang commented Aug 7, 2019

Uh oh!

michaelaye commented Mar 19, 2020

Uh oh!

2yongbum commented Jun 17, 2021

Uh oh!

engineerali2009 commented Nov 10, 2021

Uh oh!

kingdavid72 commented Nov 26, 2021

Uh oh!

wordbaosun commented Mar 20, 2023

Uh oh!

fmilano1975 commented Aug 29, 2017 •

edited by willingc

Loading

titansmc commented May 29, 2018 •

edited

Loading