Skip to content

HTTP 403: Forbidden error when starting jupyterhub #1377

New issue
New issue
Closed
#1996
Closed
HTTP 403: Forbidden error when starting jupyterhub#1377
#1996
Labels
configurationenhancement
@fmilano1975

Description

@fmilano1975
fmilano1975
opened on Aug 29, 2017 · edited by willingc

Hi,
I have been trying to use jupyterhub for a while now but I am struggling with a 403 error which I cannot sort out.

I am using jupyterhub version 0.8.0.b2 on fedora 25. I have installed all dependencies as described in the the installation instructions and configured jupyterhub to run on port 443. Apart from the IP of my server and the user admin name, all other options are set to the default ones in the jupyterhub_config.py file.

I can connect to the server through ssh and there is a webpage working at port 80. I have set selinux as permissive and all relevant ports should be properly set up in the firewall.

Now, whenever I run jupyterhub I get the following messages:

[root@psat jupyterhub]# jupyterhub -f jupyterhub_config.py --debug
[D 2017-08-29 17:27:42.508 JupyterHub application:555] Looking for jupyterhub_config in /etc/jupyterhub
[D 2017-08-29 17:27:42.509 JupyterHub application:577] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2017-08-29 17:27:42.521 JupyterHub app:843] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[D 2017-08-29 17:27:42.521 JupyterHub app:894] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2017-08-29 17:27:42.542 JupyterHub orm:503] database schema version found: 3ec6993fe20c
[D 2017-08-29 17:27:42.600 JupyterHub app:1216] Loading state for fmilano from db
[D 2017-08-29 17:27:42.605 JupyterHub app:1244] Loaded users: 
     fmilano admin
[I 2017-08-29 17:27:42.611 JupyterHub app:1531] Hub API listening on http://127.0.0.1:8081/hub/
[I 2017-08-29 17:27:42.612 JupyterHub proxy:457] Starting proxy @ https://137.43.92.162:443/
[D 2017-08-29 17:27:42.612 JupyterHub proxy:458] Proxy cmd: ['configurable-http-proxy', '--ip', '137.43.92.162', '--port', '443', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error', '--ssl-key', '/etc/letsencrypt/live/psat.ucd.ie/privkey.pem', '--ssl-cert', '/etc/letsencrypt/live/psat.ucd.ie/fullchain.pem']
17:27:42.891 - info: [ConfigProxy] Proxying https://137.43.92.162:443 to (no default)
17:27:42.894 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[D 2017-08-29 17:27:43.001 JupyterHub proxy:490] Proxy started and appears to be up
[D 2017-08-29 17:27:43.003 JupyterHub proxy:551] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
17:27:43.041 - warn: [ConfigProxy] 403 GET /api/routes 
[E 2017-08-29 17:27:43.040 JupyterHub app:1626]
    Traceback (most recent call last):
      File "/home/fmilano/jupyterhub/jupyterhub/app.py", line 1624, in launch_instance_async
        yield self.start()
      File "/home/fmilano/jupyterhub/jupyterhub/app.py", line 1572, in start
        yield self.proxy.check_routes(self.users, self._service_map)
      File "/home/fmilano/jupyterhub/jupyterhub/proxy.py", line 293, in check_routes
        routes = yield self.get_all_routes()
      File "/home/fmilano/jupyterhub/jupyterhub/proxy.py", line 588, in get_all_routes
        resp = yield self.api_request('', client=client)
    tornado.httpclient.HTTPError: HTTP 403: Forbidden

I have been looking for a solution for a week now (I have tried different ports, different configuration of apache, ssh, and the firewall), but I am always getting the same message and the jupyterhub exits as indicated above.

Any hint?

Thanks in advance for your help!

Activity

willingc
added
configuration
question
on Aug 30, 2017
willingc

willingc commented on Aug 30, 2017

@willingc
willingc
on Aug 30, 2017
Contributor

Hi @fmilano1975, There is now a 0.8.0b3 release. Would you mind giving that version a try? Thanks.

cc/ @minrk Do you see anything with the 443 port usage or permissions/SSL cert that would prevent connection? Or possibly an apache setting?

minrk

minrk commented on Aug 31, 2017

@minrk
minrk
on Aug 31, 2017
Member

This probably means that there is a configurable-http-proxy instance still running with an older API token. Try shutting down the hub and check if there are any:

ps aux | grep configurable-http-proxy

If there are, shut them down and try launching jupyterhub again.

willingc
added
needs: more user information
and removed
question
on Sep 1, 2017
sudheer0553
mentioned this on Oct 30, 2017
bottydim

bottydim commented on Dec 13, 2017

@bottydim
bottydim
on Dec 13, 2017

It would be nice if the service was doing that automatically on shutdown.

a-bangk

a-bangk commented on May 18, 2018

@a-bangk
a-bangk
on May 18, 2018

Is there a permanent solution for this? Our Jupyterhub restarts every night and fails to start again.

titansmc

titansmc commented on May 29, 2018

@titansmc
titansmc
on May 29, 2018 · edited by titansmc

Hi,
Since I run Jupyterhub through systemd, I have a

ExecStartPre=/var/local/anaconda3/jupyter/home/kill-configurable-http-proxy.sh

that will kill any remaining process:

#!/bin/bash
ps aux | grep "/var/local/anaconda3/bin/configurable-http-proxy" |grep -v grep|awk '{print "kill -9 " $2}'|sh

It is just a dirty fix , but it works for me.

willingc
added
enhancement
and removed
needs: more user information
on Jun 8, 2018
willingc

willingc commented on Jun 8, 2018

@willingc
willingc
on Jun 8, 2018
Contributor

I ran up against a stale configurable-http-proxy when running tests yesterday. @minrk, is there a straightforward way to do the cleanup automatically or should I document the solution you mention above to check for stale processes?

minrk

minrk commented on Jun 14, 2018

@minrk
minrk
on Jun 14, 2018
Member

If jupyterhub wrote a pid file for the proxy, it could find the previous process and clean it up on startup.

@Adam-Origamiiris how is your jupyterhub restarting every night? Do you have logs? Even if JupyterHub crashes it should shutdown CHP unless it's dying of a pretty severe error

minrk
mentioned this on Jun 18, 2018
willingc
closed this as completedin #1996on Jun 26, 2018
SamirJabari
mentioned this on Sep 26, 2018
Koshmaar

Koshmaar commented on Oct 29, 2018

@Koshmaar
Koshmaar
on Oct 29, 2018

I was getting something similar

[W 2018-10-26 13:37:16.209 JupyterHub app:1173] No admin users, admin interface will be unavailable.
[W 2018-10-26 13:37:16.210 JupyterHub app:1174] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2018-10-26 13:37:16.210 JupyterHub app:1201] Not using whitelist. Any authenticated user will be allowed.
[D 2018-10-26 13:37:16.264 JupyterHub app:1489] Loaded users:
    
[I 2018-10-26 13:37:16.281 JupyterHub app:1855] Hub API listening on http://0.0.0.0:8081/hub/
[I 2018-10-26 13:37:16.281 JupyterHub app:1857] Private Hub API connect url http://10.100.240.93:8081/hub/
[I 2018-10-26 13:37:16.281 JupyterHub app:1870] Not starting proxy
[D 2018-10-26 13:37:16.281 JupyterHub proxy:296] Fetching routes to check
[D 2018-10-26 13:37:16.323 JupyterHub proxy:686] Proxy: Fetching GET http://10.100.231.253:8001/api/routes
[E 2018-10-26 13:37:16.595 JupyterHub app:1958]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1956, in launch_instance_async
        await self.start()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1901, in start
        await self.proxy.check_routes(self.users, self._service_map)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 57, in locked_method
        return await method(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 297, in check_routes
        routes = await self.get_all_routes()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 733, in get_all_routes
        resp = await self.api_request('', client=client)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 694, in api_request
        result = await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden

I was following zero-to-jupyterhub and didn't replace secretToken='generate with openssl rand -hex 32' in config file with the output of that command.....

whlteXbread
mentioned this on Nov 4, 2018
makarov20211221

makarov20211221 commented on May 31, 2019

@makarov20211221
makarov20211221
on May 31, 2019

The configurable-http-proxy is Restarting all the time. And the Pid is changing all the time.
Anyone got a solution ?

ChandlerBang

ChandlerBang commented on Aug 7, 2019

@ChandlerBang
ChandlerBang
on Aug 7, 2019

The configurable-http-proxy is Restarting all the time. And the Pid is changing all the time.
Anyone got a solution ?

Same problem here. Anyone can help?

michaelaye

michaelaye commented on Mar 19, 2020

@michaelaye
michaelaye
on Mar 19, 2020

As this issue is closed, someone needs to make a new issue out of it.

2yongbum

2yongbum commented on Jun 17, 2021

@2yongbum
2yongbum
on Jun 17, 2021

In my case, server is behind corporate proxy and firewall and jupyterhub tries connect http://127.0.0.1:8001/api/routes using proxy setting.
Unset the proxy enviromnet and it works.

unset http_proxy
rajaprerak
mentioned this on Sep 30, 2021
engineerali2009

engineerali2009 commented on Nov 10, 2021

@engineerali2009
engineerali2009
on Nov 10, 2021

Kindly use
jupyter notebook --ip 127.0.0.1

kingdavid72

kingdavid72 commented on Nov 26, 2021

@kingdavid72
kingdavid72
on Nov 26, 2021

This probably means that there is a configurable-http-proxy instance still running with an older API token. Try shutting down the hub and check if there are any:

ps aux | grep configurable-http-proxy

If there are, shut them down and try launching jupyterhub again.

Thanks kill ^^ solved the problem

tinaok
mentioned this on Oct 4, 2022
wordbaosun

wordbaosun commented on Mar 20, 2023

@wordbaosun
wordbaosun
on Mar 20, 2023

hello，maybe ,u should reinstall configurable-http-proxy

sudo npm install -g configurable-http-proxy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    configurationenhancement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @michaelaye@minrk@willingc@Koshmaar@engineerali2009

      Issue actions
        HTTP 403: Forbidden error when starting jupyterhub · Issue #1377 · jupyterhub/jupyterhub