409

I am working with configuring Django project with Nginx and Gunicorn.

While I am accessing my port gunicorn mysite.wsgi:application --bind=127.0.0.1:8001 in Nginx server, I am getting the following error in my error log file;

2014/05/30 11:59:42 [crit] 4075#0: *6 connect() to 127.0.0.1:8001 failed (13: Permission denied) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "localhost:8080"

Below is the content of my nginx.conf file;

server {
    listen 8080;
    server_name localhost;
    access_log  /var/log/nginx/example.log;
    error_log /var/log/nginx/example.error.log;

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;
    }
}

In the HTML page I am getting 502 Bad Gateway.

What mistake am I doing?

17 Answers 17

990

Disclaimer

Make sure there are no security implications for your use-case before running this.

Answer

I had a similar issue getting Fedora 20, Nginx, Node.js, and Ghost (blog) to work. It turns out my issue was due to SELinux.

This should solve the problem:

setsebool -P httpd_can_network_connect 1

Details

I checked for errors in the SELinux logs:

sudo cat /var/log/audit/audit.log | grep nginx | grep denied

And found that running the following commands fixed my issue:

sudo cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M mynginx
sudo semodule -i mynginx.pp

Option #2 (probably more secure)

setsebool -P httpd_can_network_relay 1

https://security.stackexchange.com/questions/152358/difference-between-selinux-booleans-httpd-can-network-relay-and-httpd-can-net

References

http://blog.frag-gustav.de/2013/07/21/nginx-selinux-me-mad/
https://wiki.gentoo.org/wiki/SELinux/Tutorials/Where_to_find_SELinux_permission_denial_details
http://wiki.gentoo.org/wiki/SELinux/Tutorials/Managing_network_port_labels

13
  • 24
    thanks. I needed to yum install policycoreutils-python in order to get audit2allow first. Reference: centos.org/forums/viewtopic.php?t=5012 Nov 13, 2014 at 9:57
  • 1
    See also here. In my case I had to add nginx to the group of the user in whose home directory the wwwroot was stored.
    – user113397
    Apr 13, 2015 at 16:51
  • 2
    On Fedora 23 installing the policycoreutils-python did not provide the command audit2allow. After some research I found you should install the devel packageyum install policycoreutils-devel. Reference: danwalsh.livejournal.com/61710.html
    – Joseph N.
    Jan 31, 2016 at 13:29
  • 3
    This SHOULD be in the cherrpy and nginx documentation for many unix OSes, because I wasted 8 hours trying to figure it out after following all the docs! Dec 6, 2020 at 5:00
  • 3
    Please note that setting httpd_can_network_connect may pose as a serious security risk on your public facing server if turned on. See this and this Dec 12, 2020 at 17:56
228

I’ve run into this problem too. Another solution is to toggle the SELinux boolean value for httpd network connect to on (Nginx uses the httpd label).

setsebool httpd_can_network_connect on

To make the change persist use the -P flag.

setsebool httpd_can_network_connect on -P

You can see a list of all available SELinux booleans for httpd using

getsebool -a | grep httpd
2
  • 1
    This worked, thanks. I updated from CentOS 6.5 -> 6.7 and it must have defaulted the value to off during the update, because it was working fine before the update. Simple fix. Oct 6, 2015 at 14:04
  • 1
    Please note that setting httpd_can_network_connect may pose as a serious security risk on your public facing server if turned on. See this and this Dec 12, 2020 at 17:56
30

I have solved my problem by running my Nginx as the user I'm currently logged in with, mulagala.

By default the user as nginx is defined at the very top section of the nginx.conf file as seen below;

user nginx; # Default Nginx user

Change nginx to the name of your current user - here, mulagala.

user mulagala; # Custom Nginx user (as username of the current logged in user)

However, this may not address the actual problem and may actually have casual side effect(s).

For an effective solution, please refer to Joseph Barbere's solution.

0
17

Had a similar problem on Centos 7. When I tried to apply the solution prescribed by Sorin, I started moving in cycles. First I had a permission {write} denied. Then when I solved that I had a permission { connectto } denied. Then back again to permission {write } denied.

Following @Sid answer above of checking the flags using getsebool -a | grep httpd and toggling them I found that in addition to the httpd_can_network_connect being off. http_anon_write was also off resulting in permission denied write and permission denied {connectto}

type=AVC msg=audit(1501830505.174:799183): avc:  
denied  { write } for  pid=12144 comm="nginx" name="myroject.sock" 
dev="dm-2" ino=134718735 scontext=system_u:system_r:httpd_t:s0 
tcontext=system_u:object_r:default_t:s0 tclass=sock_file

Obtained using sudo cat /var/log/audit/audit.log | grep nginx | grep denied as explained above.

So I solved them one at a time, toggling the flags on one at a time.

setsebool httpd_can_network_connect on -P

Then running the commands specified by @sorin and @Joseph above

sudo cat /var/log/audit/audit.log | grep nginx | grep denied | 
audit2allow -M mynginx
sudo semodule -i mynginx.pp

Basically you can check the permissions set on setsebool and correlate that with the error obtained from grepp'ing' audit.log nginx, denied

16

if "502 Bad Gateway" error throws on centos api url for api gateway proxy pass on nginx , run following command to solve the issue

sudo setsebool -P httpd_can_network_connect 1
8
  1. For first see what is denied:
sudo cat /var/log/audit/audit.log | grep nginx | grep denied
type=AVC msg=audit(1618940614.934:38415): avc:  denied  { connectto } for
pid=18016 comm="nginx" path="/home/deployer/project/tmp/sockets/puma.sock" scontext=system_u:system_r:httpd_t:s0
tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
tclass=unix_stream_socket permissive=1
  1. In my case it helps on CentOS7:
sudo setenforce 0

setsebool httpd_can_network_connect on -P
setsebool httpd_can_network_relay on -P

After you can see what is enable:

getsebool -a | grep httpd
httpd_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_connect_ftp --> off
httpd_can_connect_ldap --> off
httpd_can_connect_mythtv --> off
httpd_can_connect_zabbix --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> on
httpd_can_network_memcache --> off
httpd_can_network_relay --> on
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_dbus_sssd --> off
httpd_dontaudit_search_dirs --> off
httpd_enable_cgi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_graceful_shutdown --> on
httpd_manage_ipa --> off
httpd_mod_auth_ntlm_winbind --> off
httpd_mod_auth_pam --> off
httpd_read_user_content --> off
httpd_run_ipa --> off
httpd_run_preupgrade --> off
httpd_run_stickshift --> off
httpd_serve_cobbler_files --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_sys_script_anon_write --> off
httpd_tmp_exec --> off
httpd_tty_comm --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_fusefs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off
httpd_use_openstack --> off
httpd_use_sasl --> off
httpd_verify_dns --> off
3
  • Thank you! setenforce 0 resolve my issue
    – Kamal
    Oct 6, 2021 at 13:06
  • @Kamal Ok, good! Oct 7, 2021 at 10:08
  • Please don't do this in a secure environment, turning off security policies is a band-aid fix and makes the system insecure. Nov 9, 2021 at 18:22
8
  1. Check the user in /etc/nginx/nginx.conf
  2. Change ownership to user.
sudo chown -R nginx:nginx /var/lib/nginx
0
7

I had the same issue. I tried @joebarbere solution of running the below command which fixed the issue but I was concerned about security

setsebool -P httpd_can_network_relay 1

I then came across the following solution

https://serverfault.com/questions/634294/nodejs-nginx-error-13-permission-denied-while-connecting-to-upstream

I was using non standard port and SELinux was blocking the port. Use the following command to check if the port is allowed

sudo semanage port --list | grep 4343

And added the port to allowed list

sudo semanage port --add --type http_port_t --proto tcp 4343

I restarted the nginx service, then the url was accessible. I would assume this is more secure.

2
  • thank you! what operating system you're on? I'm using CentOS Stream release 9 and this solved my issue
    – Fauzan
    Nov 3, 2023 at 12:47
  • I was using CentOS 7
    – Surakshith
    Nov 4, 2023 at 13:47
4

13-permission-denied-while-connecting-to-upstreamnginx on centos server -

setsebool -P httpd_can_network_connect 1

2

I’ve run into this problem too. I'm using Nginx with HHVM, below solution fixed my issue:

sudo semanage fcontext -a -t httpd_sys_rw_content_t "/etc/nginx/fastcgi_temp(/.*)?"

sudo restorecon -R -v /etc/nginx/fastcgi_temp
2

Another reason could be; you are accessing your application through nginx using proxy but you did not add gunicorn.sock file for proxy with gunicorn.

You need to add a proxy file path in nginx configuration.

location / {
        include proxy_params;
        proxy_pass http://unix:/home/username/myproject/gunicorn.sock;
    }

Here is a nice tutorial with step by step implementation of this

https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04#configure-nginx-to-proxy-pass-to-gunicorn

Note: if you did not created anyname.sock file you have to create if first, either use above or any other method or tutorial to create it.

1
  • I do not think you need to use a gunicorn.sock file. That is optional. You should be able to setup Nginx, Gunicorn, and Django without a gunicorn.sock if desired. Apr 18, 2021 at 7:06
0

The SELinux Policy management tool has a set of standard ports that are allowed.
Usually 8001, 5000, 4000, etc are not one of them.

Step 1

check if the port you are trying to use is allowed (or assigned to something else)

sudo semanage port --list | grep http_port_t 

Output

http_port_t  tcp  80, 81, 443, 488, 8008, 8009, 8443, 9000

You can also confirm its use (assigned to something else)

sudo semanage port --list | grep 8001

Step 2

If it is not available then you can either add it or use one of the above options

Option 1

To add the port.

sudo semanage port --add --type http_port_t --proto tcp 8001
Option 2 (I recommend)

To use one of the allowed ports (instead of adding a new one)

server {
    ....

    location / {
        proxy_pass http://127.0.0.1:9000;
        ...
    }
}

In our case, we have changed 8001 to 9000.

Note: you should also change the port on your project to 9000
For this case is gunicorn mysite.wsgi:application --bind=127.0.0.1:9000

Security Implication

I consider this more secure than the previous answers

2
  • I am running on perfect fine http_port_t 9000, however, i still have the same error
    – Wang
    Oct 9, 2023 at 3:09
  • Hey @Wang, this solution assumes all other setups on your server are good such as Nginx and project-specific settings and the only thing that is denying your requests is SELinux Policy management for denying ports. You can also visit Using NGINX and NGINX Plus with SELinux for more details.
    – melo
    Nov 9, 2023 at 7:34
0

In my case, it was fixed (from this blog) by running the command: semanage permissive -a httpd_t and restarting NGINX service.

1
  • this is an insecure fix in the long run. Better approach is to check logs, and correct the labels that are causing SELinux to behave in that way. Jun 24, 2023 at 19:31
0

I encountered a similar issue while deploying my Django application locally with Nginx as a reverse proxy. The error message (13: Permission denied) while connecting to upstream typically indicates a permission problem with accessing the upstream server, in this case, your Django app.

The solution that worked for me was related to the nginx.conf file. Inside this file, the user directive plays a significant role, as it defines the user and group that Nginx runs as. Sometimes, a comment preceding the user directive can cause permission issues.

To resolve the error, follow these steps:

  1. Locate your nginx.conf file (usually in /etc/nginx/).
  2. Find the user directive. It should look like: user <user>;.
  3. Make sure there are no comments (lines starting with #) before the user directive. Comments can sometimes disrupt the configuration parsing.
  4. Save the nginx.conf file.
  5. Restart Nginx with: sudo systemctl restart nginx.

After applying these steps, the (13: Permission denied) error should no longer occur, and your Django application should function properly when accessed through the Nginx proxy.

0

I had the same error. These commands work for me

sudo chmod +x /home/(username)/
sudo chmod +x /home/(username)/(your_project_folder)/
0

Details: You can grep in auditd.log and identify the exact reason.

In my case the problem was that nis_enabled option was set to off.

You can check the current config with:

semanage boolean --list | grep nis

And you can enable it with:

setsebool -P nis_enabled 1

Then you can try start nginx service.

-1

Do not install SELinux on Ubuntun 22.10. I did that, and I could no longer boot. Ubuntun 22.10 gave me the GRUB menu, I don't know how to recover: I ended up reinstalled Ubuntun 22.10.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.