NGINX log files permission issue

As far as the .1, there should also be a .0 file. This is log rotation. The owner should be forge with a group of adm in a Forge initialized server. If you are SSH in as forge user, then you should have access to the contents of the log files in /var/log/nginx/.

Could the access.log or the access.log file be the .0 files? Because I have no files that have a .0, I forced logrotate and it created a .1 file then a .2

It depends on your nginx version I believe. If somewhat current, then the file will just be access.log until it is rotated. If you examine the log directory, you will see similar footprints with older log files being g-zipped for storage. Have you considered Papertrail to help read and sort your logs on many servers? It works great, easy setup and is inexpensive.

I've been going back and forth with the Forge team regarding a LetsEncrypt SSL install issue and one of the things they got back to me with was this:

Also we've noticed that there's a permission issue in your log files where they now belong to a group called adm instead of forge, this is causing writing to the log files to fail:

open() "/var/log/nginx/access.log" failed (13: Permission denied)

I did not make any permissions changes to the server so I'm not sure why this is happening but it seems like it coincides with what you guys are discussing here. When I visit /var/log I can see that /var/log/nginx is owned by www-data and group adm. Any idea why or how this could have changed from forge user to www-data and how I can fix it?

Hmm, I have not experienced that issue on any of my servers. I still have forge as the owner and adm as the group.

Not sure if this has anything to do with it, but this is the contents of my /etc/logrotate.d/nginx file:

/var/log/nginx/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}

Is that what's causing the permissions issues? Is that what yours looks like too?

@ejdelmonico the log files are getting zipped up and rotated, but it's like @tptompkins explained, they're being created under www-data adm.

NGINX is logging to access.log.1 then zipping it but it still creates an empty access.log.

What I did notice is that the access.log.1 file is owned by root in the group root. Is it the same in your case @tptompkins?

I do have the same configuration in my /etc/logrotate.d/nginx. I do suspect that www-data and adm don't have permission, but it makes no sense to why that's the default configuration and as to how the access.log.1 has a different owner and group.

@ejdelmonico thank you for the suggestion. I will have a look at Papertrail. At the moment I am using Elasticsearch to gather log files, pares them and create a monitoring system, which is why I'm facing this problem.

I am going to try and change the owner and group and see what happens. My only concern is security if I changed it to root or forge.

For some reason CloudFlare on Laracasts is blocking me when I try to post this so trying again..

@nmaptomyheartbeat My /var/log/nginx directory permissions are listed as www-data adm but all of the files inside of it are owned by forge adm. I also have an empty access.log file and all the logging seems to be going to access.log.1:

-rw-r----- 1 forge adm      0 Jan 28 06:25 access.log
-rw-r----- 1 forge adm  31781 Jan 29 10:53 access.log.1
-rw-r----- 1 forge adm   8267 Dec 17 12:40 access.log.10.gz
-rw-r----- 1 forge adm  13975 Dec 10 12:20 access.log.11.gz
-rw-r----- 1 forge adm  30125 Dec  3 12:23 access.log.12.gz
-rw-r----- 1 forge adm  17846 Nov 26 11:37 access.log.13.gz
-rw-r----- 1 forge adm  22509 Nov 19 11:33 access.log.14.gz
-rw-r----- 1 forge adm   5076 Jan 27 11:25 access.log.2.gz
-rw-r----- 1 forge adm   2350 Jan 26 22:10 access.log.3.gz
-rw-r----- 1 forge adm   6260 Jan 25 14:05 access.log.4.gz
-rw-r----- 1 forge adm   8748 Jan 21 12:50 access.log.5.gz
-rw-r----- 1 forge adm  12538 Jan 14 10:31 access.log.6.gz
-rw-r----- 1 forge adm   9617 Jan  7 12:46 access.log.7.gz
-rw-r----- 1 forge adm   8589 Dec 31 12:39 access.log.8.gz
-rw-r----- 1 forge adm   9029 Dec 24 11:41 access.log.9.gz
-rw-r----- 1 forge adm      0 Jan 26 06:25 default-error.log
-rw-r----- 1 forge adm    837 Jan 26 06:21 default-error.log.1
-rw-r----- 1 forge adm    717 Dec  3 11:59 default-error.log.10.gz
-rw-r----- 1 forge adm    715 Nov 26 11:37 default-error.log.11.gz
-rw-r----- 1 forge adm    499 Nov 17 04:59 default-error.log.12.gz
-rw-r----- 1 forge adm    544 Nov 12 08:26 default-error.log.13.gz
-rw-r----- 1 forge adm    753 Nov  5 00:32 default-error.log.14.gz
-rw-r----- 1 forge adm    344 Jan 23 23:10 default-error.log.2.gz
-rw-r----- 1 forge adm    812 Jan 20 19:17 default-error.log.3.gz
-rw-r----- 1 forge adm    588 Jan 14 03:22 default-error.log.4.gz
-rw-r----- 1 forge adm    565 Jan  5 20:40 default-error.log.5.gz
-rw-r----- 1 forge adm    554 Dec 29 02:28 default-error.log.6.gz
-rw-r----- 1 forge adm    568 Dec 23 21:44 default-error.log.7.gz
-rw-r----- 1 forge adm    738 Dec 16 23:11 default-error.log.8.gz
-rw-r----- 1 forge adm    605 Dec 10 05:18 default-error.log.9.gz
-rw-r----- 1 forge adm      0 Jan 28 06:25 error.log
-rw-r----- 1 forge adm   1195 Jan 28 06:25 error.log.1
-rw-r----- 1 forge adm    227 Dec 12 06:25 error.log.10.gz
-rw-r----- 1 forge adm    205 Dec  4 06:25 error.log.11.gz
-rw-r----- 1 forge adm    234 Nov 29 06:25 error.log.12.gz
-rw-r----- 1 forge adm    234 Nov 21 06:25 error.log.13.gz
-rw-r----- 1 forge adm    235 Nov 14 06:25 error.log.14.gz
-rw-r----- 1 forge adm   1476 Jan 27 06:25 error.log.2.gz
-rw-r----- 1 forge adm    203 Jan 26 06:25 error.log.3.gz
-rw-r----- 1 forge adm    230 Jan 23 06:25 error.log.4.gz
-rw-r----- 1 forge adm    206 Jan 15 06:25 error.log.5.gz
-rw-r----- 1 forge adm    230 Jan 10 06:25 error.log.6.gz
-rw-r----- 1 forge adm    214 Jan  1 06:25 error.log.7.gz
-rw-r----- 1 forge adm    231 Dec 27 06:25 error.log.8.gz
-rw-r----- 1 forge adm    230 Dec 19 06:25 error.log.9.gz

By any chance, have you performed system updates and did not keep the modified config when asked by apt? This could cause nginx or anything else to change permissions and ownership to the defaults. It seems like that is very likely what happened.

@ejdelmonico I provisioned this server back in Sept of 2016 using Forge and I haven't touched it since. Any updates that were performed were done through it's own self-updating mechanism.

@ejdelmonico Not sure if this has anything to do with hit, but the only change that I did make was update the LetsEncrypt SSL through Forge. This issue was brought to my attention by the Forge team when I ran into errors trying to re-install LetsEncrypt. The only reason why I tried to reinstall the LetsEncrypt certificate was because I noticed the other day that the LetsEncrypt config in Forge had a warning on the screen that said that I couldn't use "default" for a site name with LetsEncrypt even though this is how it was always setup from the very beginning using Forge and my SSL was still working fine. Taylor asked me to update the site name on the Meta tab in Forge so I did, but my SSL still didn't show up on the LetsEncrypt tab so I clicked to install a new one. Once I did that, I got a bunch of errors and my site stopped loading, I contacted Forge support and that's when they told me about the log file permission issue. I ended up fixing the LetsEncrypt issue because my Nginx config file in Forge had TWO server {} sections. Forge support thought that I mistakenly added the second one but I didn't. I suspect that maybe this was added automatically to the Nginx config by Forge when it tried to reinstall the SSL and that's when my site went down. Might all this be related???

@tptompkins I checked several of my Forge servers and the only difference in permissions is /var/log/nginx is root adm which makes sense. As far as the default site, you never want to use that inForge unless you know you won't add another site or subdirectory and you won't need an ssl cert. Also, the nginx install should be root root in /etc.

Also, check your /etc/nginx/nginx.confline 1 and make sure you have user forge.

@nmaptomyheartbeat You shouldn't have a problem with changing the the permissions because your SSH config should not allow root to log in unless you changed the standard Forge configuration. They would have to steal your ssl key and log in as forge and also have the root password to escalate privileges to root.

@ejdelmonico

Also, the nginx install should be root root in /etc.

Yep, it's root root.

Also, check your /etc/nginx/nginx.conf line 1 and make sure you have user forge.

Yep, I have that too.

So it appears that the only difference is the permissions on /var/log/nginx are set to www-data adm. Do you think it'd be okay to just change this to root adm? If so, would I be able to access the log files with the forge user instead of having to su to root?

Should I also manually update the /etc/logrotate.d/nginx file and update the create 0640 www-data adm line to create 0640 root adm?

@tptompkins Those are good questions, let me look around for areas that are affected by a change and the owner of the file should be able to access the log file. I use the forge user to access mine. Of course, forge is the owner. Somehow, you ended up with the default permissions for nginx.

As noted in the nginx.conf file, the user is forge so the logs should be available to the forge user so I think changing the user might work but I would make a snapshot before any changes.

Do not change the permissions on /etc/logrotate.d/ unless they are not root root. One interesting thing though, the forge user does belong to the www-data group as well as the forge sudo lxd groups.

Looking at what you guys wrote, I tried seeing what effect changing the user and group would have. I tried to alternate between www-data, forge and root on two different servers. I did find that the version of nginx changes the outcome.

I logged in as forge. One server had nginx 1.9.13 which didn't allow me to access /var/log/nginx unless I used sudo. While the other had nginx 1.6.2 and that allowed me to access the log files without sudo.

Strangely enough, I updated nginx to the latest version on one of the servers to 1.13.6. So in /etc/nginx/nginx.conf the user at line 1 changed from forge to www-data and the user and owner for /var/log/nginx changed from the default www-data :adm (also from when I changed it) to root :adm.

I'm gonna leave it like this for a bit and see if the same error still persists.

@tptompkins Have you updated the /etc/logrotate.d/nginx file and updatedcreate 0640 www-data adm line to create 0640 root adm?

Has it solved the problem? It does seem like this is the problem. However, I do also have my security concerns. Wonder if @ejdelmonico has found anything regarding that.

Also thank you very much gentlemen for your help on this subject.

@nmaptomyheartbeat No, I haven't made any changes. I sent some questions over to the Forge team a few weeks ago about how to handle this and I never received a reply back.

@tptompkins I do hope you reply soon. I would love to solve this problem once and for all.