Setting up logrotate in RHEL

Setting up logrotate in RHEL

So on today’s fun things to deal with, I had a teammate come to me with a problem with one of the servers. It seems nginx just refused to startup.

Well it turns out that the reason Nginx wouldn’t start, was because the hard drive was completely full, and there was no space left on the device.

But how did I figure this out?

1. Check the systemctl status for the service

The first problem to contend with was figuring out why nginx refused to start. So I did the obligatory look into the systemctl status to see what was going on.

This didn’t really give me much info as the service was just dead.

2. Check the nginx logs

I then went to check out the nginx logs.

So it seems there are fresh error logs to look at. This should give me something.

The clue here is “No space left on device”; whoops,  the disk is out of space.

3. See where the space is being used up.

So now that I know there is no space left on the disk, I can start to see where the true problem is. I need to find out what is taking up all the disk space.

Luckily the df command is there for us. I use it with the -h option to show the size in human readable format.

And there it is. The var partition is completely used up. Now I need to figure out which file or directory it is.

I can use the Disk Usage du command to figure out what folder it is.

And bingo. It looks like app1 has a huge log.

And here we are. One huge log. So now I know the problem.  I ended up having to truncate the entire file and start it over.

Now that this is taken care of, and we have space on the drive, I was able to start nginx and get app1 back up and running.

4. Future proofing with logrotate 

Now that I know the problem, it’s time for the solution. Logrotate the files, and get rid of them before they take up to much space. So I create the logrotate file and let it do it’s thing. I am not writing a full logrotate tutorial here, but if you are looking for a good one, I might suggest the following site:

HowTo: The Ultimate Logrotate Command Tutorial with 10 Examples

Until next time.