How to secure and protect Nginx on Linux with Fail2ban

When I managed to deploy an ASP.NET core app on Linux with Nginx, I noticed a lot of rogue and spam internet traffic in the Nginx logs. Mostly mild hacking attempts and rather a lot of php requests. Its mostly noise, but I wanted a way of banning these ip addresses as they are no good and this is how I came across using Fail2ban.

What is Fail2ban?

In short, its a tool that scans log files based on a defined regular expression. If it matches, it will perform an action on it. Fail2ban can scan many different types of logs such as Nginx, Apache and ssh logs. Based on matches, it is able to ban ip addresses for a configured time period.

Installing Fail2ban doesn't take long especially if you're familiar with the Linux eco-system. Its setting up and making adjustments to your specifications that may take some time. It all depends on your needs.

Summary

  • Install Fail2ban.
  • Configure Fail2ban defaults.
  • Add our first jail rule so it can monitor Nginx logs.
  • Testing it all works.

Install Fail2ban

  1. Update the list of packages on Ubuntu by running the following command.
sudo apt-get update
  1. Install Fail2ban
sudo apt-get install fail2ban

Done. That was the easy part. Now its time to configure it.

Configure Fail2ban defaults

  1. Make a copy of the original jail.conf so that it can be modified.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
  1. Now, lets modify the file:
nano /etc/fail2ban/jail.local
  1. Modify the ignoreip so that it ignores your host ip address. This should be your public ip address.
[DEFAULT]

#
# MISCELLANEOUS OPTIONS
#

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8 your-ip-address-here

If your public ip address was 54.15.55.01 then it should look something like this:

ignoreip = 127.0.0.1/8 54.15.55.01
  1. Modify default bantime and findtime and maxretry. I've set it to permanent with a minimum retry time as shown below:
# "bantime" is the number of seconds that a host is banned.
bantime  = -1 # this is permanant

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 1

# "maxretry" is the number of failures before a host get banned.
maxretry = 1

Configuring Jails in Fail2ban so it can monitor Docker Nginx Logs

Within the file /etc/fail2ban/jail.local there is a section called Jails. This is where pre-defined rules are set for Fail2ban so that it knows what kind of filter to apply and where to monitor log files. First we need to understand how docker files are logged and where.

How to find out where docker logs are located

Docker log files are generally logged as json and in order to find out where docker log files are located, simply run the docker inspect [containername] command like below

docker inspect nginx-web

And the result will be something like this:

...
"HostsPath": "/var/lib/docker/containers/94dc5a3**/hosts",
"LogPath": "/var/lib/docker/containers/94dc5a3**/94dc5a3**-json.log",
"Name": "/nginx-web"
...

Notice that the LogPath is where docker logs file for that particular container.

Creating our first Jail

Now we have the logpath, we can create our first jail.
The jail below is for Nginx to deny any requests that attempt to request for types such as .exe .php etc.

[nginx-noscript]
enabled   = true
port      = http,https
filter    = nginx-noscript
logpath   = /var/lib/docker/containers/*/*-json.log 

Notice that the enabled flag is set to true and the logpath has been set but with a wildcard path for the docker logs. Now you can be specific, but be warned, each time a docker container restarts, it will be located in a different folder within containers.

Create the corresponding filter

Now, create a file if one does not exist already called /etc/fail2ban/filter.d/nginx-noscript.conf. The filename must match the filter name specified in the jail.

nano /etc/fail2ban/filter.d/nginx-noscript.conf

Below is the regular expression. Notice that it has a prefix of ^{"log". The reason for this is that docker logs in json format.

[Definition]

failregex = ^{"log":"<HOST> -.*GET.*(\.php|\.asp|\.exe|\.pl|\.cgi|\.scgi)

ignoreregex =

Restart Fail2ban

Any changes made in jail.local a restart of Fail2ban is required. Below are commands which will restart the service.

service fail2ban restart

fail2ban-client reload

If there is anything wrong with the fail or filter, an error would be reported.

Testing filters

To ensure that filters are working as expected with the defined regular expression, Fail2ban offers two ways to test that they are working.

Option 1 - Specify the log file and the filter file

The fail2ban-regex command line tool offers a way of testing if a regex is working as expected.

You can use it as follows:

fail2ban-regex <logpath> <filterpath>

Here is a sample of it in use.

fail2ban-regex /var/lib/docker/containers/7c2442*/*-json.log /etc/fail2ban/filter.d/nginx-noscript.conf

And a sample output report is shown below.

Running tests
=============

Use   failregex filter file : nginx-noscript, basedir: /etc/fail2ban
Use         log file : /var/lib/docker/containers/94dc5*/94dc5*-json.log
Use         encoding : UTF-8


Results
=======

Failregex: 233 total
|-  #) [# of hits] regular expression
|   1) [233] ^{"log":"<HOST> -.*GET.*(\.php|\.asp|\.exe|\.pl|\.cgi|\.scgi)
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [724] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 724 lines, 0 ignored, 233 matched, 491 missed
[processed in 0.10 sec]

Notice towards the end of the report, it states 233 matched, 491 missed. This means that the filter was able to match 233 lines and 491 were not applicable. This is a good indication that the regex is matching and working.

Option 2 - Specify inline

The fail2ban-regex also offers a way to test inline rather then specifying a log file.

Below is the syntax of how to use it.

fail2ban-regex '<logline>' '<regex>'

And below is a real life example.

fail2ban-regex '{"log":"111.22.333.444 47.95.1.195 - - [05/Jul/2018:21:42:40 +0000] \"GET /phpMyadmin_bak/index.php HTTP/1.1\" 503 213 \"-\" \"Mozilla/5.0\"\n","stream":"stdout","time":"2018-07-05T21:42: 40.24318153Z"}' '{"log":"<HOST>.*GET.*.php.*'

The sample below specifies a single log file entry:

'{"log":"111.22.333.444 47.95.1.195 - - [05/Jul/2018:21:42:40 +0000] \"GET /phpMyadmin_bak/index.php HTTP/1.1\" 503 213 \"-\" \"Mozilla/5.0\"\n","stream":"stdout","time":"2018-07-05T21:42:40.24318153Z"}' 

and the regex {"log":"<HOST>.*GET.*.php.*' is used to verify if it works.

Below is the sample report:

Running tests
=============

Use   failregex line : {"log":"<HOST>.*GET.*.php.*
Use      single line : {"log":"111.22.333.444 47.95.1.195 - - [05/Jul/201...


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] {"log":"<HOST>.*GET.*.php.*
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.03 sec]

Verify its all working

From time to time you can run the status command to verify that fail2ban is indeed banning and working.

sudo fail2ban-client status

You will get the following output:

Status
|- Number of jail:      2
`- Jail list:   nginx-noscript, sshd

Notice the jail list has two, nginx-noscript and, sshd. You can now run an additional status command by specifying the jail such as:

sudo fail2ban-client status nginx-noscript

Below is the output.

Status for the jail: nginx-noscript                                                                                                                                                                                            
|- Filter                                                                                                                                                                                                                      
|  |- Currently failed: 0                                                                                                                                                                                                      
|  |- Total failed:     0                                                                                                                                                                                                      
|  `- File list:        /var/lib/docker/containers/**-json.log                                                                                                                                                                                                                
`- Actions                                                                                                                                                                                                                     
   |- Currently banned: 0                                                                                                                                                                                                      
   |- Total banned:     0                                                                                                                                                                                                      
   `- Banned IP list:                                                                                                                                                                                                          

From the above, you can see a summary of how many have been banned.

A list of useful jails

Below are a list of jails and their respective filters that I personally use. Feel free to use and modify.

Nginx x00 requests

Block potentially malicious x03\x00 requests.

Sample malicious request

77.72.83.87 - - _ [28/Dec/2018:14:13:14 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 150 "-" "-"

Jail

[nginx-x00] 
enabled   = true 
port      = http,https 
filter    = nginx-x00
logpath   = /var/lib/docker/containers/*/*-json.log

Filter
Filename: /etc/fail2ban/filter.d/filter.d/nginx-x00.conf

[Definition] 

failregex = ^{"log":"<HOST> .* .*\\x.*$

ignoreregex =

Php requests

Blocks php attacks

Sample malicious request

106.12.127.143 - - 209.97.142.137 [28/Dec/2018:14:27:46 +0000] "GET /cmd.php HTTP/1.1" 503 190 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"

Jail

[php-custom] 
enabled   = true
port      = http,https
filter    = php-custom 
logpath   = /var/lib/docker/containers/*/*-json.log

Filter
Filename:/etc/fail2ban/filter.d/filter.d/php-custom.conf

[Definition]

failregex = {"log":"<HOST>.*(GET|POST).*(.php).*$

ignoreregex = 

Git

Blocks requests at attempting to obtain git files. This is an interesting read:
https://slashcrypto.org/2018/11/28/eBay-source-code-leak/

Jail

[git-bad]
enabled   = true 
port      = http,https 
filter    = git-bad
logpath   = /var/lib/docker/containers/*/*-json.log 

Filter
Filename: /etc/fail2ban/filter.d/filter.d/git-bad.conf

[Definition]

failregex = {"log":"<HOST>. *(GET|POST).*(/.git).*$

ignoreregex =

Summary

In the post I demonstrated how to install Fail2ban on a Linux machine and configure it so that you can protect your server and Nginx from potentially malicious http requests.

References:

Newsletter

Receive new content like this and more. No spam. Ever. Unsubscribe whenever you want.

or subscribe via RSS with Feedly!