When I managed to deploy an ASP.NET core app on Linux with Nginx, I noticed a lot of rogue and spam internet traffic in the Nginx logs. Mostly mild hacking attempts and rather a lot of php requests. Its mostly noise, but I wanted a way of banning these ip addresses as they are no good and this is how I came across using Fail2ban.
What is Fail2ban?
In short, its a tool that scans log files based on a defined regular expression. If it matches, it will perform an action on it. Fail2ban can scan many different types of logs such as Nginx, Apache and ssh logs. Based on matches, it is able to ban ip addresses for a configured time period.
Installing Fail2ban doesn’t take long especially if you’re familiar with the Linux eco-system. Its setting up and making adjustments to your specifications that may take some time. It all depends on your needs.
Summary
- Install Fail2ban.
- Configure Fail2ban defaults.
- Add our first jail rule so it can monitor Nginx logs.
- Testing it all works.
Install Fail2ban
- Update the list of packages on Ubuntu by running the following command.
sudo apt-get update
- Install Fail2ban
sudo apt-get install fail2ban
Done. That was the easy part. Now its time to configure it.
Configure Fail2ban defaults
- Make a copy of the original jail.conf so that it can be modified.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
- Now, lets modify the file:
nano /etc/fail2ban/jail.local
- Modify the
ignoreipso that it ignores your host ip address. This should be your public ip address.
[DEFAULT]
#
# MISCELLANEOUS OPTIONS
#
# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8 your-ip-address-here
If your public ip address was 54.15.55.01 then it should look something like this:
ignoreip = 127.0.0.1/8 54.15.55.01
- Modify default
bantimeandfindtimeandmaxretry. I’ve set it to permanent with a minimum retry time as shown below:
# "bantime" is the number of seconds that a host is banned.
bantime = -1 # this is permanant
# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime = 1
# "maxretry" is the number of failures before a host get banned.
maxretry = 1
Configuring Jails in Fail2ban so it can monitor Docker Nginx Logs
Within the file /etc/fail2ban/jail.local there is a section called Jails. This is where pre-defined rules are set for Fail2ban so that it knows what kind of filter to apply and where to monitor log files. First we need to understand how docker files are logged and where.
How to find out where docker logs are located
Docker log files are generally logged as json and in order to find out where docker log files are located, simply run the docker inspect [containername] command like below
docker inspect nginx-web
And the result will be something like this:
...
"HostsPath": "/var/lib/docker/containers/94dc5a3**/hosts",
"LogPath": "/var/lib/docker/containers/94dc5a3**/94dc5a3**-json.log",
"Name": "/nginx-web"
...
Notice that the LogPath is where docker logs file for that particular container.
Creating our first Jail
Now we have the logpath, we can create our first jail. The jail below is for Nginx to deny any requests that attempt to request for types such as .exe .php etc.
[nginx-noscript]
enabled = true
port = http,https
filter = nginx-noscript
logpath = /var/lib/docker/containers/*/*-json.log
Notice that the enabled flag is set to true and the logpath has been set but with a wildcard path for the docker logs. Now you can be specific, but be warned, each time a docker container restarts, it will be located in a different folder within containers.
Create the corresponding filter
Now, create a file if one does not exist already called /etc/fail2ban/filter.d/nginx-noscript.conf. The filename must match the filter name specified in the jail.
nano /etc/fail2ban/filter.d/nginx-noscript.conf
Below is the regular expression. Notice that it has a prefix of ^{"log". The reason for this is that docker logs in json format.
[Definition]
failregex = ^{"log":"<HOST> -.*GET.*(\.php|\.asp|\.exe|\.pl|\.cgi|\.scgi)
ignoreregex =
Restart Fail2ban
Any changes made in jail.local a restart of Fail2ban is required. Below are commands which will restart the service.
service fail2ban restart
fail2ban-client reload
If there is anything wrong with the fail or filter, an error would be reported.
Testing filters
To ensure that filters are working as expected with the defined regular expression, Fail2ban offers two ways to test that they are working.
Option 1 - Specify the log file and the filter file
The fail2ban-regex command line tool offers a way of testing if a regex is working as expected.
You can use it as follows:
fail2ban-regex <logpath> <filterpath>
Here is a sample of it in use.
fail2ban-regex /var/lib/docker/containers/7c2442*/*-json.log /etc/fail2ban/filter.d/nginx-noscript.conf
And a sample output report is shown below.
Running tests
=============
Use failregex filter file : nginx-noscript, basedir: /etc/fail2ban
Use log file : /var/lib/docker/containers/94dc5*/94dc5*-json.log
Use encoding : UTF-8
Results
=======
Failregex: 233 total
|- #) [# of hits] regular expression
| 1) [233] ^{"log":"<HOST> -.*GET.*(\.php|\.asp|\.exe|\.pl|\.cgi|\.scgi)
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [724] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-
Lines: 724 lines, 0 ignored, 233 matched, 491 missed
[processed in 0.10 sec]
Notice towards the end of the report, it states 233 matched, 491 missed. This means that the filter was able to match 233 lines and 491 were not applicable. This is a good indication that the regex is matching and working.
Option 2 - Specify inline
The fail2ban-regex also offers a way to test inline rather then specifying a log file.
Below is the syntax of how to use it.
fail2ban-regex '<logline>' '<regex>'
And below is a real life example.
fail2ban-regex '{"log":"111.22.333.444 47.95.1.195 - - [05/Jul/2018:21:42:40 +0000] \"GET /phpMyadmin_bak/index.php HTTP/1.1\" 503 213 \"-\" \"Mozilla/5.0\"\n","stream":"stdout","time":"2018-07-05T21:42: 40.24318153Z"}' '{"log":"<HOST>.*GET.*.php.*'
The sample below specifies a single log file entry:
'{"log":"111.22.333.444 47.95.1.195 - - [05/Jul/2018:21:42:40 +0000] \"GET /phpMyadmin_bak/index.php HTTP/1.1\" 503 213 \"-\" \"Mozilla/5.0\"\n","stream":"stdout","time":"2018-07-05T21:42:40.24318153Z"}'
and the regex {"log":"<HOST>.*GET.*.php.*' is used to verify if it works.
Below is the sample report:
Running tests
=============
Use failregex line : {"log":"<HOST>.*GET.*.php.*
Use single line : {"log":"111.22.333.444 47.95.1.195 - - [05/Jul/201...
Results
=======
Failregex: 1 total
|- #) [# of hits] regular expression
| 1) [1] {"log":"<HOST>.*GET.*.php.*
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [1] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-
Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.03 sec]
Verify its all working
From time to time you can run the status command to verify that fail2ban is indeed banning and working.
sudo fail2ban-client status
You will get the following output:
Status
|- Number of jail: 2
`- Jail list: nginx-noscript, sshd
Notice the jail list has two, nginx-noscript and, sshd. You can now run an additional status command by specifying the jail such as:
sudo fail2ban-client status nginx-noscript
Below is the output.
Status for the jail: nginx-noscript
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/lib/docker/containers/**-json.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
From the above, you can see a summary of how many have been banned.
A list of useful jails
Below are a list of jails and their respective filters that I personally use. Feel free to use and modify.
Nginx x00 requests
Block potentially malicious x03\x00 requests.
Sample malicious request
77.72.83.87 - - _ [28/Dec/2018:14:13:14 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 150 "-" "-"
Jail
[nginx-x00]
enabled = true
port = http,https
filter = nginx-x00
logpath = /var/lib/docker/containers/*/*-json.log
Filter
Filename: /etc/fail2ban/filter.d/filter.d/nginx-x00.conf
[Definition]
failregex = ^{"log":"<HOST> .* .*\\x.*$
ignoreregex =
Php requests
Blocks php attacks
Sample malicious request
106.12.127.143 - - 209.97.142.137 [28/Dec/2018:14:27:46 +0000] "GET /cmd.php HTTP/1.1" 503 190 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"
Jail
[php-custom]
enabled = true
port = http,https
filter = php-custom
logpath = /var/lib/docker/containers/*/*-json.log
Filter
Filename:/etc/fail2ban/filter.d/filter.d/php-custom.conf
[Definition]
failregex = {"log":"<HOST>.*(GET|POST).*(.php).*$
ignoreregex =
Git
Blocks requests at attempting to obtain git files. This is an interesting read: https://slashcrypto.org/2018/11/28/eBay-source-code-leak/
Jail
[git-bad]
enabled = true
port = http,https
filter = git-bad
logpath = /var/lib/docker/containers/*/*-json.log
Filter
Filename: /etc/fail2ban/filter.d/filter.d/git-bad.conf
[Definition]
failregex = {"log":"<HOST>. *(GET|POST).*(/.git).*$
ignoreregex =
Summary
In the post I demonstrated how to install Fail2ban on a Linux machine and configure it so that you can protect your server and Nginx from potentially malicious http requests.
References: