null 💻 notes

Three Ways to Increase Security against Bad Bots and Spam

DashingWP's servers were under attack today from what appears to be a pretty nasty DDoS attack that originated in China, bounced off of Germany, and then hit really hard from the UK, Portugal, Minnesota (USA), New Jersey (USA), and parts of Utah (USA). In total, there were 26,491 IP addresses that were discovered to be part of the attack, and we're not entirely sure whether or not it was a roving attack or if something targeted us specifically.

After defeating this problem, I thought about some of the things people could be doing better in order to help prevent their susceptibility to DDoS attacks and spam/bad bots in general. While we'll never be able to fully prevent a DDoS attack from happening, we can go out of our way to strengthen our infrastructures defenses against spammy and malicious bots that lurk out there, and in doing so, lessen the likelihood of irreparable damage if and when we fall victim to a DDoS attack. In this article, I'll share with you three preventive controls that you should be implementing in order to lessen your vulnerability and harden both your server and your sites.

I first realized something was wrong when my servers were spiking. Normally this is just a site that is getting a lot of traffic (maybe they're on the front page of Reddit or something), but I always start checking logs with an automated program that checks for intrusive behavior. What is intrusive behavior during a traffic spike? Well, first I asked myself, "Is someone trying to access my server?"

cat /your/servers/auth.log

As it turned out, there were a lot of of IPs trying to brute force one of our web servers on port 22 (SSH). ** **

**Preventive Control #1: **Use SSH key authentication if you can, but if you're like us and require password-based authentication, then either 1) restrict SSH traffic to known IP addresses (not likely feasible if you're running a web hosting company), or 2) set `max retries` to `0` in your fail2ban configuration file.

We have users logging into jailed SFTP accounts all the time, and trying to get them to forward us all the IP addresses they'll be working on is just not feasible. Instead, we have a multi-level port sniffer (including fail2ban, which should be more than enough in most scenarios) that catches all failed SSH logins and bans the originating IP indefinitely. This is done by setting `bantime` in your default fail2ban settings to `-1`.

With all of the traffic trying to brute force its way into our servers via ssh, I then took a look at my email (yes, my email). As expected, my Spam folder was blowing up. I took this as meaning the attacking entities found the WordPress contact forms on one or all of my sites and started blowing up the contact forms. I don't really care that much because the fix (below) is extremely easy, but what you should know about all this spam traffic is that it's probably costing you money (especially if you're on a managed WordPress host that charges you for additional traffic above and beyond your already limited account threshold).

If the contact forms are being spammed, then you bet your WordPress is trying to be brute-forced, too.

**Preventive Control #2: **Lock down your contact forms with CAPTCHA or cool-down periods. If you're running WordPress, run plugins like Limit Login Attempts and for the love of managed hosting, please make your passwords random and your username's something other than ‘admin'.

These preventive controls were in place for me, so the last thing for me to do was to take up arms and begin my counter-attack.

Preventive Control #3: Have an ipset list ready to load up with bad IPs that you find in your logs, actively scrub your logs for bad traffic, and stay up-to-date on how people are trying access your system.

If you don't know what ipset is or are not using it, please take a look at this very simple tutorial on how to get it up and running. It's used in cohesion with iptables and when it comes to blocking thousands of IPs at a time, it's a much faster solution than iptables alone.

The way I combated the traffic was to first determine whether or not the traffic coming in was good or bad traffic. This was easy enough by designing a bash script to look through my log files and report suspicious activity on recent log entries. What exactly is suspicious activity? My script asked the following as it went through each entry:

Writing a bash script to do this for you helps to prevent the arduous task of shutting down your network interfaces to prevent all traffic from hitting your server while the attack subsides and you parse log files by hand in order to manually block the bad visits. Of course, using automation comes at a risk: twice now I've locked out one of our client managers because her activity was flagged by my script, which automatically added her to an ipset list. *So remember that automating your anti-spam/anti-attack workflow comes at the cost of possibly locking out your users and/or their visitors. *

I ended up with a massive list of over twenty thousand IPs that all flagged positive on several online blacklists. They're all residing happily in an ipset list that I've used and I don't worry about them anymore. That being said, there's nothing that says this wont happen again, and there's nothing that says I will be able to just block all the IPs that are ever bad because I don't want to waste server resources (albeit negligible) by giving them the satisfaction. They want to deny service by destroying my ability to serve my customers and their vistors — and that's just not going to happen.

So as a final note, remember that these three preventive controls are not meant to be the only thing you do to prevent bad traffic, but rather, a starting point for you to begin researching techniques and best-practices in protecting your servers, your clients, and their visitors from the bad traffic and spam that we all hate and can't seem to get rid of. It's an arduous and time consuming job, but in the end, it's worth it.