How to deal with TCP connections attack on Linux

Question

My server is being flooded until apache becomes non-responsive, and I need some help finding and blocking the responsible IP address(es).

Normally, I don't have more than 150 connections. Now I have thousands:

netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n
      1 established)
      1 Foreign
     13 LAST_ACK
     20 CLOSING
     30 SYN_RECV
     41 LISTEN
     44 FIN_WAIT1
     74 FIN_WAIT2
     77 CLOSE_WAIT
    273 ESTABLISHED
   1960 TIME_WAIT

MRTG graph clearly shows the normal connections until the attack begins:

This is the result of counting connections per IP (end of list only):

netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n
  5 4.59.90.216
  5 4.59.90.222
  5 4.59.90.237
  5 4.59.90.242
  5 74.125.26.95
  6 186.158.143.202
  6 216.58.219.162
  6 4.59.90.251
  7 104.24.5.60
  7 216.58.192.66
  7 4.59.90.212
  7 4.59.90.231
  7 4.59.90.241
  9 216.58.192.98
 10 189.177.214.89
 10 23.10.101.162
 11 4.59.90.226
 12 85.94.197.200
 25 216.58.219.66
 31 216.58.219.130
 40 0.0.0.0
 86 83.101.136.42
1026 10.0.0.2

The last one is the server's IP, I have not idea why it's shown. Thanks.

Answer 1

One thing you could try to limit the impact of the connections on apache is limiting them.

There are a few ways of doing this, but one way could be:

iptables -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 600 --hitcount 5 -j DROP

That would allow 5 new connections to port 80 per source IP within 10 minutes, which may help buy you some time to dig further. If the issue is hitting more than just apache, you might consider applying it to all TCP connections - but I would also suggest you add an exception for your SSH connection in that case.

Possibly worth looking into if you haven't is fail2ban, which you can configure to look at your apache logs, and block hosts based on certain criteria.

I am not sure I would recommend deploying that right now (the potential for something to go wrong seems a bit high), but it would be worth putting on the list of potential solutions in the longer term.

In passing: all of this is assuming the issue is Apache's responsiveness/load, not the load on your internet connection.

If your entire uplink is getting saturated, then there isn't much you can do on your host, since the issue is further up than that.

If you are cloud-based, maybe worth seeing what options exist. If you are not, maybe worth seeing if a CDN/DDoS protection company (cloudflare springs to mind, but I can't speak for their services) could help.

I would also tend to want to find out what your machine is doing with all those connections - netstat -ptn might be of some assistance

Beyond that, you could see if iftop or iptraf help you get a bit more insight into what is going on. See also https://stackoverflow.com/questions/368002/network-usage-top-htop-on-linux