This one has had me baffled for a while. From time to time in my logs, I'd see repeated attempts by an unknown IP to connect to one of the services here, usually trying to log into the mail server via a POP account. Normally I'd simply update the /etc/firewall.user file to drop connections from that IP, and not worry about it, but I knew there had to be a way to stop such attacks in their tracks...
Finally after getting fed up and doing some reading on connection throttling and other topics, I found a number of articles that all focused on blocking brute force dictionary attacks to the SSH service, using the ipt_recent module for iptables. Trying to adapt these by simply changing the port number didn't work, changing the rule they were applied to didn't work, many variations either caused errors in iptables, or simply failed to stop any attacks.
After more Googling, and closing in on exactly what I was searching for, I finally found an article that had much the same information as the others, but some comments there led me to the answer I needed.
The steps documented in many places for SSH will work great, but they are intended to block traffic destined for a service on the router itself. My situation was to block traffic destined for a server behind the firewall, but not on the router, a service that traffic was being forwarded to.
This is a key difference, and iptables handles these chains in different ways, and the rules I was putting in were only acting on the INPUT chain, not the forwarding_rule.
My firewall.user file was cobbled together from various places, including a number of threads on the OpenWrt forums, and so is likely a bit different from what others may have, so below is the relevant code from my file showing how my rules are defined, and more importantly, the working code that will block repeated attempts at connecting to the POP service here (port 110).
In my example, 10 connections within 60 seconds from the same IP will cause further attempts to simply drop, until the oldest connections age past 60 seconds and drop off. The second rule will drop connections with 20 attempts in 10 minutes (600 seconds), which should be enough to stifle most attacks. I plan to adapt this code to FTP and other services also.
It is important to understand how users connect to these services, and when multiple connections within a short timespan may be required, so that you can then identify what may actually constitute an attack to effectively block it.
WAN=$(nvram get wan_ifname)
LAN=$(nvram get lan_ifname)
iptables -F input_rule
iptables -F output_rule
iptables -F forwarding_rule
iptables -t nat -F prerouting_rule
iptables -t nat -F postrouting_rule
# Note: Firewall must be re-run if WAN IP address changes
WANIP=`ifconfig $WAN | grep inet\ addr | sed -r 's/.*inet addr:([0-9.]+) .*/\1/'`
iptables -t nat -A postrouting_rule -o $WAN -j SNAT --to-source $WANIP
iptables -A FORWARD -i br0 -o br0 -j ACCEPT
iptables -A FORWARD -i $LAN -o $WAN -j ACCEPT
# Lines below let WAN address work from LAN
iptables -N POP_CHECK
iptables -A forwarding_rule -p tcp --dport 110 -m state --state NEW -j POP_CHECK
iptables -A POP_CHECK -m recent --set --name POP
iptables -A POP_CHECK -m recent --update --seconds 60 --hitcount 10 --name POP -j DROP
iptables -A POP_CHECK -m recent --update --seconds 600 --hitcount 15 --name POP -j DROP
iptables -t nat -A prerouting_rule -d $WANIP -p tcp --dport 110 -j DNAT --to $server:110
iptables -A forwarding_rule -d $server -p tcp --dport 110 -j ACCEPT
iptables -t nat -A postrouting_rule -o $LAN -p tcp -s $network -d $server -m multiport --dports 110 -j SNAT --to-source $WANIP