RT Cunningham


Developing a Strict Country and Web Robot Blocking Policy

robots policy I no longer get the web traffic I used to get. It’s my fault, of course, but I don’t really care anymore. What I care about is not being annoyed. That’s where my new blocking policy comes in.

My blocking policy is going to be strict, stricter than you can imagine. You won’t have to imagine it, though, because I’ll be describing it. I have all the TCP and UDP ports closed except for six. Those six are for the SSH, FTP and web servers along with the Webmin control panel.

Country Blocking Policy

My country blocking policy will start relaxed and get stricter as time goes by. There are far too many connections coming from countries where English isn’t even a second language. Even when it is, the percentage of English speakers doesn’t matter. Like the country of China.

Most of the country IP addresses can be blocked at the web server. Some of them have to be blocked at the firewall (iptables). There are some IP addresses port scanning the same ports over and over (which aren’t open) and doing nothing more than filling up log files.

Web Robot Blocking Policy

Many of the web robots can be identified by their user agent strings. Some of them fake their user agent strings to make the people viewing the logs think they’re people.

My web pages are static. I’ve ignored a lot of the web robots (bots) because they don’t have any real effect on resources. I won’t ignore them anymore.

I recently discovered a bot net attacking my website. It couldn’t do anything because, you know, they’re all static files. The person who initiated the attack probably thought I was still running WordPress. The hits weren’t coming quickly enough to slow things down. I suppose it would have been a lot worse if I had a dynamic content management system (CMS) online.

The Next Step

I plan to write some more articles, with the question “block or not?” prominent. I know more about some of the bots out there than I probably should know.

The master plan is to unload all that on you, the website owner.

Share: Facebook | Twitter

By RT Cunningham
April 11, 2017
Web Development

My Other Sites:

Coffee Corner

Linux Lines