Menu

RTCXpression

Close

Developing a Strict Country and Web Robot Blocking Policy

robots policy I no longer get the web traffic I used to get. It’s my fault, of course, but I don’t really care anymore. What I care about is not being annoyed. That’s where my new blocking policy comes in.

My blocking policy is going to be strict, stricter than you can imagine. You won’t have to imagine it, though, because I’ll be describing it. I have all the TCP and UDP ports closed except for six. Those six are for the SSH, FTP and web servers along with the Webmin control panel.

Country Blocking Policy

My country blocking policy will start relaxed and get stricter as time goes by. There are far too many connections coming from countries where English isn’t even a second language. Even when it is, the percentage of English speakers doesn’t matter. Like the country of China.

Most of the country IP addresses can be blocked at the web server. Some of them have to be blocked at the firewall (iptables). There are some IP addresses port scanning the same ports over and over (which aren’t open) and doing nothing more than filling up log files.

Web Robot Blocking Policy

Many of the web robots can be identified by their user agent strings. Some of them fake their user agent strings to make the people viewing the logs think they’re people.

My web pages are static. I’ve ignored a lot of the web robots (bots) because they don’t have any real effect on resources. I won’t ignore them anymore.

I recently discovered a bot net attacking my website. It couldn’t do anything because, you know, they’re all static files. The person who initiated the attack probably thought I was still running WordPress. The hits weren’t coming quickly enough to slow things down. I suppose it would have been a lot worse if I had a dynamic content management system (CMS) online.

The Next Step

I plan to write some more articles, with the question “block or not?” prominent. I know more about some of the bots out there than I probably should know.

The master plan is to unload all that on you, the website owner.

By:
April 11, 2017

Categories:
Web Development

Previous and Next Articles:

« »

You May Also Like:

Comments:

Your comment will appear below the form when it's approved. When the page redisplays after hitting the send button (it can take a few seconds), your comment has been sent.

When replying to someone else's comment, please start the comment with "@" and the name so I can put it in the right place.

Books by William James Asberry
Comments Policy
Privacy Policy

RTCXpression established Feb 28, 2011
Copyright © 2013-2017 RT Cunningham