I just put the finishing touches on My Bot Project. I didn’t have much to touch up because I used the same theme for that website as this one. It’s a fast theme.
According to the Alexa.com extension, pages on this website load as fast as 1.102 seconds. Keeping bad bots away may have some bearing on it.
Since I block a lot of user agents and their associated bots at this website, I needed another website to identify and classify them. I don’t even let the blocked user agents show up in the daily access log for this website. I’m almost convinced that too much logging can have negative effects on server responsiveness.
To be clear, my bot project isn’t something I need. It’s something others may need, a resource to look to when trying to determine whether to block or not. I’ve found resources like it, some designed way better than mine. The problem with most of them is you don’t know who’s updating the data and how often, or if they have some other motive.
Some resources have certain user agents marked as “good” while other resources mark the same user agents as “bad”. Why the conflict? Some user agents are completely missing. Why? Here are a few resources I check from time to time:
If you know of better resources than these, feel free to chime in.
I’m adding both the common and uncommon bots to my bot project website. What I consider common may not be common to everyone, especially outside of the United States. I’ve already added the “Googlebot” and the “bingbot” user agents. I’ll be copying the information I have for Baidu and Yandex from this website to that one next.
Some bots have multiple user agents. Googlebot and bingbot, for example, have two each that I know about. One for desktop computers and one for mobile devices. All I can do is double check user agent strings to make sure they’re not duplicates.
I make mistakes just like other people and I can’t guarantee the information I collect will be better than the information from other websites. I can only guarantee to update it from time to time, probably more frequently than others.
My bot project is only one of my projects. I’m planning on a link directory sometime in the future, for more than one reason. Although I bought specialized software in the past (and still have it available), I’ll probably just copy the same theme and similar formats. I like simplicity.
To be honest, this website doesn’t keep me busy enough. I’m not as prolific a writer as I used to be and it’s probably due to my age and my general lack of interest. The other projects should help fill in the gaps a bit.
I used to write something somewhere every day of the week. In may cases, multiple times a day. Those days are long gone. Now I’m lucky if I can write anything anywhere once a week.