While we figure out a more comprehensive product we’ve decided to work on the the simplest thing that could possibly work.
A lot of you out there don’t block annoying/useless bots with identifiable agent-users. Those who do don’t keep it up to date with all the new ‘social media’ bots that are multiplying every day.
Our first product is going to dynamically check for these & block ‘em for you. This certainly isn’t going to block spammers/emailharvesters/bandwidth-eaters with truly evil intent, but it will save you a ton of bandwidth and user angst.
NO, it will not block good bots like google, adsense, yahoo, msn, baidu, etc. who generally behave themselves and are on a leash by masters who monitor their actions.
Anyhoo, it will be really simple. Stand by.
One Comment
Hi
I have a suggestion.
Draft clear, detailed guidelines on what distinguishes a “bad” bot from a “good” one.
Is a “good” bot simply one that not bad? Or does it go further?
*Specifically*, what is it that makes a bot “bad”?
What’s the differenc between an spammer and an emailharvester? Doesn’t the latter gather addresses for the former?
What is bandwidth eating? How many HTTP transactions per unit if time is “reasonable”? Does it depend on the server’s hardware?
You’ve taken on a very worthwhile task, but I think there is much more to it perhaps. Filtering by user-agent is a start. If you search for a guide to designing “good” bots, you find something written many years ago. Does it need to be revised?
This is an important issue since large players like MSN, Yahoo and Google rely on bots. Do they scrape to cache and to index content? Do they hold others’ content? And then charge others to advertise on their database?
Are search engines “middle men” that a user goes through to access content presented by someone else?
Bots are used perhaps because automation yields benefits such as speed and efficiency. Is automation inherently “bad”. Or does it depend on how it is used?