Re: Inconsiderate Robot
From: "?????.com Support" <support@??????.com>
Date: Today 06:45:25
Firstly, thank you for your thoughtful suggestions.
After careful reviews, we have adopted your suggestions and our robots
now operate in strict adherence to the Web Robots Protocol (
The two changes made are:
1. Our robots now read the /robots.txt file (if exists) before spidering
2. Out robots now identify themselves as "www.?????.com"
Thank you once again.
On Wed, 2003-04-02 at 12:56, ????@condordes.net wrote:
> Name : Josh
> Email : ????@condordes.net
> Username :
> About 2 minutes ago, your robot hit my site, http://www.condordes.net. The
> relevant log entry is as follows:
> 18.104.22.168 - - [01/Apr/2003:20:32:31 -0800] "GET / HTTP/1.0" 200 1869
> "http://www.?????.com/FREE_SERVER_MONITORING_SERVICE/" "Mozilla/4.0
> (compatible; MSIE 6.0; Windows NT 5.1)"
> There are a couple issues with this you should be aware of:
> - Your robot did NOT first attempt to retrieve the robots.txt file to
> determine whether or not it should access the site. This behavior is
> dangerous, as it could cause unwanted side-effects on the web server. (In
> my case, none, but accessing sites in this way is generally frowned-upon
> by system administrators.)
> - Your robot accessed my site with a user agent of "Mozilla/4.0
> (compatible; MSIE 6.0; Windows NT 5.1)". As your programmers should know,
> this is NOT a valid user agent for a robot.
> - Accessing sites in this fashion demonstrates that hyperspin.com is
> trying to use other people's resources to advertise their services. Not
> only is this inconsiderate and just plain rude, it steals the resources of
> the servers you hit, which in some cases may be considered outright
> illegal. ?????.com was NOT given permission to access my webserver
> for this purpose, and I find it utterly detestable that you believe it is
> acceptable for you to misuse others' resources in this way.
> I hope you will rethink your marketing strategy, as it is only serving to
> piss off those who might be your customers.