Des (deskitty) wrote,

  • Mood:
  • Music:
Hah, so people actually DO read nasty emails. Cool! Maybe I'll send out more of them... :-P

Just kidding...

Re: Inconsiderate Robot
From: "?????.com Support" <support@??????.com>
To: ????
Date: Today 06:45:25

Dear Josh,

Firstly, thank you for your thoughtful suggestions.

After careful reviews, we have adopted your suggestions and our robots
now operate in strict adherence to the Web Robots Protocol ( ).

The two changes made are:
1. Our robots now read the /robots.txt file (if exists) before spidering
a website.
2. Out robots now identify themselves as "www.?????.com"

Thank you once again.

Best regards,
?????.com Support

On Wed, 2003-04-02 at 12:56, ???? wrote:
> Name : Josh
> Email : ????
> Username :
> Message:
> About 2 minutes ago, your robot hit my site, The
> relevant log entry is as follows:
> - - [01/Apr/2003:20:32:31 -0800] "GET / HTTP/1.0" 200 1869
> "http://www.?????.com/FREE_SERVER_MONITORING_SERVICE/" "Mozilla/4.0
> (compatible; MSIE 6.0; Windows NT 5.1)"
> There are a couple issues with this you should be aware of:
> - Your robot did NOT first attempt to retrieve the robots.txt file to
> determine whether or not it should access the site. This behavior is
> dangerous, as it could cause unwanted side-effects on the web server. (In
> my case, none, but accessing sites in this way is generally frowned-upon
> by system administrators.)
> - Your robot accessed my site with a user agent of "Mozilla/4.0
> (compatible; MSIE 6.0; Windows NT 5.1)". As your programmers should know,
> this is NOT a valid user agent for a robot.
> - Accessing sites in this fashion demonstrates that is
> trying to use other people's resources to advertise their services. Not
> only is this inconsiderate and just plain rude, it steals the resources of
> the servers you hit, which in some cases may be considered outright
> illegal. ?????.com was NOT given permission to access my webserver
> for this purpose, and I find it utterly detestable that you believe it is
> acceptable for you to misuse others' resources in this way.
> I hope you will rethink your marketing strategy, as it is only serving to
> piss off those who might be your customers.


-- Des

  • (no subject)

    Well, I'm off to Dreamwidth. I hope to see you all there! Nice knowing you, LJ. It's been grand. — Des

  • A fresh start?

    So I'm thinking of moving away from LJ. Every time I glance at my ad blocker, there are an uncomfortably-large number of advertising and tracking…

  • 2012: Ramp It Up

    It’s that time of the year again -- another year has passed, and as usual, I don’t finish reflecting on it until the first 3 months of the following…

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded