Posted by Chris on May 16, 2001 at 14:05:13:
In Reply to: Re: Why did this happen? posted by floorcookie on May 16, 2001 at 13:03:14:
I know about robots.txt and have used it for years on others sites I manage.
Using that would only prevent legitimate search engines from indexing the site.
I have very few robots.txt hits that are not from legitimate search engines.
Most of the SPAM bots ignore it.
All that would do is keep the site from being indexed by search engines, the spam bots would still grab email addresses.
Unfortunately that is not a solution.
Many SPAM bots now disguise themselves as web browses, and ignore robots.txt.
If there was a solution, I and many others would be using it!