Corralling Robots and Spiders
Basically:
In a file named robots.txt within a particular directory. In the root directory, that keeps all robots out. Or, to specify particular directories:
In the root directory, that would keep robots out of the "images" and "scripts" directories.
User-agent: *
Disallow: /
Disallow: /
In a file named robots.txt within a particular directory. In the root directory, that keeps all robots out. Or, to specify particular directories:
User-agent: *
Disallow: /scripts/
Disallow: /images/
Disallow: /scripts/
Disallow: /images/
In the root directory, that would keep robots out of the "images" and "scripts" directories.
posted by Tom at
6:35 PM
0 Comments:
Post a Comment
Subscribe to Post Comments [Atom]
<< Home