User-agent: the robot you want the following rule to apply to.
Disallow: the URL you want to block from the web spiders and web robots.Examples: To disallow all robots from a folder content:
Disallow: /myprivatefolder/To allow a single robot and disallow others:
Disallow: /To disallow a single robot:
Disallow: /Note: Bad robots can ignore your robots.txt instructions. Especially malware robots such as email address harvesters, spammers, etc. will pay no attention. Read Caspio Best Practices Guide to SEO Deployment. For more information, look through online resources.