| Home | Free Articles for Your Site | Submit an Article | Advertise | Link to Us | Search | Contact Us | |
Custom Search
|
Article Surfing ArchiveWhat is the Robot Text File? - Articles SurfingThe robot text file is used to disallow specific or all search engine spider's access to folders or pages that you don't want indexed. Why would you want to do this? You may have created a personnel page for company employees that you don't want listed. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file. How do I use it? You need to upload it to the root of your web site or it will not work - if you don't have access to the root then you will need to use a Meta tag to disallow access. You need to include both the user agent and a file or folder to disallow. What does it look like? It's really nothing more than a "Notepad" type .txt file named "robots.txt" The basic syntax is User-agent: spiders name here If you use User-agent: * The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages. To disallow an entire directory use Disallow:/mydirectory/ To disallow an individual file use Disallow:/file.htm You have to use a separate line for each disallow. You cannot you for example use Disallow:/file1.htm,file2.html You should use Use-agent/* For a list of spider names visit http://www.robotstxt.org/wc/active/html/ Make sure you use the right syntax if you don't it will not work. You can check you syntax here http://www.searchengineworld.com/cgi-bin/robotcheck.cgi For help on creating robot text files there is a program call robogen. There is a free version and an advanced version, which costs $12.99 http://www.rietta.com/robogen/
RELATED SITES
Copyright © 1995 - Photius Coutsoukis (All Rights Reserved). |
ARTICLE CATEGORIES
Aging Arts and Crafts Auto and Trucks Automotive Business Business and Finance Cancer Survival Career Classifieds Computers and Internet Computers and Technology Cooking Culture Education Education #2 Entertainment Etiquette Family Finances Food and Drink Food and Drink B Gadgets and Gizmos Gardening Health Hobbies Home Improvement Home Management Humor Internet Jobs Kids and Teens Learning Languages Leadership Legal Legal B Marketing Marketing B Medical Business Medicines and Remedies Music and Movies Online Business Opinions Parenting Parenting B Pets Pets and Animals Poetry Politics Politics and Government Real Estate Recreation Recreation and Sports Science Self Help Self Improvement Short Stories Site Promotion Society Sports Travel and Leisure Travel Part B Web Development Wellness, Fitness and Diet World Affairs Writing Writing B |