Categories
Categories
robots.txt text files Print
- 98
# Below is the text we place in the robots text file. This allows search engines like google to crawl the website. Notice how the disallow has been commented out :)
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml
User-agent: *
#Disallow: /administrator/
#Disallow: /cache/
#Disallow: /cli/
#Disallow: /components/
#Disallow: /images/sampledata
#Disallow: /includes/
#Disallow: /installation/
#Disallow: /language/
#Disallow: /libraries/
#Disallow: /logs/
#Disallow: /media/
#Disallow: /modules/
#Disallow: /plugins/
#Disallow: /templates/
#Disallow: /tmp/
Was this answer helpful?
Related Articles
Remove www from any URLs
# Remove www from any URLs that have them:
(paste in .htaccess file and swap ‘example’)...
emulating systems
To test websites on multiple devices we use a tool at https://www.browserling.comThis website...
Website Speed
To speed up your website use your cache. We use a website speed test by Google you can find...
