Stay organized with collections
Save and categorize content based on your preferences.
Wednesday, July 16, 2014
To crawl, or not to crawl, that is the robots.txt question.
Making and maintaining correct robots.txt files can sometimes be difficult. While most sites have
it easy (tip: they often don't even need a robots.txt file!), finding the rules within a
large robots.txt file that are or were blocking individual URLs can be quite tricky. To make that
easier, we're now announcing an updated
robots.txt testing tool
in Webmaster Tools.
You can find the updated testing tool in
Webmaster Tools
within the Crawl section:
Here you'll see the current robots.txt file, and can test new URLs to see whether they're
disallowed for crawling. To guide your way through complicated rules, it will highlight the
specific one that led to the final decision. You can make changes in the file and test those too,
you'll just need to upload the new version of the file to your server afterwards to make the
changes take effect. Our developers site has
more about robots.txt rules and how the files are processed.
Additionally, you'll be able to review older versions of your robots.txt file, and see when access
issues block us from crawling. For example, if Googlebot sees a 500 server error for
the robots.txt file, we'll generally pause further crawling of the website.
Since there may be some errors or warnings shown for your existing sites, we recommend
double-checking their robots.txt files. You can also combine it with other parts of Webmaster
Tools: for example, you might use the updated
Fetch as Google tool
to render important pages on your website. If any blocked URLs are reported, you can use this
robots.txt tester to find the rule that's blocking them, and, of course, then improve that.
A common problem we've seen comes from old robots.txt files that block CSS, JavaScript, or mobile
content—fixing that is often trivial once you've seen it.
We hope this updated tool makes it easier for you to test and maintain the robots.txt file. Should
you have any questions, or need help with crafting a good set of rules, you can drop by our
webmaster's help forum!
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eGoogle has updated the robots.txt testing tool in Webmaster Tools to make it easier to identify and fix crawl issues.\u003c/p\u003e\n"],["\u003cp\u003eThe tool allows users to test new and existing URLs, review robots.txt file history, and pinpoint problematic rules blocking Googlebot access.\u003c/p\u003e\n"],["\u003cp\u003eGoogle recommends reviewing robots.txt files for potential errors, especially for older files that might unintentionally block essential resources like CSS or JavaScript.\u003c/p\u003e\n"],["\u003cp\u003eUsing the robots.txt testing tool alongside other Webmaster Tools features, like "Fetch as Google," can provide a comprehensive approach to website crawl optimization.\u003c/p\u003e\n"],["\u003cp\u003eGoogle provides resources like their developer site and webmaster help forum for further assistance and guidance with robots.txt files.\u003c/p\u003e\n"]]],["An updated robots.txt testing tool is now available in Webmaster Tools, under the Crawl section. This tool allows users to test URLs against their robots.txt file, highlighting the specific rule affecting crawlability. Users can also modify and test changes before uploading the updated file. The tool also reviews older file versions and identifies access issues, such as `500` server errors. It's advised to check for errors and combine with other Webmaster Tools to resolve any blocking rules.\n"],null,["Wednesday, July 16, 2014\n| It's been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore). Check out the newer [robots.txt report](https://support.google.com/webmasters/answer/6062598).\n\nTo crawl, or not to crawl, that is the robots.txt question.\n\n\nMaking and maintaining correct robots.txt files can sometimes be difficult. While most sites have\nit easy (tip: they often don't even need a robots.txt file!), finding the rules within a\nlarge robots.txt file that are or were blocking individual URLs can be quite tricky. To make that\neasier, we're now announcing an updated\n[robots.txt testing tool](https://www.google.com/webmasters/tools/robots-testing-tool)\nin Webmaster Tools.\n\n\nYou can find the updated testing tool in\n[Webmaster Tools](https://search.google.com/search-console)\nwithin the Crawl section:\n\n\nHere you'll see the current robots.txt file, and can test new URLs to see whether they're\ndisallowed for crawling. To guide your way through complicated rules, it will highlight the\nspecific one that led to the final decision. You can make changes in the file and test those too,\nyou'll just need to upload the new version of the file to your server afterwards to make the\nchanges take effect. Our developers site has\n[more about robots.txt rules and how the files are processed](/search/docs/crawling-indexing/robots/intro).\n\n\nAdditionally, you'll be able to review older versions of your robots.txt file, and see when access\nissues block us from crawling. For example, if Googlebot sees a `500` server error for\nthe robots.txt file, we'll generally pause further crawling of the website.\n\n\nSince there may be some errors or warnings shown for your existing sites, we recommend\ndouble-checking their robots.txt files. You can also combine it with other parts of Webmaster\nTools: for example, you might use the updated\n[Fetch as Google tool](https://support.google.com/webmasters/answer/158587)\nto render important pages on your website. If any blocked URLs are reported, you can use this\nrobots.txt tester to find the rule that's blocking them, and, of course, then improve that.\nA common problem we've seen comes from old robots.txt files that block CSS, JavaScript, or mobile\ncontent---fixing that is often trivial once you've seen it.\n\n\nWe hope this updated tool makes it easier for you to test and maintain the robots.txt file. Should\nyou have any questions, or need help with crafting a good set of rules, you can drop by our\n[webmaster's help forum](https://support.google.com/webmasters/go/community)!\n\nPosted by [Asaph Arnon](https://www.linkedin.com/in/asaph-arnon-4627987/), Webmaster Tools team"]]
