Google introduced a brand new robots.txt report inside Google Search Console and on the similar time mentioned it would sundown the previous robots.txt tester software. The brand new software exhibits which robots.txt recordsdata Google discovered for the highest 20 hosts in your website, the final time they have been crawled, and any warnings or errors encountered.

Google additionally mentioned on X, “We’re additionally making related information obtainable from the Web page indexing report. As a part of this replace, we’re sunsetting the robots.txt tester.”

To entry this, go to the settings hyperlink to your Search Console profile or click on right here. That is what it appears to be like like:

Here’s what the report appears to be like like for this website, sure, I’ve a essential error. Listed here are these particulars and notice, you may request recrawl of your robots.txt file:

This assist doc goes via much more about this report.

Some usually are not joyful that the robots.txt tester software is gone however Bing nonetheless has one:

Good alternate options to Google’s depreciated software to validate your robots.txt:

• Merkle: https://t.co/33yMW9IlPB
• Ryte: https://t.co/jYAGuHN2q5
• Screaming Frog (in-crawl): https://t.co/zsr2KMiNmC

— Cyrus search engine marketing (@CyrusShepard) November 15, 2023

I really like the brand new robots.txt reporting (it is in settings btw), however eradicating the robots.txt Tester? I really like that software and it may be super-helpful. There are different instruments that do comparable issues, together with one from Bing, however I might like to see that keep in GSC. 🙂 https://t.co/hMIJQjfpQV

— Glenn Gabe (@glenngabe) November 15, 2023

Discussion board dialogue at X.