To update the rules in your existing robots.txt file, download a copy of your robots.txt file from your site and make the necessary edits.
You can download your robots.txt file various ways, for example:
https://example.com/robots.txt and copy its contents into a new text file on computer. Make sure you follow the guidelines related to the file format when creating the new local file.curl https://example.com/robots.txt -o robots.txt
Open the robots.txt file you downloaded from your site in a text editor and make the necessary edits to the rules. Make sure you use the correct syntax and that you save the file with UTF-8 encoding.
Upload your new robots.txt file to the root directory of your site as a text file named robots.txt. The way you upload a file to your site is highly platform and server dependent. Check out our tips for finding help with uploading a robots.txt file to your site.
During the automatic crawling process, Google’s crawlers notice changes you made to your robots.txt file and update the cached version every 24 hours. If you need to update the cache faster, use the Request a recrawl function of the robots.txt report.
Apply for your exclusive plan for free