Become a Member and receive automatic updates and support for our 47+ plugins!
We are removing our free plugins from Please join our email newsletter to receive updates.
250,000+ websites use our FREE WordPress plugins!  Download Now250,000+ sites use our WP plugins!  Click Here

How can I generate a Robots.txt file?

If you are not aware, maintaining a robots.txt file for your WordPress site (or any website) is very important for proper SEO, including Google indexing, accessibility, and more. This file should always reside on your site “root” meaning or likewise. You can either login to SFTP and manually create the file, or use a free WordPress plugin like Yoast SEO to automatically generate the file via the WordPress backend instead. If you are not sure which rules to include in your robots.txt file there are tons of free resources and generator tools online for that. In general you should make sure to enable Google Ad (or any ad network bot) that you are part of, and then block access to all other bots on “archive” type pages that can introduce duplicate content on your domain to Google.

Last modified:  29 Aug, 2015