Magic robots.txt

Description

In a nutshell: You don’t want robots wasting your resources and slowing down your site if they don’t give you some kind of direct or indirect benefit. An optimal configuration will maximize your SEO results and reduce your operating costs.

You’ll need less than a minute to set your site characteristics and get a customized robots.txt configuration installed, matching your needs and easily updated without headaches. Let us to apply our experience in maintaining and optimizing sites. A clear knowledge of what we want to do and why, moved us to change the decission method from human created robots.txt to an automated system, things are simpler as they seems. Here comes to help this plugin, doing all the magic for you.

A few simple questions about your site will be enought to vast majority of sites.

Screenshots

  • Settings view.
  • Sample of resulting robots.txt.
  • Sample of resulting robots.txt in debug mode.
  • Additional information in settings view while in debug mode.

Installation

  1. Upload files to the /wp-content/plugins/magic-robots-txt directory or install plugin directlly through the Plugins menu of your WordPress.
  2. Activate the plugin through the Plugins menu of your WordPress.
  3. Go to Magic robots.txt settings from the WordPress admin menu to configure the plugin.
  4. Adjust settings to fit your site or maintain the selected defaults and press at Save changes.

FAQ

How can I verify the robots.txt content?

Go to https://www.yoursite.com/robots.txt to see the current robots.txt. Remember your browser may cache it, operate as required to refresh.

Can I modify the robots.txt content?

Select the Use real file option and modify the created file outside the plugin generated block.

There is a bunch of garbage in the robots.txt, can be cleaner?

May be your WordPress is in debug mode, disable it to reduce the included comments.

Can I obtain more information about the generated content in the robots.txt?

Enable debug mode in your WordPress and read it again. If using real file mode, is required to save settings to regenerate the file.

Does the plugin work in subdirectories?

Yes. As virtual file is unavailable because need to be at domain root, the plugin will run in real file mode.

Does the plugin work in multisite?

No, it hasn’t been planned.

Is compatible with Yoast SEO robots.txt?

In virtual file mode we process its data.

Is compatible with Yoast SEO sitemaps?

Yes, we detect and process them.

Is compatible with All In One SEO plugin?

We don’t tried it, testing is needed.

Is compatible with Windows servers?

We don’t tried it, the real robots.txt file mode may fail. If WordPress is installed on a subdirectory of the domain documents root, be sure it will fail on Windows right now. Contact support if you are interested and ready to make some testing.

How to add a new robot?

If the robot obeys robots.txt, contact support and give us this information:

*User-agent.
*Robot information page URL.
*Type: Search engine / link analyzer / advertising management / offline navigator or site downloader.

Robots outside these types are out of the scope of the plugin.

Can I use the plugin to block attacks?

This is not a security plugin, sorry, you should use a firewall or something like fail2ban.

Reviews

නොවැම්බර් 20, 2022
Me ha sorprendido por su sencillez, soy una usuaria sin grandes conocimientos y me fue muy simple comprender el funcionamiento del plugin Magic Robot.txt. No me hicieron falta más que 3 minutos para configurarlo. Muchas gracias por pensar en quienes no somos expertos en informática.
Read all 1 review

Contributors & Developers

“Magic robots.txt” is open source software. The following people have contributed to this plugin.

Contributors

“Magic robots.txt” has been translated into 1 locale. Thank you to the translators for their contributions.

Translate “Magic robots.txt” into your language.

Interested in development?

Browse the code, check out the SVN repository, or subscribe to the development log by RSS.

Changelog

1.0.1

Nov 20 2022
* Syntax corrections.
* Untranslatable strings corrections.

1.0.0

Nov 13 2022
* First public version.