The Standard robots.txt Sample File for WordPress (Secure & SEO)

Using standard robots.txt file and optimizing help your website more secured and SEO better. It also help you fix error “your page is not mobile-friendly” when searching for your website on search engines.

What is robots.txt file?

Robots.txt is shortly known as a file initiatively created for the purpose of taking control of searching engines bots, spider (crawl, index…) by webmaster concerning collectable or uncollectible information. Hence it makes the website more secured and SEO better.

You can view robots.txt as a board of regulations about your website. When the bot gets access to your website, it reads data in the robots.txt file to know whether it is able to collect information or not, or if they can, which categories they can collect information from?

Once understanding the importance of robots.txt file, its configuration has to be taken good care of.

Why should we use robots.txt file?

You probably don’t want to use the file because it is too complicated or unnecessary

It might not be the case, if it is too complicated, there are many tools helping you create such a file

Saying that it’s not necessary might not be correct because it secures your website and helping to conceal your website sensitive files from users, especially from the searching tool.

Without robots.txt file, the searching tool would index all of the files on your website, including sensitive data like control files of your website.

Therefore, you must have a robots.txt file in your source code. To use a robots.txt file in a proper way, you have to understand the rules of creating it first.

Rules to configure a robots.txt file

In a robots.txt file there are usually following categories:

• User-agent: what bot can gain access to your website
• Disallow: not allowing bot to gain access to a certain part that you require.
• Allow: allowing bot to gain access to a certain part that you require.
• Sitemap: the guide map for bot to follow the path you regulate

There are some symbols:

• * : symbolizing everything
• / : symbolizing file, folder location
• # : symbolizing comments

With the above example, you are much likely to know how to configure a robots.txt file. A thing to pay attention to is that robots.txt can differentiate between lowercase and uppercase, blank space or strange characters; therefore you have to be careful not to make mistakes.

What to notice when optimizing robots.txt file

• Every line is a statement
• Avoid odd or missing characters.
• Don’t use blank space or strange characters
• Uppercase and lowercase words are different
• Created robots.txt file has to be in the same line as (root directory)
• The name has to exactly be “robots.txt”. Avoid using the wrong name or uppercase characters such as “RoBots.txt” or “Robots.txt”.

Create a standard robots.txt (sample file) for your website

You can check whether your website has robots.txt file or not by clicking on this link: http://yourdomain.com/robots.xt. If the browser doesn’t display the content or showing “404 page not found”, or redirect to the home page. That means you haven’t had robots.txt file yet. You need to create a robots.txt for your website.

   1. Create robots.txt file automatically for your website with internet tools

Visit this website: http://seocheckpoints.com/robots-txt-generator

Select options for your robots.txt file based on which purpose you want for your website. Then click on “Create and Save as Robots.txt” button. Finally upload robots.txt file to your root directly website (same location with index.php)

   2. Standard robots.txt file for WordPress website

This is the standard robots.txt file for WordPress (reference our robots.txt file here: https://ohiwill.com/robots.txt)

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /search?q=*
Disallow: /readme.html
Disallow: /wp-includes/
Disallow: /wp-content/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-includes/*.js
Allow: /wp-includes/*.css
Allow: /wp-content/cache/*.js
Allow: /wp-content/cache/*.css
Allow: /wp-content/cache/*.gzip
Allow: /wp-content/themes/*.js
Allow: /wp-content/themes/*.css
Allow: /wp-content/themes/*.ttf
Allow: /wp-content/themes/*.woff
Allow: /wp-content/themes/*.svg
Allow: /wp-content/plugins/*.js
Allow: /wp-content/plugins/*.css
Allow: /wp-content/plugins/*.ttf
Allow: /wp-content/plugins/*.woff
Allow: /wp-content/plugins/*.svg
Allow: /wp-content/uploads/*.jpg
Allow: /wp-content/uploads/*.png
Allow: /wp-content/uploads/*.gif
Allow: /wp-content/uploads/*.mp3
Allow: /wp-content/uploads/*.mp4
Allow: /wp-content/uploads/*.ogg

With this robots.txt file, your website will be well indexed and fixed such error as “your page is not mobile-friendly” when searching for your website on Google (your index website is still maintained with searching tools). By using this robots.txt file, these search engines (here is Google) will render your pages correctly, guarantee all page resources could be loaded ((js, css…). So it helps how these search engines see and understand your page. Before, our website usually encountered this error but we already fixed it thanks to using this robots.txt file and Google Search Console to update the robots.txt file (with “Ask Google to update” button)

and this is result when using correct robots.txt file

I hope this article will help you understand more about robots.txt file. If you have any problem with robots.txt please let us know in the comment below.