How to create a robots.txt file?
How to create a robots.txt file?
What is it? The robots.txt file is a plain text file that must comply with the robots exclusion standard.
You can create the file with Windows Notepad and save it with the name robots.txt
This file consists of one or several rules, and each of them blocks or allows access for a specific crawler to a particular file path of a website.
The robots.txt file is used to manage the traffic of crawlers to your site.
It is used to prevent the requests your website receives from overloading it; with a well-configured robots.txt file, you can prevent the speed of your web or even the Cloud itself from being negatively affected when you receive several visits from these indexers at the same time.
What do we block? Crawlers, also known as spider crawlers, robots, or bots. It is a program that analyzes the documents of the website. Search engines use very powerful crawlers that navigate and analyze websites, creating a database with the collected information.
What elements make up the robots.txt? When generating the robots.txt file, you must take into account the commands and specific rules.
Commands User agent: This is the command used to specify the search engine robots/spiders that we allow to crawl our web.
The syntax for this command is: User-agent: (robot name)
(Each rule must have at least one Disallow or Allow entry)
Disallow: Indicates a directory or a page of the root domain that you do not want the user-agent to crawl.
Allow: Indicates the directories or pages of the root domain that the user-agent specified in the group should crawl. It serves to override the Disallow directive and allow a specific subdirectory or page of a blocked directory to be crawled.
One option is to put an asterisk; this means that you allow all search engines to crawl the web.
User-agent: (*)
Disallow
The following command is to indicate to search engines that they should not crawl, access, or index a specific part of the web, such as the wp-admin folder.
Disallow: /wp-admin/
Allow
With the following command, you indicate the opposite; you mark for search engines what they can crawl. In this example, it only allows one file from a specific folder.
Allow: /wp-admin/admin-ajax.php
Other elements to consider.
When adding elements for blocking, you must place the forward slash (/) at the beginning and end. The code can be simplified. *. The asterisk is used to block a sequence of characters. $. The dollar symbol is used when you want to block URLs with a specific ending.
Examples of commands used in robots.txt.
Exclude all robots from the server:
User-agent: *
Disallow: /
Allow all robots to have access to scan everything:
User-agent: *
Disallow:
Exclude only one bot, in this case Badbot:
User-agent: BadBot
Disallow: /
Allow only one bot, in this case Google:
User-agent: Google
Disallow:
User-agent: *
Disallow: /
Exclude a directory for all bots:
User-agent: *
Disallow: /directory-name/
Exclude a specific page:
User-agent: *
Disallow: /page-url.html
Block images of the web:
User-agent: Googlebot-Image
Disallow: /
Block an image for only one bot:
User-agent: Googlebot-Image
Disallow: /image/blocked.webp
Exclude a specific file type:
User-agent: Googlebot
Dissallow: /*.webp$
Exclude URLs with a certain ending:
User-agent: *
Disallow: //pdf$
These are usage examples; use the one that fits your needs or create a custom one.
Once the robots.txt file is created, upload it via FTP within the directory /yourdomain/data/web/