The second line of any directive block is the "Disallow" line

[sharminakter]

The User-agent directive
The first line of each directive block is the "user-agent", which identifies the crawler it is aimed at.

So, if you want to tell Googlebot not to crawl your WordPress admin page, your directive will start with:

User-agent: Googlebot
Remember that most search engines have multiple crawlers. They use different crawlers for their regular index, images, videos, etc.

Search engines always choose the most specific block of directives they can find.

Let's say you have three sets of directives: one for *, one for Googlebot, and one for Googlebot-Image .

If the Googlebot-News user agent crawls your site, it will follow Googlebot directives.

In contrast, the Googlebot-Image user agent will india telegram data follow the more specific directives of Googlebot-Image .

Here you will find a detailed list of web crawlers and their different user agents.

The Disallow Directive

You can have multiple "Disallow" directives that specify parts of your site that the crawler cannot access.

An empty "Disallow" line means that you are not disallowing anything, and a crawler can access all sections of your site.

For example, if you want to allow all search engines to crawl your entire site, your block would l