As we all know, the primary function of the Google, as well as the other major search engine bots, is crawl and index the pages from the website. But what we do if we don’t want to add a particular Post or a Page in the Serch index.
Well, the meta robots tag was an open standard created over a decade ago and designed initially to allow the website owner to tell Google bots which page you want to add in the search index and which one is not.
We already have a robots.txt file to do this work, but robots meta tag is a little advance extension of the robots.txt file, in which we place meta tag in the head of the HTML documents.
In this tutorial, we will discuss, what is robots meta tag, how it looks, and its different values as well as its function.
And, In the upcoming tutorial, we will explore it more and learn its two dominated feature which is nofollow and noindex meta tags to control the link juice of the website and improve the quality of the Overall website.
So, lets we dig-dipper and try to understand the syntax of robots meta tag and its different value.
What is Robots meta tag?
Robots Meta Tag is a special HTML <META> tag to tell robots not to index all the content of the page (or a particular page), and/or not scan it for links to follow.
For example:
<html>
<head>
<title>…</title>
<meta name=”robots” content=”selection“>
</head>
<body>….</body>
Selection: You can define the value (an action that you want to tell google)
Robots: Robots is used for all the bots, but you can define particle bots in place of Robots
Just, for an example, look at the below code,
<meta name=”googlebot” content=”noindex“>
Here, I’m blocking, content from a specific search engine is Google bot.
Look at the next example,
<meta name=”robots” content=”noindex“>
Here I’m telling all the bots (by defining robots) no need index my content.
Difference Between Robots Meta Tag & Robots.txt File
It’s important to understand our terms before we dive into the world fo the SEO. Both serve a similar purpose of informing to search engines of special handling, but they work on different levels.
Robots.txt file is a simple TEXT file and placed on the server level, in which we use a command to block the whole website or a whole area of a website.
Where, Robota meta tag is an HTML Meta tag and placed on the page level, in which we placed the Meta tag in the Head of the document to tell search engine No need to index that page.
The robots meta tag gives you a finer level of control than robots.txt.
Proper Sytex of Robots meta tag
Here is a syntax that you can use in your HTML document to add robots meta tag.
<meta name=”robots” content=”selection“>
You can also define more than one value in single HTML tags, for example
<meta name=”robots” content=”noindex, nofollow“>
Googlebot understands any combination of uppercase and lowercase. So each of these meta tags is interpreted in exactly the same way:
<meta name=”robots” content=”noodp“>
<meta name=”Robots” content=”NoOdp“>
<meta name=”ROBOTS” content=”NOODP“>
Anyone can see this code by simply checking the source code of the page.
- Press Cltr+U in the chrome browser and use search command Cltr+F.
<meta name=”robots” content=”noindex“>
Robots-> refers to all search engines while means all bots
noindex-> disallows the search engine to index this page.
nofollow-> disallows the search engine to follow the link on a page.
Types of Values & Its function
There is a lot of robots meta tag and over the years, various search engines have added additional support to the tag. Here is a list of most use robots meta tag on the website.
- Index: Allow search engines robots to index the page and show a cached link in search results. you don’t have to add this to your pages. It’s the default.
- noindex –Disallow the search engines from showing this page in their results.
- follow – Tells the search engines robots to follow the links on the page, whether it can index it or not.
- nofollow – Tells the search engines robots to do not follow any links on the page at all.
- none – This is a shortcut for noindex, nofollow, or basically saying to search engines don’t do anything with this page at all.
- noarchive – Prevents the search engines from showing a cached copy of this page.
- nocache – Do not show a cached link in search results, same as noarchive but only used by MSN/Live.
- nosnippet – Do not show a snippet for the page in search results
- noodp – Do not use the metadata from the Open Directory Project for titles or snippets for this page
- noydir – Do not use the metadata from the Yahoo! Directory for titles or snippets for this page
- notranslate – Do not offer translation for the page in search results
- noimageindex – Disallow search engines from spidering images on that page. Of course, if images are linked to directly from elsewhere, Google can still index them, so using an X-Robots-Tag HTTP header is a better idea.
- unavailable_after: [RFC-850 date/time] – Do not show the page in search results after a date and time specified in the RFC 850 format
List of search engine that Supports Robots meta tag
| Robots value | Yahoo! | MSN / Live | Ask | |
|---|---|---|---|---|
| index | Yes | Yes | Yes | Yes |
| noindex | Yes | Yes | Yes | Yes |
| none | Yes | Doubt | Doubt | Yes |
| follow | Yes | Doubt | Doubt | Yes |
| nofollow | Yes | Yes | Yes | Yes |
| noarchive | Yes | Yes | Yes | Yes |
| nosnippet | Yes | No | No | No |
| noodp | Yes | Yes | Yes | No |
| noydir | No use | Yes | No use | No use |
Note: If the robots.txt and meta tag instructions for a page conflict, Googlebot follows the most restrictive.
For example;
- If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page.
- If you allow a page with robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.
Resouce
- Google clear how to use it in his Google Webmaster Central Blog
- Bing has a webmaster help page for it
- Yandex also has a help page for HTML meta tags.
Conclusion
Robots Meta Tag can use to increase the quality of the overall website and help you to control the page that you want to add the search indexed. A well use of this HTML Meta Tag can increase the Domain authority of the website and make strong relation in the eye of google.
If you have any suggestion or problem about no index no follow meta tag please feel free to comment below
