What is Meta Robots Tag?
The meta robots tag is an HTML element placed in the page head that provides directives to search engine crawlers about how to index and follow links on specific pages, offering page-level control beyond site-wide robots.txt rules. Common directives include noindex (don’t add to search index), nofollow (don’t follow links), noarchive (don’t show cached versions), and nosnippet (don’t display text snippets), enabling granular crawling and indexing management critical for technical SEO.
Ecommerce SEO Glossary > On-Page SEO > Meta Robots Tag
What You Need to Know about Meta Robots Tag
Noindex Directive Usage
The noindex directive prevents pages from appearing in search results while still allowing crawlers to access and follow links, making it ideal for low-value pages like filter combinations, thank-you pages, or duplicate content you need crawlable for link equity but don’t want indexed.
Nofollow Link Control
Nofollow in meta robots tells crawlers not to follow any links on the page, preventing PageRank flow to linked pages. This site-wide link nofollow differs from individual link-level nofollow attributes, with limited legitimate uses since blocking all link equity flow rarely benefits site architecture.
Combined Directive Strategy
Multiple directives can be combined like to prevent both indexing and link following. This aggressive combination suits pages you want completely excluded from search engine consideration while maintaining technical accessibility.
X-Robots-Tag HTTP Headers
Server-level X-Robots-Tag headers provide identical functionality to meta robots tags but work for non-HTML files like PDFs, images, and other resources. This HTTP header approach enables crawler control for file types where HTML meta tags can’t be inserted.
Specific Bot Targeting
Tags can target specific crawlers like to give different instructions to different search engines. This granular control enables varied treatment across search engines when platform-specific strategies require different indexing approaches.
Common Implementation Mistakes
Accidentally noindexing important pages, combining noindex with robots.txt blocks (preventing crawlers from seeing the noindex directive), or forgetting to remove temporary noindex tags after development cause frequent indexing problems that silently eliminate search visibility.
Frequently Asked Questions about Meta Robots Tag
1. When should you use noindex versus robots.txt?
Use noindex for pages you want crawled for link equity but not indexed, since crawlers must access pages to see noindex directives. Use robots.txt only to prevent crawling entirely for resource management, understanding it blocks both crawling and prevents seeing noindex instructions.
2. Does nofollow in meta robots affect all links?
Yes, meta robots nofollow affects all links on the page, preventing PageRank flow through any of them. For selective link control, use individual link-level nofollow attributes on specific anchor tags rather than page-wide meta robots directives.
3. Can you override meta robots with X-Robots-Tag?
X-Robots-Tag HTTP headers and meta robots tags work together, with most restrictive directives taking precedence. If both exist with conflicting instructions, search engines typically honor the most restrictive directive between the two implementation methods.
4. Why would pages be noindexed but still crawled?
Noindex allows crawling so search engines can see the noindex directive and follow links for PageRank flow, while removing pages from indexes. This combination is strategic for maintaining site architecture and link equity distribution without creating indexed low-value pages.
Explore More EcommerCe SEO Topics
Related Terms
Let’s Talk About Ecommerce SEO
If you’re ready to experience the power of strategic ecommerce seo and a flood of targeted organic traffic, take the next step to see if we’re a good fit.