Meta Robots Tags: 18 Directives to Control Indexing & Crawling
Meta robots tags give surgical control over how Google indexes and displays your pages--far more precise than robots.txt. This guide covers all 18 robots directives, when to use each one, and how to keep low-value pages from wasting crawl budget.
TL;DR
- Noindex low-value pages such as filters, pagination and thank-you pages so Google spends its crawling on the pages that matter
- Keep duplicates out of the index by using noindex on parameter URLs, faceted navigation, and print versions
- Control snippets with max-snippet, max-image-preview, and max-video-preview
- Use noarchive if you do not want Google to show a cached link for a page
- Use nofollow selectively on user-generated content and paid links
- Seology audits your meta robots tags and fixes directive issues on connected sites
Why Meta Robots Tags Matter
Meta robots tags control how search engines index and display individual pages. Unlike robots.txt (blocks entire directories), meta robots tags provide surgical, page-level control. Google has limited crawl capacity per site, and on large sites time spent on low-value pages can delay the crawling of important ones.
Using noindex on filters, pagination, and thin content keeps them out of the index, which helps Google focus on your important pages. On e-commerce sites it also keeps duplicate product URLs from being indexed. Note that Google still has to crawl a noindexed page to see the tag, so noindex mainly controls indexing, not crawling.
Meta robots tags also control how pages appear in search results. max-snippet, max-image-preview, and max-video-preview let you decide how much text, image and video Google may show. noarchive stops Google from showing a cached link to the page.
The 18 Meta Robots Tag Directives
Category 1: Core Indexing Directives
Foundation directives for controlling whether Google indexes and follows your pages
1. noindex -- Prevent Page from Appearing in Search Results
noindex tells Google not to include the page in search results. Use on: thank-you pages, internal search results, filters, pagination, staging/dev environments, thin/duplicate content.
<!-- Block page from search results --> <meta name="robots" content="noindex" /> <!-- Common use cases --> <meta name="robots" content="noindex, follow" /> <!-- Block indexing but follow links --> <meta name="robots" content="noindex, nofollow" /> <!-- Block indexing AND following links -->
Result: Noindexed pages drop out of search results once Google recrawls them, so low-value pages stop competing with your important ones.
2. nofollow -- Don't Pass Link Equity Through Links on This Page
nofollow tells Google not to follow links on the page or pass PageRank. Use on: user-generated content pages (comments, forums), untrusted content, paid/sponsored content, login/signup pages.
<!-- Don't follow links on this page --> <meta name="robots" content="nofollow" /> <!-- Combine with noindex for complete blocking --> <meta name="robots" content="noindex, nofollow" />
Result: Nofollow tells Google not to vouch for the links on the page, which suits untrusted or paid links.
3. index, follow -- Explicit Permission (Default Behavior)
index, follow explicitly tells Google to index the page and follow links. This is the default behavior, so you rarely need to specify it--only use when overriding conflicting directives.
<!-- Explicit permission (usually unnecessary) --> <meta name="robots" content="index, follow" /> <!-- Omitting the tag has same effect (default behavior) -->
Note: Only use when robots.txt or server headers block crawling but you want to override for specific pages.
4. all -- Allow Everything (Equivalent to index, follow)
all is shorthand for index, follow. Rarely used since it's the default behavior.
<!-- Allow indexing and following (shorthand) --> <meta name="robots" content="all" />
5. none -- Block Everything (Equivalent to noindex, nofollow)
none is shorthand for noindex, nofollow. Use on pages you want completely hidden from search engines.
<!-- Block indexing AND following (shorthand) --> <meta name="robots" content="none" />
Use cases: Internal tools, admin pages, test environments, private content.
Category 2: Search Result Display Directives
Directives controlling how pages appear in search results (snippets, previews, caching)
6. noarchive (nocache) -- Prevent Google from Caching Page
noarchive prevents Google from showing "Cached" link in search results. Use on: frequently updated content, time-sensitive pages, private/sensitive content.
<!-- Prevent caching (Google syntax) --> <meta name="robots" content="noarchive" /> <!-- Prevent caching (older syntax, some crawlers) --> <meta name="robots" content="nocache" />
Result: Google will not show a cached link for the page. This does not stop anyone from scraping the live page.
7. nosnippet -- Prevent Any Text Snippet in Search Results
nosnippet prevents Google from showing text snippets or video previews. Only shows title and URL. Use on: pages where snippet might reveal sensitive info, premium content teasers.
<!-- No snippet text, no video preview --> <meta name="robots" content="nosnippet" />
Note: Nosnippet also implies noarchive--if no snippet, no cached version shown.
8. max-snippet:[number] -- Limit Snippet Length
max-snippet limits snippet to N characters. Use 0 for no snippet (same as nosnippet), -1 for unlimited (Google decides).
<!-- Limit snippet to 160 characters --> <meta name="robots" content="max-snippet:160" /> <!-- No snippet --> <meta name="robots" content="max-snippet:0" /> <!-- Unlimited snippet (default) --> <meta name="robots" content="max-snippet:-1" />
Result: You set the maximum snippet length Google may show. Very low limits can make your result less informative, so test before applying site-wide.
9. max-image-preview:[setting] -- Control Image Preview Size
max-image-preview controls image thumbnail size in search results. Options: none, standard (default), large.
<!-- No image preview --> <meta name="robots" content="max-image-preview:none" /> <!-- Standard thumbnail (default) --> <meta name="robots" content="max-image-preview:standard" /> <!-- Large preview (recommended for visual content) --> <meta name="robots" content="max-image-preview:large" />
Result: Large image previews allow bigger thumbnails for visual content (recipes, products, infographics), which can make results more engaging in search.
10. max-video-preview:[seconds] -- Limit Video Preview Length
max-video-preview limits video preview to N seconds. Use 0 for static image only, -1 for unlimited.
<!-- Static thumbnail only --> <meta name="robots" content="max-video-preview:0" /> <!-- 30-second preview --> <meta name="robots" content="max-video-preview:30" /> <!-- Unlimited preview (default) --> <meta name="robots" content="max-video-preview:-1" />
Use case: Set 0 for premium video content to prevent full previews; set -1 for marketing videos to maximize exposure.
Category 3: Link & Content Directives
Directives controlling how Google handles links and page elements
11. noimageindex -- Prevent Images from Being Indexed
noimageindex prevents images on the page from appearing in Google Images. Use on: pages with stock photos, copyrighted images, images you don't want competitors using.
<!-- Block images from Google Images --> <meta name="robots" content="noimageindex" />
Result: The images won't appear in Google Images search. This does not stop people from copying them from the live page.
12. notranslate -- Prevent Google from Offering Translation
notranslate prevents Google from showing "Translate this page" option. Use on: pages with code snippets, technical content where translation breaks meaning, pages already multilingual.
<!-- Prevent translation prompt --> <meta name="robots" content="notranslate" /> <!-- Alternative: use lang attribute --> <meta name="google" content="notranslate" />
Use case: Developer documentation, legal content, code examples where machine translation creates errors.
13. nositelinkssearchbox -- Remove Sitelinks Search Box
nositelinkssearchbox prevents Google from showing sitelinks search box in brand SERPs. Use on: sites where internal search is poor quality or shows private content.
<!-- Remove sitelinks search box from brand SERP --> <meta name="robots" content="nositelinkssearchbox" />
Note: Only affects homepage--Google shows sitelinks search box for established brands with good internal search.
14. unavailable_after:[date] -- Remove Page After Specific Date
unavailable_after tells Google to remove page from search results after a specific date. Use on: time-sensitive promotions, event pages, limited-time offers.
<!-- Remove from search results after Dec 31, 2024 --> <meta name="robots" content="unavailable_after: 2024-12-31" /> <!-- ISO 8601 format required --> <meta name="robots" content="unavailable_after: 2024-06-30T23:59:59+00:00" />
Use case: Black Friday sales pages, webinar registration pages, seasonal promotions--auto-deindex after expiration.
Category 4: Advanced Implementation Tactics
Pro-level tactics for robots tag implementation and troubleshooting
15. Use X-Robots-Tag HTTP Header for Non-HTML Files
For PDFs, images, videos--use X-Robots-Tag HTTP header instead of meta tags. Configure in server response headers.
# Apache .htaccess
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>
# Nginx
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}Result: X-Robots-Tag controls indexing for non-HTML files--block old PDFs, private documents, or duplicate images.
16. Combine Multiple Directives with Commas
You can combine multiple directives in one tag using commas. This is more efficient than multiple tags.
<!-- Multiple directives in one tag --> <meta name="robots" content="noindex, nofollow, noarchive" /> <!-- Advanced combination --> <meta name="robots" content="noindex, max-snippet:160, max-image-preview:large" />
Note: All directives apply simultaneously--Google respects all instructions in the tag.
17. Target Specific Bots with Named Tags
Use bot-specific meta tags to control individual crawlers. name="robots" applies to all bots; use name="googlebot" for Google only.
<!-- All bots --> <meta name="robots" content="noindex" /> <!-- Google only --> <meta name="googlebot" content="noindex" /> <!-- Bing only --> <meta name="bingbot" content="noindex" /> <!-- Override: allow Google, block others --> <meta name="robots" content="noindex" /> <meta name="googlebot" content="index, follow" />
Use case: Allow Google but block other bots, or give Google more lenient rules than competitors.
18. Validate with Google Search Console
Use Google Search Console → Coverage report to verify noindex/nofollow implementation. Check "Excluded" tab for pages blocked by robots tags.
Result: Regular validation catches accidental noindex on important pages, such as a homepage or key landing page.
Common Meta Robots Tag Mistakes
- ✗Accidentally Noindexing Homepage or Important Pages:
A stray noindex on the homepage or a key page is easy to ship by accident--always validate robots tags in Google Search Console before deploying site-wide.
- ✗Using Robots.txt Instead of Noindex (Wrong Tool):
Robots.txt blocks crawling but doesn't prevent indexation--if page has backlinks, Google can index URL without crawling. Use noindex for true de-indexation.
- ✗Blocking Robots.txt AND Using Noindex (Conflict):
If robots.txt blocks Googlebot, it can't see noindex tag--page stays indexed with URL. Remove robots.txt block, let Google see noindex, then re-block.
- ✗Using Nofollow Site-Wide (Kills Internal Linking):
Site-wide nofollow prevents internal PageRank flow--only use nofollow on specific pages with untrusted links, never site-wide.
- ✗Not Using Noindex on Pagination (Wastes Crawl Budget):
Pagination pages (/page/2/, /page/3/) waste crawl budget--use
noindex, followon paginated pages (keep follow to pass equity to page 1). - ✗Mixing Canonical Tags and Noindex (Conflicting Signals):
Don't use canonical + noindex on same page--canonical says "index this other page" while noindex says "don't index me". Pick one.
Tools for Meta Robots Tags
- Google Search Console: "Coverage" report shows pages blocked by robots tags--verify noindex implementation and catch accidental blocks
- Screaming Frog SEO Spider: Crawls site and identifies all robots tags--export report of noindex/nofollow pages for audit
- Chrome DevTools: Inspect
<head>section to verify robots tags are present and correct syntax - Robots Meta Tag Generator: Online tools generate proper syntax for complex directive combinations
- Google URL Inspection Tool: Test individual pages to see how Google interprets robots tags--shows indexation status
Worked Example: Cleaning Up a Large Online Store
Suppose a fashion store has far more indexed URLs than real products. Faceted navigation creates filter combinations (/products?color=red&size=M&brand=nike), pagination pages are indexed, and print versions, thank-you pages and internal search results are all crawlable.
A sensible fix:
- Noindex faceted navigation--add
noindex, followto filter URLs (color, size, brand combinations) - Noindex deep pagination--add
noindex, followto /page/2+ so links can still be followed - Noindex utility pages--thank-you pages, print versions, internal search results, wishlist pages
- Use canonical tags on near-duplicate product variants to consolidate signals
- Use max-image-preview:large on product pages so Google may show larger images
How to check it worked: in Google Search Console, watch the indexed page count fall toward your real product count and confirm your product pages stay indexed. Noindexed URLs still need to be crawled before they drop out, so allow time for Google to recrawl.
How Seology Automates Meta Robots Tags
Manual robots tag management requires auditing hundreds of page types, identifying low-value content, implementing tags, and monitoring indexation--taking weeks. Seology handles all of this automatically:
- Automated Page Type Detection: AI identifies filters, pagination, thank-you pages, and other low-value content that should be noindexed
- Smart Directive Recommendations: Suggests optimal robots tags for each page type based on content value and crawl budget
- Bulk Implementation: Applies noindex, nofollow, noarchive, and other directives across thousands of pages automatically
- Ongoing Audits: Re-checks your pages over time and flags crawl budget waste or accidental noindex
- Snippet Optimization: Configures max-snippet, max-image-preview, and max-video-preview for optimal SERP display
- Minimal Manual Work: Connect your Shopify store or GitHub repo and Seology audits robots tags and applies fixes--directly on Shopify, or as pull requests on GitHub
Automate Your Meta Robots Tag Optimization
Seology audits indexation issues and fixes robots directives on connected sites--cutting crawl waste without manual tag management.
Start FreeFinal Verdict: Meta Robots Tags Give Page-Level Control
Meta robots tags give you page-level control over what Google indexes and displays. Unlike robots.txt (directory-level blocking), meta robots provide page-level precision for filters, pagination, and thin content.
Focus on noindex, follow for pagination and filters (blocks indexing but preserves internal linking), noarchive for content you do not want cached (removes the cached link), and max-image-preview:large for visual products (allows larger image previews). Always validate in Google Search Console so an accidental noindex does not hit your homepage or key pages.
Ready to optimize meta robots tags automatically? Seology audits indexation issues and fixes robots directives on connected sites--without manual tag management. Start free today →
Related Posts:
Tags: #MetaRobotsTags #Noindex #Nofollow #CrawlBudget #TechnicalSEO #Indexation #Seology #SEOAutomation
Part of the Technical SEO Guide
Put this into practice with the Seology SEO automation.
Related articles
Robots.txt Configuration: Control What Google Crawls
One robots.txt mistake can deindex your entire site. This guide shows the exact configuration used by Fortune 500 sites.
Mobile-First Indexing 2026: The Complete SEO Guide
Master mobile-first indexing in 2026. Google indexing differences, content parity, Core Web Vitals thresholds, and URL Inspection troubleshooting.
Mobile-First Indexing: The Complete Checklist
Google mainly uses the mobile version of your pages. This 27-point checklist helps you find and fix mobile indexing problems.
Canonical Tags: The Definitive Guide to Fixing Duplicate
Canonical tags prevent duplicate content penalties. This guide shows when and how to use rel=canonical correctly.
Robots.txt & Crawl Control for Shopify: 2026 Guide
30% of websites have robots.txt errors harming SEO. 70% of searches influenced by AI crawlers by 2026.
Twitter Card Meta Tags: Setup, Types & Fixes (2026)
How to implement Twitter Card meta tags for rich link previews on X. Setup, validation, card types, and troubleshooting guide.