Robots directives explained
Robots directives tell compliant crawlers how they may treat a page or resource. In HTML, they can appear in a meta element such as meta name="robots" content="noindex,follow". Similar instructions can be sent through the HTTP X-Robots-Tag header, which is especially useful for non-HTML resources.
Continue exploring: Use the ViewSourcePage source analyzer to inspect a public page, then compare your findings with our guide on Website Performance: What You Can Spot in Page Source.
Index and noindex
The index directive allows indexing while noindex asks a crawler not to include the page in search results. A noindex page can still be accessible to users. Before publishing a page intended to rank, inspect the source and confirm that an accidental noindex is not present.
Follow and nofollow
Follow and nofollow relate to whether crawlers should follow links. Modern search engines treat these directives as signals rather than a simple guarantee. If a site needs strong internal discovery, make sure important links are not unnecessarily restricted.
Robots meta versus robots.txt
Robots.txt controls crawler access at the URL path level, while robots meta directives are delivered inside a page and can provide indexing instructions. They are related but not interchangeable. Blocking a page in robots.txt can prevent a crawler from seeing a noindex directive inside the page.
X-Robots-Tag
HTTP headers can send crawler instructions for PDFs, images and other non-HTML files. A source viewer that only reads HTML cannot always reveal this header. Use network or HTTP header inspection when you need a complete robots audit.
How to check the source
Open the HTML source and search for name="robots". Also look for googlebot-specific directives if the site uses them. Record the values and compare them with the page’s intended role, such as public article, private utility, staging page or duplicate URL.
Common mistakes
Accidental noindex directives can occur during staging or template changes. Another mistake is assuming that robots.txt is a security boundary. It is not. Sensitive information should be protected with authentication and authorization, not merely omitted from a robots file.
Source inspection and migrations
During a redesign or domain migration, robots directives deserve special attention. A staging environment may correctly use noindex, but that directive must not accidentally follow the site into production. Source checks across representative templates can catch this kind of deployment error.
A simple checklist
Check the HTML robots meta tag, HTTP X-Robots-Tag when relevant, robots.txt, canonical URL and sitemap inclusion. These signals should tell a consistent story. A page intended to rank should not simultaneously declare noindex or point its canonical at an unrelated URL.
Conclusion
Robots directives are easy to overlook because they are often invisible to normal visitors. Page source inspection puts them in front of you and makes accidental indexing restrictions easier to spot. For a complete audit, combine HTML, HTTP header and robots.txt checks.
A practical checklist
Use this checklist when reviewing a real page so the process stays repeatable rather than relying on memory.
- Find robots meta
- Check X-Robots-Tag
- Review robots.txt
- Check canonical and sitemap
- Confirm production templates
Frequently asked questions
Does source inspection show private backend code?
No. A browser receives only the public response needed to render or process the page. Private application logic, databases, server credentials and protected files remain on the server.
Is a source viewer a replacement for a full audit?
No. It is a focused inspection layer. Use dedicated SEO, accessibility, performance, security and analytics tools when you need deeper evidence.
Final thoughts
Source inspection is valuable because it turns an invisible technical layer into something you can read and reason about. The best workflow is simple: observe the public markup, identify a specific question, verify the relevant element, and then use a specialist tool when the question goes beyond HTML. This keeps technical analysis practical, evidence-based and easy to repeat across pages.
Deeper practical notes
Indexing directives should be treated as deployment-sensitive configuration. A staging site often uses noindex, but that setting can accidentally survive a production release. A source check on the live site is therefore a valuable post-deployment smoke test, especially after changing templates or SEO plugins.
Robots directives are not a substitute for authentication. If a page contains private information, putting noindex in HTML does not protect it. A crawler or user who can access the page can still receive the content. Sensitive resources need access controls, authorization and secure server configuration.
When a page is blocked by robots.txt, a crawler may not be able to retrieve the HTML and therefore may not see its meta robots directive. This is one reason the two mechanisms should be designed together. Use robots.txt for crawl management and page-level directives for indexing behavior where appropriate.
Specific crawler directives can complicate templates. A site may include a general robots meta tag and a crawler-specific tag. If the values conflict, the final behavior can be difficult to reason about. Keep directives simple and document why exceptions exist.
Non-HTML resources need header inspection. A PDF can carry an X-Robots-Tag response header even though there is no HTML head to inspect. If your audit covers documents, feeds or images, add HTTP header checks to the workflow rather than relying solely on a source viewer.
Always verify the intended outcome in Search Console or another indexing diagnostic after making a major change. Source tells you what the page declares; search tools tell you what search systems have observed and indexed. Those are different kinds of evidence.
Applying the guide in a real audit
When applying this workflow to robots meta tags and x-robots-tag: what source inspection reveals, begin with one concrete question rather than trying to audit everything at once. A focused question makes source inspection faster and produces evidence that can be shared with a developer, editor or SEO specialist. Record the page URL, the element you inspected and the observation you made. This simple habit turns an informal browser check into a repeatable technical review.
It is also useful to compare more than one page template. A homepage, article, category page and utility page can have different HTML structures even when they share the same brand. Looking at several examples helps distinguish a one-off issue from a template-level problem. If the same markup appears across many pages, the best fix is usually at the template or component level rather than by editing individual pages.
Keep the difference between evidence and interpretation clear. Seeing an element in source is evidence that the public response contains that element. It is not automatically evidence that a search engine indexed it, that a browser loaded every referenced asset, or that a backend process is secure. Strong technical analysis connects source observations with the right secondary test instead of making conclusions that the HTML alone cannot support.
Another practical benefit is documentation. A source viewer with readable line numbers makes it easier to describe an issue precisely: for example, a missing canonical, an unexpected robots directive, a duplicate heading, or a third-party script reference. Clear documentation shortens the feedback loop between SEO teams, developers and content owners because everyone can discuss the same technical evidence.
Finally, treat the public source as part of the user experience. The document should be understandable, semantically organized and consistent with the visible interface. Good HTML structure supports accessibility, maintainability and search understanding at the same time. A clean source document is not a guarantee of quality, but it is a strong foundation for a website that is easier to maintain and easier for people and machines to interpret.