Scope: HubSpot SEO – Confirm Correct Pages Are Blocked by robots.txt

1. Audit & Identification

• Review the existing robots.txt file for the HubSpot website
• Identify all pages, directories, and files currently blocked from search engine crawlers
• Compare blocked pages against the site’s SEO strategy and content visibility requirements

2. Analysis & Strategy

• Determine which pages should be blocked (e.g., admin pages, thank-you pages, duplicate content)
• Identify pages that should not be blocked but currently are
• Develop a plan to adjust the robots.txt file for optimal crawl control and SEO performance

3. Implementation & Optimization

• Update robots.txt directives to block or allow pages as needed
• Ensure correct syntax and compatibility with major search engines
• Test the file using HubSpot SEO tools and Google Search Console to confirm proper implementation
• Document all changes for future reference and website governance

4. Technical Verification

• Verify blocked pages return proper “Disallow” directives to crawlers
• Confirm allowed pages are accessible and indexable by search engines
• Use crawling tools to simulate search engine behavior and check for unintended blocks

5. Monitoring & Reporting

• Continuously monitor search engine indexing to ensure only intended pages are blocked
• Track HubSpot SEO and Google Search Console reports for crawl errors or blocked content issues
• Provide periodic reports highlighting robots.txt compliance and recommendations

6. Continuous Improvement

• Establish best practices for future content to ensure proper inclusion or exclusion in robots.txt
• Train content and technical teams to review crawl directives during page creation or updates
• Periodically audit robots.txt file as the website evolves to maintain SEO compliance

Outcome:

• Only intended pages are blocked from search engines
• Improved crawl efficiency and SEO performance
• Reduced risk of accidentally blocking important content
• Standardized robots.txt management within HubSpot CMS