Version History

The main logic behind this version is:
OpenAI uses OAI-SearchBot for ChatGPT search visibility, while GPTBot is the training crawler; ChatGPT-User is user-initiated and not the main automatic search crawler. Google’s AI features in Search are governed by normal Googlebot crawling, while Google-Extended is the separate control for Gemini/Vertex training and grounding. Perplexity uses PerplexityBot for search surfacing, and Anthropic distinguishes ClaudeBot, Claude-SearchBot, and Claude-User.

For the blocked competitors, Semrush documents SemrushBot and SiteAuditBot, Ahrefs documents AhrefsBot and AhrefsSiteAudit, Majestic documents MJ12bot, and Moz’s common crawler tokens are Rogerbot and DotBot.

One caution: robots.txt is advisory, not hard security. User-agents can be spoofed, and some user-triggered fetchers are not governed the same way as automatic crawlers. For strict enforcement, mirror these rules in Cloudflare/WAF/server bot rules and allowlist approved bot IP ranges where vendors publish them.

Version History Overhaul

• Slug-based Routing:

  2. Moved detail pages to /version-history/[slug].

  3. Updated Admin UI to automatically generate slugs from version numbers and titles.

  4. Updated List page and Homepage to use slugs for navigation.

Rich Sidebar (Detail Page)

• 1. Added a two-column layout to the version detail page.

  2. Implemented sidebars with accordions for:

     • Recent Updates: Last 5 versions.

     • Latest Articles: Last 5 blog posts.

     • Podcasts: Last 5 podcast episodes.

• Homepage Accordion:

  • Replaced the card-based version list with a clean accordion-style list of the latest 5 updates.

Refactor Single Client Dashboard to Accordion Layout

Planning and Analysis

• Implement Technical SEO Audit History Accordion

• Update header to include summary stats and "New Audit" button

• Ensure accordion functionality works as expected

• Implement PageSpeed Performance Accordion

• Update header to include summary metric scores and "Monitor Details" link

• Implement paginated table (10 results per page)

• Ensure accordion is collapsed by default

Refine Layout and Information Architecture

Implement Two-Column Layout (Left: Client Info, Right: Audit Results)

Make Header Sticky with Save Button

Collapse all sections by default

Implement full-width layout

Fix "Monitor Details" deep linking

Refine PageSpeed Monitor Page

Handle dashboardId from URL params

Wrap Audit Scheduling in Accordion (Collapsed)

Wrap Audit History in Accordion (Collapsed)

Verification and Polish

Test accordion interactions

Verify data fetching and display

Final UI/UX review

Phase 6: Advanced Dashboard Integration & Persistence

Redesign Dashboard Header:

Make header fixed/sticky with high z-index

Integrate aggregate scores (Client Score, Tech SEO, PageSpeed)

Add "Save as Preset" and "Delete Preset" functionality

Enhance GSC Performance Controls:

Add checkboxes to

GscStatsCards

for metric toggling

Implement

GscDateSelector

with GSC-style presets (24h, 7d, 28d, 3m, 6m, 12m, custom)

Update

PerformanceChart

to support dynamic line visibility

Improve GSC "Pages" Insights:

Add "PageSpeed" and "Tech SEO" status columns linked to audit data

Implement toggle for audit data integration in GSC header

Migrate Domain Analysis to Sub-collection:

Create domain_analysis sub-collection for history

Update UI to show analysis count and latest date in accordion header

Implement Settings Persistence:

Persist "Rows per page" and "Metric selection" per dashboard

Phase 7: PageSpeed Scheduling & History Enhancements

Update Types: Add auditType to

PageSpeedAudit

Dashboard Integration: Display scheduling settings in user dashboard

PageSpeed Monitor UI: Add Desktop/Mobile tabs for Detailed History

Audit Persistence: Ensure auditType (manual/scheduled) is recorded

Component Refinement: Update

AuditHistory

for tabbed navigation

Phase 8: Sitemap Discovery & URL Management

Create Sitemap Fetcher API: Parse XML sitemaps and indexes

UI Integration: Add Sitemap section to Domain Analysis

Firestore Persistence: Store sitemap URLs in sitemaps_settings and sitemap_urls

Integrated Auditing: Allow URL selection from sitemaps in Tech SEO and PageSpeed pages

Phase 9: PageSpeed Monitor Enhancements & Refinements

Debug & Fix Sitemap URL Picker (Ensure URLs are stored and fetched correctly)

PageSpeed Data Model: Ensure pageUrl is tracked consistently

PageSpeed Monitor UI: Add Page Selector/Filter (All, Specific Pages)

PageSpeed Charts: Update Trends and Vitals to respect Page Filter

PageSpeed History: Add "Page" column to tables

Client Dashboard: Update PageSpeed Accordion with Page Filtering & URL visibility

Create Admin UI for Version History

Create Public Version History Pages

Integrate Version History with Homepage

Phase 3: Advanced Dashboard Refinements

Implement Brand Name Repeater (Multiple Names)

Implement Domain Analysis System

Enhance Sticky Header

Implement Google Search Console (GSC) Import (v1)

Phase 4: GSC Data Expansion & Sub-collection Storage

Migrate GSC data to gsc_reports sub-collection

Expand Links tab to include sub-navigation:

External Links

Top Linking Sites

Internal Links

Top Linking Text

Implement CSV parsing for all 4 link categories

Create detailed tables for each link category

Update data fetching to use the new sub-collection

Implement data deletion for all GSC report types

Phase 5: GSC Performance Dashboard Refactor

Update

GscReportType

with sub-performance types (Queries, Pages, Countries, Devices, Appearance, Days)

Update

handleGscImport

with advanced CSV parsers for each sub-type

Implement Performance Overview Stats Cards (Real-time calculation)

Implement Dynamic Line Chart for Clicks/Impressions (driven by perf_days)

Implement Nested Tabs for Performance Table:

Queries

Pages

Countries

Devices

Search Appearance

Days

Implement Advanced Pagination (Rows per page, page selector)

Implement Sortable Table Headers

Add "Delete Data" functionality for all performance sub-types