# Build Strategy: SEO + AI Search Optimization Platform

A roadmap for evolving this toolkit from an audit engine into a full optimization platform that stays relevant as search shifts from Google's ten blue links to AI-generated answers.

---

## 1. The landscape you're optimizing for

Search is now three overlapping games, and your tool must play all of them:

**Classic SEO (Google/Bing organic).** Still the largest traffic source for most businesses. Ranking depends on technical health, content relevance, backlinks, and Core Web Vitals. This is a mature, slow-moving game.

**AEO — Answer Engine Optimization (Google AI Overviews, Bing Copilot, featured snippets).** The engine reads your page and lifts an answer directly into the results. Winning requires question-phrased headings, concise answer-first paragraphs, and FAQ/HowTo schema.

**GEO — Generative Engine Optimization (ChatGPT, Claude, Perplexity, Gemini).** LLMs decide whether to *cite* you when answering users. Research on generative engines shows citations favor pages with concrete statistics, quotations, named sources, clear entity identity (Organization schema, consistent NAP data, Wikipedia/Wikidata presence), and machine-readable structure. New conventions like `llms.txt` and permissive robots.txt rules for GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot determine whether you're even in the candidate pool.

The strategic insight: **the same well-structured, factual, entity-rich content wins all three games.** So the tool is built as one pipeline with three scoring lenses, not three separate tools.

## 2. Architecture (what's built and why)

The current codebase is deliberately modular so each layer can be swapped without touching the others:

```
config.yaml  →  runner.py (orchestrator + scheduler)
                   ├── audit.py    technical SEO crawl + scoring
                   ├── geo.py      AI-readiness scoring + llms.txt/schema generators
                   ├── content.py  keyword coverage + lead-gen signals
                   └── report.py   JSON/HTML reports + run-over-run diffs
```

Design principles baked in:

1. **Config-driven everything.** Modules toggle on/off, cadences and score weights are tunable, multiple sites run from one config. New functionality = new module + one config flag.
2. **Polite, white-hat crawling.** Respects robots.txt, rate-limits itself, identifies its user agent. This isn't just ethics — aggressive crawlers get IP-banned and scraping-based "guaranteed ranking" tactics get sites penalized.
3. **Diffable history.** Every run is stored; the scheduler compares against the previous run and alerts on regressions. Rankings are won by catching decay (broken schema, deleted meta tags after a CMS update) within hours, not months.
4. **Generators, not just audits.** The tool produces paste-ready artifacts: `llms.txt`, Organization/LocalBusiness JSON-LD, FAQ schema. Auditing tells you what's wrong; generators cut the fix time to minutes.

## 3. Phased roadmap

### Phase 1 — Foundation (done: this codebase)
Crawl, four-lens scoring (technical / AI-readiness / content / lead-gen), scheduled re-audits with regression alerts, schema and llms.txt generation, HTML dashboards.

### Phase 2 — Data integrations (weeks 2–6)
- **Google Search Console API**: pull real queries, impressions, CTR per page — replaces guesswork with actual demand data. The config already has the hook.
- **PageSpeed Insights API**: real Core Web Vitals (LCP, INP, CLS) instead of response-time heuristics.
- **Rank tracking**: use a legitimate SERP API (DataForSEO, SerpApi, or GSC position data) rather than scraping Google, which violates its ToS and produces unreliable data.
- **AI citation tracking**: periodically ask ChatGPT/Perplexity/Claude APIs the questions your customers ask ("best coffee shop in Indiranagar") and log whether your site is cited. This becomes your GEO rank tracker — almost nobody measures this yet, which is your edge.
- **Alerting**: wire the Slack webhook and email hooks so regression alerts reach humans.

### Phase 3 — LLM-assisted optimization (weeks 6–12)
Add an `ai_writer` module that calls an LLM API to:
- Rewrite titles/meta descriptions for pages the audit flags, constrained by your keywords.
- Draft FAQ sections (question + 40–60-word answer) from existing page content, output as ready-to-paste HTML + FAQPage schema.
- Generate content-gap briefs by comparing your keyword coverage against competitor snapshots.
Keep a **human-approval step** — auto-publishing AI text unreviewed risks quality collapses and Google's scaled-content-abuse policy.

### Phase 4 — Continuous publishing loop (months 3–6)
- CMS connectors (WordPress REST API, Webflow, Shopify) so approved fixes deploy with one click.
- A review queue UI (FastAPI + React, or even a Streamlit panel) showing: issue → proposed fix → approve/reject.
- Lead-side automation: form-to-CRM webhooks, UTM-tagged CTA experiments, review-request email sequences (reviews feed both Google local rank and AI-answer trust).

### Phase 5 — Multi-tenant SaaS (if you commercialize)
- Postgres for report history, Celery/Redis replacing APScheduler for distributed job queues, per-client dashboards, white-label PDF reports. The module boundaries above map cleanly onto microservices when scale demands it.

## 4. Future-proofing: how to stay relevant for 5+ years

**Bet on entities, not keywords.** Knowledge graphs and LLMs reason about *entities* (your business, its services, its location, its reputation). Everything that strengthens your entity — consistent schema, Wikidata/Wikipedia presence, citations on authoritative sites, Google Business Profile — survives every algorithm change. Keyword tricks don't.

**Treat AI crawlers as a first-class audience.** Maintain the robots.txt AI-bot matrix (the tool already audits GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended). Standards will keep churning — llms.txt today, possibly MCP-style site endpoints or agent-facing APIs tomorrow. The `geo.py` module is the single place to add each new convention as it emerges.

**Prepare for agentic commerce.** The next wave is AI *agents* completing tasks — booking, ordering, comparing — not just answering. Sites that expose structured actions (schema.org `potentialAction`, clean APIs, machine-readable pricing/availability) will get chosen by agents. Add an "agent-readiness" check module when these standards solidify.

**Keep scoring weights external.** When AI referral traffic overtakes organic for your niche, you shift `scoring:` weights in config — no code change. That's why weights live in YAML.

**Instrument reality, not proxies.** Wire analytics to segment traffic by source (organic vs. ChatGPT/Perplexity referrals — they send identifiable referrers) so the tool optimizes for what actually generates leads, closing the loop from audit → fix → measured revenue.

**Stay white-hat, permanently.** Every major algorithm update (Panda, Penguin, Helpful Content, spam updates) wiped out sites built on manipulation. LLM providers are building similar defenses against "GEO spam." Structural quality + genuine expertise is the only strategy that compounds instead of expiring.

## 5. Honest constraints

- No tool or agency can guarantee a #1 ranking; anyone promising it is selling risk.
- Ranking movement takes 4–12 weeks for established sites, longer for new domains.
- Backlinks and reviews — two of the strongest signals — require real-world effort (PR, partnerships, great service). The tool can find opportunities and track them, but can't fabricate them.
- Scraping Google results or auto-publishing mass AI content violates platform policies and this tool intentionally does neither.

## 6. Tech stack summary

| Layer | Now | Scale-up path |
|---|---|---|
| Language | Python 3.10+ | same |
| Crawling | requests + BeautifulSoup | httpx async / Playwright for JS-rendered sites |
| Scheduling | APScheduler | Celery + Redis, or cron in containers |
| Storage | JSON files | SQLite → Postgres |
| Reports | Self-contained HTML | FastAPI + React dashboard, PDF export |
| AI | rule-based heuristics | LLM API for rewriting + citation tracking |
| Deploy | any machine / cron | Docker → Kubernetes, GitHub Actions CI |