Free Audit Integrationsubmit → confirm → watch it run → report

3.2Audit engine — free-audit

The Python audit pipeline. Already built and running; v1 work on it is purely additive.

Data model — core/models.py

Entities: Audit, AuditKeyword, AuditResult, AuditPSI, AuditCrawlPage, DailySpend, AuditAccessToken.

Incremental commits — what is observable

The operator dashboard (/audits/{id}) and public /start/{token} flow already render a live run — web/templates/audit.html + HTMX's SSE extension subscribe to GET …/stream, which replays a Redis log stream (audit:{id}:log) and emits event: done. v1 does not consume that SSE stream (cross-origin, operator-authed) — it polls the JSON API (§10.2, §12.2).

Location plumbing — already end-to-end

Both POST /audits and POST /start/{token} accept location / physical_locations_count via _parse_audit_fields (web/routes.py ~54–75): count < 0 → 400; count > 0 requires a non-empty location → else 400; count == 0 forces location to NULL. When count > 0 and location is set, worker/jobs.py _localize (~174–184) appends " in {location}" to every extracted keyword before storage and ranking. No free-audit schema or pipeline change is needed for v1 location support.

Config & deploy assets

config

core/config.py: ENGINES = ("gemini",) (line 6); Settings (Pydantic BaseSettings); MAX_KEYWORDS_PER_AUDIT = 15; assert_production_safe() today checks only APP_PASSWORD and SESSION_SECRET (§15 extends it).

reports

web/reports.py: summarize(), crawl_highlights(), group_by_keyword(). No PSI helper.

migrations

A healed branch (0004_audit_psi and 0004_locations_and_tokens); the current single head is 0007_merge_heads.

nginx

deploy/nginx/free-audit.conf exists but is pinned to a different hostname/port and to nginx zones defined elsewhere — a template, not a drop-in.

Load-bearing traps

3.3Audit VPS — the free-audit host

Read-only inspection, verified 2026-05-18. free-audit is already deployed and running — Stage 0 (§16) is not a fresh deploy; it is DNS + nginx + TLS for the new hostname, plus the §11.2 env vars.

connection

SSH config alias myvps → host 149.50.146.215, port 5854, user root, key ~/.ssh/id_ed25519 (path only — the private key never enters this repo or git).

OS / resources

Ubuntu 24.04.4 LTS; 2 vCPU; 7.8 GiB RAM (~5 GiB free, no swap); 25 GB disk. Docker 29.1.2 + Compose plugin v5 (docker compose; no standalone docker-compose).

free-audit state

The 5-service stack (web, worker, browser, postgres, redis) up 10–11 days. web published on 127.0.0.1:8004; /healthz returns {"status":"ok"}.

current URL

https://vps-4849885-x.dattaweb.com:3007 — the nginx free-audit block listens on :3007 for the VPS's own hostname. nginx 1.24 running with :80/:443 free for a new server block.

worker concurrency

VPS .env: MAX_KEYWORDS_PER_AUDIT=15; KEYWORD_PARALLELISM=5 (asyncio semaphore — up to 5 keyword lookups concurrent within an audit, why a run finishes in ~1 min); MAX_CONCURRENT_AUDITS_PER_WORKER=4 (arq max_jobs). One worker container; no Docker-level horizontal scaling.

12.2 · Progress status endpoint

GET /wp-json/demandnow/v1/audit/status/{verifyToken} — the live audit page's browser script polls this. permission_callback => __return_true; the unguessable verify token is the credential (no nonce — §12.1).

1. Look up the lead by _dn_verify_token
   Unknown → 404, no free-audit call (the browser treats 404 as terminal "link invalid" and stops polling). A generous, shared-NAT-sized per-IP rate limit applies — a ~1-min audit at one poll / 5 s ≈ 12–15 polls and several visitors can share an IP; cap e.g. 600 / 15 min, transient demandnow_auditpoll_{md5(ip)}. The browser treats 429 as "back off and retry", never as terminal.
2. Terminal short-circuit
   If _dn_status is already complete/viewed/failed, answer from post meta with no free-audit call — {state:"complete"|"failed", done:true}. This drops all polling load for the tail of every audit.
3. No _dn_audit_token yet
   Return {state:"pending", percent:5, label:"Starting your audit…"} — no free-audit call.
4. Otherwise fetch free-audit — guarded two ways
   Fetch GET {dn_audit_api_url}/api/audits/{_dn_audit_token} with X-API-Key. Short transient cache: store the raw response in dn_audit_poll_{token} for ~10 s (TTL ≥ 2× the poll interval, so a lone poller almost always hits cache; TTL is best-effort — object-cache.php may evict early). Single-flight lock: before a cache-miss fetch, wp_cache_add('dn_audit_polllock_{token}', 1, '', 15) (atomic add); if it fails another request is already fetching — return the last-known transient (or a holding running frame) rather than a parallel call. Collapses concurrent same-token misses (multiple tabs) to one fetch.
5. On a terminal free-audit status, call §12.6
   dn_audit_apply_completion() for completed; dn_audit_apply_failure() for failed/halted_budget_exceeded/cancelled. Prime the §12.4 report transient only from a fully-completed payload — audit.status=="completed" and summary/results present and progress.results_completed >= progress.total_pairs. If the terminal frame came from the ~10 s poll cache and might predate full result population, do one fresh uncached GET /api/audits/{token} before priming, or skip priming. Never prime the 1 h transient with a non-completed snapshot (§12.4 invariant).
6. Respond with a trimmed, whitelisted payload
   Never raw findings, never cost/error detail — only the five fields below.

12.2 — browser-facing status framecopy
{ "state": "running",          // pending | running | complete | failed
  "stage": "ranking",          // scraping | extracting | ranking | finalizing | null
  "percent": 62,               // 0–100, coarse, advisory
  "label": "Seeing how Gemini ranks you — 8 of 15 queries…",
  "done": false }             // true ⇒ the script reloads the page

state / stage / percent mapping

Computed WordPress-side from §10.2's audit.status + audit.stage + progress. The mapping must be total over every (status, stage) pair free-audit can produce. Tunable without an API change.

12.3 · Webhook receiver

POST /wp-json/demandnow/v1/audit/webhook. permission_callback => __return_true; authentication is the HMAC. The handler holds no business logic — it verifies, finds the lead, and hands off to §12.6.

1. Verify the signature
   Reconstruct the §10.3 base_string from the parsed JSON params — audit_id via strict integer cast, absent fields → "". Strip a leading sha256= (case-insensitive) from X-DN-Signature. hash_equals(computed_hex, received_hex). Mismatch → 401, log.
2. Freshness window
   Reject (401, log) if finished_at is more than ~1 hour old.
3. Find the lead
   By _dn_audit_token (= payload token); else fall back to callback_ref (lead_{id}) — safe because callback_ref is signed (§10.3). Not found → 200 (ack, log).
4. Apply via the shared §12.6 routine
   audit.completed → dn_audit_apply_completion(); audit.failed → dn_audit_apply_failure(). §12.6 owns the callback_ref backfill, the failed → complete supersession, the never-downgrade rule, and the idempotent email guards — the webhook handler holds none of that logic.
5. Always 200 {ok:true} once authenticated
   So free-audit stops retrying. A duplicate webhook for an already-terminal lead resolves to a logged no-op inside §12.6.

12.4 · Report page

GET /audit-report/{token} — the durable, emailable, return-to-it copy of the report. Keyed by the free-audit token (which never expires), independent of the verify token. Routing: add_rewrite_rule('^audit-report/([^/]+)/?$', …).

1. Look up the lead by _dn_audit_token
   Unknown → themed 404.
2. Fetch the audit
   GET {api_url}/api/audits/{token} with X-API-Key (short timeout, redirection => 0 — rule 5). Inspect audit.status per the invariant above. On a fetch timeout/error, fall back to _dn_report_cache; if that too is empty, render "still generating" + auto-refresh — never block the page on a hung free-audit.
3. Non-terminal status
   "Still generating" + light auto-refresh (rare here — a durable link is normally opened post-completion).
4. completed → render the hybrid report
   Render §13.4–13.5 via the shared report renderer, which consumes a whitelisted projection of the §10.2 JSON — no total_cost_cents, no audit.error (§10.2 invariant). The same renderer serves the §12.1 complete branch. On first successful render, backfill _dn_completed_at if somehow empty (§12.6), then set status viewed, _dn_viewed_at=now.
5. failed/halted/cancelled
   Apology state + "book a call" CTA — generic copy, no audit.error.

12.5 · Reconciliation — real cron, not WP-Cron

A scheduled job every ~5 min — the authoritative backstop for dropped webhooks, closed live pages, and the reconcile_orphans worker-restart path.

What the job does each run

12.6 · The shared completion & failure routines

Lead completion is reachable from three independent channels — the webhook (§12.3), the live progress poll (§12.2), and the reconciliation cron (§12.5). All three call one of two shared functions. This subsection is the canonical home of the idempotency story.

12.6 — dn_audit_try_lock: atomic test-and-setcopy
-- Returns true IFF a row was inserted. A duplicate-key failure
-- (the unique index on option_name) ⇒ false. No seeded row needed.
INSERT INTO wp_options (option_name, option_value, autoload)
VALUES (%s, '1', 'no');

dn_audit_apply_completion($lead, $payload)

1. Backfill
   If matched via callback_ref and missing _dn_audit_token / _dn_audit_id, persist them from the payload (else the report link 404s). After backfill, the report path sanity-checks audit.callback_ref == "lead_{post_id}" and 404+alerts on mismatch.
2. Status transition — timestamp before status
   Write _dn_completed_at first (if unset), then the conditional UPDATE _dn_status … WHERE meta_value IN ('running','failed') → 'complete' (idempotent; covers the §6 failed→complete supersession; never touches complete/viewed); wp_cache_delete after. Timestamp-before-status guarantees any reader that sees complete also sees _dn_completed_at — closing the §9.2 dedupe hole.
3. Email dispatch — deferred, not inline
   Never send SMTP synchronously inside a webhook or a 5 s browser poll. Schedule dn_audit_dispatch_emails($post_id,'complete') via wp_schedule_single_event, driven promptly by the §12.5 system cron.

dn_audit_apply_failure($lead, $payload)

1. Status transition — never downgrade
   Conditional UPDATE _dn_status … WHERE meta_value='running' → 'failed' + wp_cache_delete. Never downgrades complete/viewed/failed — a stale audit.failed for those produces rows_affected=0, a logged no-op (§6).
2. Email dispatch — deferred and cross-gated
   Schedule dn_audit_dispatch_emails($post_id,'failed'). Before sending, the dispatch job re-reads _dn_status cache-bypassing and aborts if the lead is complete/viewed — the cross-gate that stops a lead receiving both a soft-failure email and a results email when a late audit.completed supersedes a failed (the §6 worker-restart re-queue case). The deferral (a few minutes, via the cron) gives a fast supersession time to win. Same lock/marker mechanics, on _dn_failure_notified.

completion email

Deferred; sent at-least-once — never zero in steady state, thanks to the §12.5 cron re-drive.

failure email

Deferred and cross-gated on _dn_status; a superseded failure email is suppressed.

terminal status

Correct for every interleaving of the three channels — the conditional UPDATEs settle on the right value regardless of arrival order.

10.2GET /api/audits/{token} — results & live progress

Called server-to-server by both the WordPress report page (§12.4) and the WordPress progress endpoint (§12.2). Must answer both "is it done, render it" and "how far along is it."

Response 200 shapecopy
{
  "audit": {
    "id": 123, "url": "https://acme.com",
    "brand_detected": "Acme", "category_detected": "project management software",
    "status": "running", "stage": "ranking",
    "location": "Seattle", "physical_locations_count": 3,
    "started_at": "2026-05-17T17:59:02Z", "finished_at": null
  },
  "progress": {
    "stage": "ranking",
    "keywords_total": 15,
    "results_completed": 8,
    "total_pairs": 15
  },
  "keywords": ["project management tool in Seattle", "…"],
  "engines": ["gemini"],
  "summary": {
    "total_keywords": 15,
    "engines": [{
      "engine": "gemini", "total": 15, "mentioned": 4,
      "errors": 0, "avg_rank": 6.25, "mention_rate": 0.27
    }]
  },
  "results": [{
    "keyword": "project management tool in Seattle",
    "engine": "gemini", "mentioned": true,
    "rank": 4, "top_brand": "Asana", "context": "…",
    "top10": [
      {"rank": 1, "brand": "Asana"},
      {"rank": 2, "brand": "Notion"}
    ]
  }],
  "psi": {
    "strategy": "mobile", "performance_score": 72.0,
    "accessibility_score": 95.0, "best_practices_score": 88.0,
    "seo_score": 91.0, "lcp": 3200.0, "cls": 0.04,
    "inp": 180.0, "fcp": 1800.0, "ttfb": 600.0, "error": null
  },
  "crawl": { "pages_total": 15, "pages_ok": 14, "errors_4xx": 1, "…": "see CrawlHighlights" }
}

Live-progress contract — audit.stage + progress block

10.3Webhook — free-audit → WordPress on completion

free-audit POSTs to the per-audit webhook_url (falling back to env RESULT_WEBHOOK_URL if absent). Required even though the live page polls — it completes the lead and sends the email when the visitor has closed the tab, and it is the prompt team notification.

Webhook POST shapecopy
POST <webhook_url>
Content-Type: application/json
X-DN-Signature: sha256=<hex HMAC-SHA256(base_string, WEBHOOK_SECRET)>

{
  "event": "audit.completed",    // | "audit.failed"
  "token": "xJ3k7Q…",
  "audit_id": 123,
  "status": "completed",
  "callback_ref": "lead_8842",
  "finished_at": "2026-05-17T18:04:11Z"
}

Canonical serialization — both repos must produce byte-identical values

HMAC base string

base_string construction — fixed order, newline-joinedcopy
base_string = event + "\n" + token + "\n" + audit_id + "\n" + status + "\n" + callback_ref + "\n" + finished_at

Event mapping

Dispatch rules — core/webhook.py

1. Single dispatch point
   Refactor run_audit: collapse its five return statements into one result dict, return result once. Place the dispatch block after the inner try/except ladder, still inside the outer try, before finally: redis.aclose(). Skip dispatch when result["status"] == "missing".
2. Re-validate the effective URL
   Re-check webhook_url (or the RESULT_WEBHOOK_URL fallback) against WEBHOOK_URL_ALLOWLIST immediately before POSTing — the env fallback is otherwise unchecked.
3. Re-SELECT the audit row
   Open a fresh session; re-SELECT the audit. If its status is non-terminal, skip dispatch. Read finished_at here (avoids the SQL-expression trap above).
4. Token lookup
   Audit has no ORM relationship to AuditAccessToken. Use: select(AuditAccessToken).where(audit_id == …).order_by(id).limit(1). If no token row exists, skip the webhook — that audit was operator-initiated.
5. POST with httpx + tenacity
   Build a fresh httpx.AsyncClient(follow_redirects=False) — follow_redirects=False prevents an allowlisted host redirecting to an internal IP (classic SSRF pivot). Retry ≈3 times with backoff via tenacity (both are already in pyproject.toml). A permanently failing webhook is logged, never raised.