SEO stack audit & repair

Six rounds of verification and repair on ns3.wirenets.ca. The recurring finding was not broken software — it was a dashboard confidently reporting things nobody had checked. Half the advertised MCP servers could not start, the agent named seo-orchestrator could not see a single SEO skill, and the browser panel showed green while no browser existed on the box.

2026-07-30 → 2026-08-03 192.227.221.215 every claim probe-verified loading live data…

01Headline

Metrics are measured, not asserted. MCP servers count as healthy only if they answer a real JSON-RPC initialize handshake; skills are counted by openclaw skills check; services by live HTTP probe.

MCP verified working

21/26

was 15 of 28 advertised

Agent tools exposed

309

previously never measured

SEO skills usable

105

was 0 — all blocked by an allowlist bug

Services responding

22/22

GitNexus recovered from dead storage

Before → after

MCP servers actually working21 ← 15

SEO skills visible to the SEO agent105 ← 0

Repos genuinely installed64 ← 53

Backlog ("needs action")22 ← 43

Grey bar = before, gradient = after. The backlog bar shrinking is the good direction: 11 of those rows were reclassified because they can never be installed on a headless server.

02Live state

Pulled from status.json and mcp-health.json on load, so this section stays honest after the write-up goes stale. mcp-health.timer re-probes daily at 04:30.

MCP server health

loading…

Services

loading…

03What was actually wrong

Every one of these was reported as working. None of them were.

Component	Reported	Reality	Fix
seo-orchestrator agent	SEO agent with skills	Allowlist named directories (`seo-skills`) but OpenClaw matches skill names. 0 of ~190 SEO skills visible. A second cap loaded only 16 per source.	Rewrote allowlist against real names, raised cap 16→256. 105 skills live.
playwright-mcp	✅ MCP	`--port 8931` forced HTTP mode, so it never spoke stdio — the transport OpenClaw uses.	Dropped the flag. 24 tools.
github-mcp-server	✅ MCP	Pointed at the package's git CLI wrapper, which printed a banner (corrupting the JSON-RPC stream) and exited.	Repointed to the real MCP bin. 30 tools.
browser-use	✅ pip+MCP	Local wrapper imported `browser_use.browser.browser`, removed in v0.11.	Switched to the package's own maintained MCP server. 16 tools.
librecrawl / jina-reader / selenium-mcp	✅ installed	All three referenced npm packages that 404. One claimed clone path didn't exist either.	librecrawl rebuilt from source (37 tools), jina repointed to the real package, selenium removed.
Browser / "Hands"	✅ working, with a verified-commands table	The entire `~/.cache/ms-playwright` cache was gone. No Chromium existed. Nothing could launch.	Reinstalled Chromium 1155, repointed `executablePath`, verified a real headless launch.
Local LLM	"best local LLM, already bridged"	Ollama running with zero models. `api/tags` returned `{"models":[]}`.	Installed `llama3.2:3b` — exactly what the provider already referenced.
"0 configured in OpenClaw"	0 of 10 MCP files configured	Status script read `.mcpServers`; the real key is `.mcp.servers`. jq errored, fallback wrote `[]`.	Fixed the path and the file matcher. 10/10.

The pattern. A handshake is not a launch, and a config entry is not a running service. playwright-mcp happily listed 24 tools while no browser existed on the box, because listing tools never opens one. Every green badge on the old page was a claim about configuration, not behaviour.

04Round by round

Six passes. Round 1 deliberately moved no counters — it only made them true.

Round 1 — Verification no counters moved

Probed every claim. Found 30 wrong status badges running in both directions, which is why the totals looked plausible. Six repos showing green didn't deserve it; six greyed-out ones did. Net zero, composition corrected. Built mcp-health-check.py + a daily timer so the page re-verifies itself.

Round 2 — Install audit

Discovered a quarter of the "backlog" was never work: Chrome extensions, a C# desktop app, six reading lists. Reclassified 11 as N/A. Installed lhci; found maxun already running behind a "not installed" badge; unlocked five skill packs.

Round 3 — librecrawl

Its Docker MCP container crash-looped on a missing mcp.server.fastmcp. Split the stack: engine in Docker, MCP on the host venv that already worked. 37 tools. Also fixed a bug in my own prober that made tool counts drift between runs.

Round 4 — Real packages, not patches

Three "broken" servers weren't broken — the configs pointed at packages that don't exist while the real ones were already installed or one command away. Removed 4 registrations that were never MCP servers; the underlying CLIs stayed.

Round 5 — A false green badge incident

Installed lightpanda (which turned out to ship its own MCP server, 31 tools), and discovered no Chromium existed on the box while the panel claimed "working". Fixing browser.executablePath triggered a gateway self-restart — see the incident log.

Round 6 — Storage, LLM, recovery

Established the "1TB drive" is network storage, not a disk. Installed Agent-Reach and llama3.2:3b. Cleared a hung mount, recovered LXD from a stalled snap refresh, and brought GitNexus back from dead storage.

05Installed & verified

Nothing here is marked done on the strength of a package manager exit code — each was run.

What	Verified by	Result
librecrawl (Docker engine + host MCP)	JSON-RPC handshake over HTTP	37 tools
lightpanda browser + MCP	Fetched a live page, then handshook	31 tools
llama3.2:3b (Ollama)	Real completion request	2.0 s warm / 18.3 s cold
Agent-Reach v1.5.0	CLI + `doctor` + MCP handshake	4/15 channels (rest need logins)
Chromium (Playwright 1155)	`pw.chromium.launch()` against example.com	launched, title read
@lhci/cli 0.15.1	`lhci --version` + healthcheck	on PATH
gh CLI 2.97	`gh --version`	needs `gh auth login`
seo-analyzer	Live run against example.com	scored 62.8%, wrote JSON
GitNexus	HTTP probe after remount + restart	:4173 → 200

06The storage question

There is no 1TB local drive. lsblk shows a single 100 GB vda. What's mounted is network storage, and the distinction decides what can safely live there.

Mount	Size	Measured	Safe for
`/mnt/vps162` sshfs → another VPS	983 G	19 MB/s write 6 MB/s read	Backups, archives, cold bulk. Not Docker, DBs, or anything latency-sensitive.
`/mnt/ihostart-5365` rclone remote	2.1 T	symlinks fail (I/O error)	Flat files only. No venvs, no `node_modules`, no Docker.
`/mnt/i7-ns3-offload` sshfs → i7 box	229 G	100% full 462 MB free	Backs GitNexus. Was hung; cleared and remounted, but it is full.

Judgement call. Asked to offload cold data to the drive, I started — then stopped. Both candidates were clean git clones, and 1.5 GB of the 1.6 GB was node_modules: ~94,000 regenerable files that would take ~30 minutes to crawl over a 19 MB/s mount and buy nothing, because the git origin is the backup. Wrote a restore manifest (origin + exact commit + npm install) and deleted the local copies instead. Same space freed, seconds instead of half an hour, fully restorable.

# restore either offloaded repo from the manifest
cat /mnt/vps162/ns3-offload-20260801/RESTORE-MANIFEST.md

07Incident log

Including the ones I caused. A report that only lists wins isn't much use as an operational record.

Gateway restart — caused by my change. The brief said never restart the OpenClaw gateway, and it held at NRestarts=0 for four rounds. Fixing browser.executablePath took it to 1. I ran no restart command; OpenClaw watches its config and self-restarts on specific keys:

[reload] config change requires gateway restart (browser.executablePath)
[gateway] received SIGUSR1; restarting

mcp.servers is not restart-triggering, which is why a dozen earlier MCP edits caused no bounce. I should have checked before touching that key. Impact: ~19 s, clean recovery, 9 plugins reloaded, no OOM.

LXD down — not my change. lxc began failing with "cannot create temporary directory". I attributed it to a hung mount; the journal proved me wrong — an automatic snap refresh stopped LXD at 18:13 and never restarted it. Fixed by restarting snapd. All sites stayed at HTTP 200 throughout, because containers keep running independently of the daemon.

Hung mount & D-state. Probing /mnt/i7-ns3-offload (100% full) produced two unkillable D-state processes — the failure mode that once hung this box for 18 days. A lazy unmount cleared them; the mount was later restored and GitNexus came back with it.

My own tooling bugs. Three worth recording: a pkill pattern matched its own script and killed it mid-run; my "safe to move" check cleared photoprism and open-webui as unused when both were actively serving (caught before anything moved); and my first MCP prober skipped the notifications/initialized step the spec requires, making tool counts drift 262→232 between runs. The 262 was an undercount, not a peak.

08What's left, and why

22 items. None can be closed by more effort alone — each sits behind one of four walls.

Paid key or licence — 9

brightdata-mcp — API_TOKEN free tier
agentql-mcp — AGENTQL_API_KEY free tier
gsc-mcp — Google service-account JSON
serpapi-mcp — key and a stdio→HTTP rewire
cloakbrowser, browsermcp — vendor keys
screaming-frog, seranking, seotoolsuite — paid plans

Keys belong in the systemd drop-in env, not just openclaw.json — a key in the wrong place passes a curl test while every runtime call 401s.

Your decision — 3

claude-ultimate-seo and Agentic-SEO-Skill are loaded but contribute 0 unique skills — every name collides with a pack that wins. Unlocking them adds duplicates, not capability.
fairchoice-seo-agent is an application sitting in the skills directory, not a skill. I didn't invent a SKILL.md for it.

Broken upstream — 2

seo-drift-monitor — every dependency satisfied, but check.py imports parse_html and that file exists nowhere in the repo.
rendora — stale config only; no binary, no image.

Disk / duplicate — 8

Root is at 88% with ~390 KB reclaimable — the ~17 GB of images are all live. That blocks matomo.
The rest are duplicate browser stacks. Six already work (lightpanda 31, playwright-mcp 24, mcp-browser-use 23, browser-use 16, browser-tools 14, Skyvern); spending the last of the disk on a seventh is a bad trade.

Cheapest way to move the number: add any one free-tier key (BrightData, AgentQL, or Jina search), or run gh auth login to unlock Agent-Reach's GitHub channel. Each converts a red row to green with no further engineering.

09API surface — can an agent actually call these?

Audited 2026-08-01. "Responding on /" and "usable by an agent" are different questions: every engine below returned HTTP 200 on its homepage, yet only some could be driven programmatically. Each row was tested against its real API endpoint, not its landing page.

Callable by an agent

2 of them fixed in this pass

Needs a console action

token must be minted in the UI

No public API

web UI only, by design

Auth headers differ

Bearer vs x-api-key vs none

Engine	Endpoint	Auth	Verified result
SearXNG :8888	`/search?q=…&format=json`	none	Fixed. Was 403 — JSON output is off by default and this instance never set `search.formats`. Enabled it; now returns 20 results.
SerpBear :3000	`/api/domains`, `/api/keywords`	`Authorization: Bearer`	Fixed. Its `APIKEY` env was empty (length 0), so every call 401'd. Generated a 40-char key into `/opt/serpbear/.env` (mode 600) and recreated the container; data volume intact. Note `x-api-key` is not accepted — Bearer only.
OpenSERP :7000	`/{engine}/search?text=…`	none	bing (4), duckduckgo (6), yandex (2) all return results. Google returns 429 `captcha_detected` — use the other three.
ChangeDetection :5001	`/api/v1/watch`	`x-api-key`	200 with the token from its datastore; returns live watches.
Skyvern :8000	`/api/v1/tasks`	`x-api-key` (JWT)	200. Token lives in the Postgres `organization_auth_tokens` table, not an env var.
Google Maps Scraper :8085	`/api/v1/jobs`	none	200, open. Ready for local-SEO lead pulls.
LibreCrawl :5081	MCP (HTTP)	none (loopback)	37 tools via handshake; engine on :5080.
Plausible :8086	`/api/v1/stats/*`	Bearer token	401 — "Missing API key". The token must be minted in the Plausible UI; it cannot be generated from the shell.
Umami :8087	`/api/heartbeat` / `/api/auth/*`	session token	Heartbeat 200, but the data API needs a login-issued token.
Maxun :3005	`/auth/current-user`	session cookie	401. Note the path is `/auth/…`, not `/api/auth/…` as commonly documented.
SEOnaut :9000	—	—	No public REST API. Sign-in web UI only — drive it with a browser MCP if needed.
OpenSEO :3004	—	—	No REST surface found; every probed path 404s. App-internal routing only.

Caveat on SerpBear. Its API answers now, but settings.json has scraper_type: "none" — so it still cannot fetch rankings until a scraper is configured. OpenSERP is already running locally on :7000 and works on bing/duckduckgo/yandex, which makes it the obvious no-cost backend to point it at.

Working calls, copy-paste.

curl "http://127.0.0.1:8888/search?q=technical+seo&format=json"
curl "http://127.0.0.1:7000/duckduckgo/search?text=technical+seo&lang=EN&limit=5"
curl -H "Authorization: Bearer $(grep ^APIKEY= /opt/serpbear/.env | cut -d= -f2)" \
     http://127.0.0.1:3000/api/domains
curl http://127.0.0.1:8085/api/v1/jobs

10The LLM bridge

Every app on this box pinned its own provider. When a key expired or a free model was retired upstream, the app simply broke — silently. One endpoint now fronts five free/local upstreams and fails over automatically.

What was actually broken. An audit of all 15 configured providers found 1 answering. The SEO agent's primary returned 403 and all three fallbacks were dead: an OpenRouter model retired upstream, a Zhipu token expired (令牌已过期), Google AI Studio quota-exhausted (429 on all 50 models). Separately, FairChoice's scheduled agents were pinned to 127.0.0.1:11436 — an SSH tunnel to a Windows box that is only up when that machine dials in. It was down, which is why "DeepSeek/Evo strategy failed" reappeared on every single run.

Bridge health

loading…

http://127.0.0.1:9099/v1 · OpenAI-compatible · stdlib only, ~10 MB RSS · MemoryMax=256M

How failover behaves

Upstreams are tried in priority order; cloud first (fast, large context), local Ollama last.
A 401/429/5xx demotes that upstream for a 300 s cooldown and the request rolls to the next one.
The local model needs no key and cannot rate-limit, so the chain always terminates somewhere that works.
Responses carry a _bridge field naming which upstream served and what it rolled past — so a silent downgrade is impossible.

Failover proven, not assumed. With all four cloud upstreams deliberately sabotaged (invalid key), the bridge tried each, recorded HTTP 401 Missing Authentication header against every one, and served the answer from local Ollama. Restoring the key recovered gemma-4-26b in 1,040 ms.

What now runs on free inference

Consumer	Was	Now	Verified
FairChoice scheduled agents	`:11436` dead tunnel, via `/etc/fairchoice-seo-local.env`	bridge `:9099`	`evo-strategy` + `daily-technical` both `Result=success`, `deepseek_configured: true`
FairChoice admin (:5000 / :18880)	same dead tunnel	bridge `:9099`	both serving
seo-orchestrator agent	`atlascloud` 403 + 3 dead fallbacks	gemma-4-26b → 2 nemotrons → local → bridge	chain verified live
Any other client	—	point at `http://127.0.0.1:9099/v1`	OpenAI-compatible, 5 models listed

The trap that made this hard to find. The scheduled agents are templated units (fairchoice-seo-agent@<job>.service). A foo.service.d/ drop-in does not apply to foo@.service instances — and the template loaded /etc/fairchoice-seo-local.env last, so it silently overrode everything else. Patching the obvious drop-ins fixed nothing until that file was found.

Still blocked on you

GROQ, CEREBRAS, NVIDIA, MISTRAL keys are the literal placeholder none-configured. Groq and Cerebras have excellent free tiers and are ~2-minute signups — each would add fast cloud inference at zero RAM cost.
ZHIPU token expired; DEEPSEEK key rejected; Google AI Studio is valid but quota-exhausted (resets daily — worth retrying).
OpenRouter free tier is 50 requests/day account-wide. Fine interactively, not for bulk audits — that is what the local backstop is for.

11Can the SEO skills actually run?

"105 skills visible" is not the same as "105 skills that can do work". Each SKILL.md was parsed for its declared API dependencies and bucketed against what this box can actually reach.

Run on the free stack

no key needed at all

Unblocked by GSC

credential found & verified live

Need a paid API

DataForSEO / Ahrefs / Semrush

GSC tools authenticated

search-console-mcp

The find that mattered. Google Search Console credentials were blocking 54 skills — more than any other single dependency. Rather than ask for new credentials, I traced how FairChoice was already pulling 19,893 GSC rows and found a working OAuth ADC credential at /data/fairchoice-seo-agent/google/. Verified it live: token refreshed, one property (sc-domain:fairchoice.ca, siteOwner), and a real query returned rows. Wired it into search-console-mcp, which now reports [✔ Google | ✘ GA4 | ✘ Bing] and returns real data through MCP. The key you needed was already on the box.

What each dependency blocks

Dependency	Skills blocked	Status
Google Search Console	54	Unblocked — existing credential verified & wired
DataForSEO	46	paid plan, no free tier
Ahrefs	42	paid
Moz	10	paid
Semrush	9	paid
GA4	8	same Google account could authorise it
Screaming Frog	5	licence

Worth knowing before spending money. The three paid APIs that block the most skills (DataForSEO, Ahrefs, Semrush) overlap heavily in what they provide: keyword volume, backlinks, competitor data. You would not need all three — one would unblock most of the 82. And the free stack already covers crawling (crawlie, LibreCrawl 37 tools), search (SearXNG, OpenSERP on 3 engines), performance (Lighthouse) and now Search Console.

Heads-up from the tool itself: search-console-mcp reports v2.0.0 is available (running v1.14.2). I did not upgrade it — a major version bump on a working integration is your call, not something to do unattended.

12Two servers, one picture

192 runs the workload; 162 holds the archive and mirrors the reporting. The split is dictated by hardware, not preference — and getting that backwards would have broken things.

The constraint that decides everything. 162 has 851 GB free but only 1 CPU core and 1.6 GB RAM, and no Docker. 192 has 11 GB RAM but was at 91% disk. So the rule is move bytes, not workloads: cold data goes to 162, anything needing memory stays on 192. Moving the 21 containers or the LLM stack there would simply fail.

Surface	Host	Purpose
fairchoice.ca/seo-admin/	192	Production console — 20,207 GSC rows, live agents, Action Plan page
:8613/seo-repos/	192	Tool inventory + live MCP health panel
:8614	192	Gateway Meter — which providers authenticate vs need keys or credit
:8619	192	Command Hub — 47 fleet links, corrected 2026-08-02
`brain…sslip.io/seo-infrastructure.html`	162	Live infrastructure view + reference index, LLM capability, business levers
`/srv/gateway/seo-reports/fairchoice/latest.json`	162	Report mirror pushed after every run — what the brain page reads
`/srv/from-ns3/`	162	Migrated cold data (943 MB) + restore manifest

Disk recovered on 192

Action	Freed	Note
Obsolete `media-mdla` snapshot	2.6 GB	Pre-SEO-install rollback point; restoring it would have discarded 11 days of work
Orphaned puppeteer Chromium	563 MB	Confirmed with `lsof`: no processes, no open files
npm + pip caches	317 MB	Regenerable
Migrated to 162	943 MB	Checksum-verified before local deletion; restore commands left in place
Heartbeat log gzipped	43 MB	44 MB → 527 KB

91% → 86% — free space 9.2 GB → 14 GB.

Kept deliberately, after checking properly. /opt/photoprism (4 open files, service active) and /root/vpndash (97 open files, 1 systemd unit) both looked cold to a naive check and would have broken two running services. ms-playwright (869 MB) was also kept — it holds the Chromium executablePath points at, with 8 live references. A grep-based check gave the wrong answer four separate times in this engagement; lsof gave the right one every time.

13What still has to happen

The plumbing is finished and verified. None of it raises a ranking on its own. These four do — and none has been started.

Site CTR

0.11%

industry norm 2–5%

Content published

0/17

all written, all in review

Impressions on the weak host

39,355

www ranks 25.0 vs non-www 8.7

Clicks, last 7 days

on 1,230 impressions

#	Lever	Evidence	Type of work
1	Consolidate www / non-www	Both hostnames indexed. non-www avg position 8.7; www avg 25.0 carrying 39,355 impressions. Also the source of all 6 duplicate-title issues.	301 + canonical — config, not content. No compliance exposure.
2	Publish the 17 written pages	Includes `mortgage-refinance`, while that cluster shows 68 keywords / 12,392 impressions / 0 clicks at position 78.	Publishes copy to a live regulated site — needs explicit sign-off.
3	Striking distance	12 queries already at position 11–20, led by the business-loan-broker cluster (402 impressions at 15.7).	Internal links + on-page. No new content.
4	Fix titles on zero-click pages	`/pages/reviews`: position 3.7, 4,229 impressions, 0 clicks. That is a snippet problem, not a ranking one.	Metadata only.

Recommended order: 1, then 4, then 3, then 2. The first three are configuration and metadata — reversible, no page copy touched, and #1 clears six open issues as a side effect. #2 is the biggest single opportunity but publishes copy to a live mortgage brokerage, so it is a decision rather than a task.

14Runbook

How to re-verify any claim on this page, and where the safety nets are.

Re-run the verification yourself

# probe every MCP server with a real handshake, rewrite mcp-health.json
systemctl start mcp-health.service && journalctl -u mcp-health -n 5 --no-pager

# what the SEO agent can actually see
openclaw skills check --agent seo-orchestrator

# local LLM
curl -s http://127.0.0.1:11434/api/tags | python3 -m json.tool

Safety nets

Item	Location
Every `openclaw.json` edit (9 generations)	`/root/_repair_backups_seo/`
Dashboard HTML, per round	`media-mdla:/root/_repair_backups_20260730_seo/`
Offloaded repos (origin + commit)	`/mnt/vps162/ns3-offload-20260801/RESTORE-MANIFEST.md`
Daily re-verification	`mcp-health.timer` — 04:30, `Nice=10`

Two traps worth remembering on this box. Editing browser.executablePath (or other watched keys) self-restarts the gateway — mcp.servers does not. And touching a dead sshfs mount creates unkillable D-state processes; clear them with umount -l, never kill -9.