CrawlerClaimerAgentPhase 1 design prototype · mock data

Browser FleetsThat Never Collide

Designed to coordinate 70+ Playwright workers, crawlers and a local AI agent from one control plane — with database-enforced single claims and latency you can audit stage by stage.

Open the console
Duplicate confirmations
0
Workers in the test fleet
0+
Latency stages traced
0
Isolated cells in the demo
0

Design targets. Measured results publish after the Phase 5 load test.

The failure everyone ships

Fifty workers watch the same board. A load appears. Twelve of them click Book in the same second. Without one source of truth, you just booked the same load twelve times — and the client finds out from the broker.

Claim engine

Many vessels. Exactly one lands.

  1. DETECTED: A watcher sees the load

    Watchers read the board's JSON responses, not the DOM, and publish a DETECTED event with a seen_at timestamp.

  2. QUEUED: Every matching actor races

    Filters route the job to pre-warmed actors whose sessions are already logged in. Redis gives a fast first filter — never the final word.

  3. LEASED: Postgres picks one winner

    INSERT … ON CONFLICT DO NOTHING on (tenant_id, target_job_key). The database's unique key decides. Everyone else gets nothing to act on.

  4. ACTING: A fencing token guards the click

    The lease carries a monotonic token. A worker that paused past its lease is rejected by the database clock, not its own.

  5. CONFIRMED: Confirmed — or reconciled, never retried blind

    A crash mid-action becomes UNKNOWN. The reconciler checks the board's bookings before it confirms or releases the job.

A watcher sees the load

Watchers read the board's JSON responses, not the DOM, and publish a DETECTED event with a seen_at timestamp.

One core, three worker types

Built for the three jobs clients actually post.

What each worker type is designed to do. The panels are illustrative mock data; measured results publish after the Phase 5 load test.

Post 2 · Distributed Playwright

Claim worker

Watchers, actors and a coordinator that scale from 50 to 70+ workers without duplicate bookings. Sessions stay warm; OTP refresh is automatic.

  • Exactly-one claims enforced in Postgres
  • Warm logged-in pages per account
  • Crash recovery, verified by chaos tests (Phase 5)

Claims · last 60 s

Illustrative · mock data

jobstateworkeract→ok
LD-48213CONFIRMEDw-17212 ms
LD-48214LEASEDw-04—
LD-48209UNKNOWNw-31—
LD-48207CONFIRMEDw-22198 ms

0 duplicate confirmations · audit clean

Post 1 · Proxy-aware crawling

Crawler

Crawl jobs defined as data, per-domain politeness, retries with jitter, and alarms that catch the silent layout change before the client does.

  • Proxy and captcha adapters per client
  • Zero-result and sudden-drop alarms
  • SSRF egress guard on every fetch (planned)

catalogue-crawl · items per cycle

Illustrative · mock data

Zero-result anomaly — selector may have changed · Trace saved

Post 3 · Local AI agent

Fleetwright Agent

A lightweight local app that drives your own Chrome across tabs. It drafts, researches and fills — and asks before it publishes, sends or buys.

  • Runs on Ollama by default
  • Human approval for risky actions
  • Page text treated as untrusted data

Agent · task 3 of 5

Illustrative · mock data

  1. Opened 3 tabs: brief, drafts, calendar
  2. Drafted post from the product brief
  3. Labelled as AI-generated

Publish to the Demo Studio page?

Latency you can audit

Six timestamps. No averaged-away excuses.

Every claim is stamped at each stage. Reports split publish→detect from detect→action, with p50 / p95 / p99 per stage.

  1. 01

    published_at

    Published

    Ground truth from the board's own event log.

  2. 02

    seen_at

    Seen

    A watcher's response listener fires.

  3. 03

    queued_at

    Queued

    Filters matched; the job enters the cell's stream.

  4. 04

    leased_at

    Leased

    Postgres grants the single lease and its fencing token.

  5. 05

    act_sent_at

    Act sent

    The warm page clicks, or the session's HTTP action fires.

  6. 06

    confirmed_at

    Confirmed

    The board confirms the booking; the audit log closes the loop.

Cell architecture

When one cell fails, the fleet keeps claiming.

A cell is a Redis, its watchers and its worker pool. The coordinator routes each tenant to a cell, so adding capacity is a config change — and a failing cell takes nothing else down with it.

Cell A is restored and claiming again.

Coordinator

Cell A

claiming
t-acmet-north

Cell B

claiming
t-delta

Guardrails

Strict by default.

Design commitments from the build plan. Each ships with automated tests in its phase — none is claimed as done until those tests pass.

  • 01

    MFA for every operator

    TOTP enforced; argon2id hashing; RBAC checked server-side on every endpoint.

  • 02

    Tenant isolation in the database

    tenant_id on every row with Postgres Row-Level Security.

  • 03

    Append-only audit log

    Every mutation and every claim action is recorded.

  • 04

    No third-party captcha defeat

    Solver services are added per client, after legal review and written authorisation.

  • 05

    US + EU law register

    CFAA, GDPR, AI Act Art. 50 and the CRA mapped to design rules — reviewed before any real target.

  • 06

    Secrets never logged

    OTP codes and cookies are redacted by a logging filter, with tests that fail if one ever appears in a log.

Walk The FleetBefore It Exists

The console prototype runs on mock data generated from the same contract the backend will implement.

Open the console