Pipeline
How one snapshot goes from a request to done, including the job runner, leases and recovery, timeouts, per-host spacing and the browser guard.
A snapshot moves through the job runner into the pipeline, which fetches the page, opens it once in Chromium and runs all extractors in parallel. The outcome is a single status on the snapshot row.
One snapshot, request to done
In order:
- Create. The entry point inserts the snapshot as
pendingand ajob_queuerow whose heartbeat starts the job's lease.webfrost addruns the pipeline inline and waits; the server, imports and schedules hand it to the job runner and return at once (202). - Wait for a slot. The runner waits for the host's spacing slot, then for a free concurrency slot.
- Fetch. The page is fetched with the SSRF-guarded
safeFetch(redirects re-checked on every hop, body capped at 50 MB, 30 s timeout). If the fetch fails or the server answers 400 or higher, no extractor runs and the snapshot isfailed. - Browser. Chromium launches once and navigates once. The browser guard checks every request (see below). If the browser or navigation fails, the four browser extractors are recorded as failed and everything else continues.
- Extract. All extractors run in parallel, each under its own timeout. Text extractors read the cached response, browser extractors use the open page, and media runs
yt-dlpas a subprocess. The browser closes as soon as its extractors finish rather than waiting for a long media download. - Record. Every extractor writes either a result or an error row with its duration. The snapshot becomes
doneif at least one extractor produced output, otherwisefailed. Statuses are onlypending,doneandfailed. - Index.
donesnapshots are indexed for full-text search from the Readability result. An indexing failure does not fail the snapshot.
Jobs, leases and recovery
Every pipeline the server starts goes through one JobRunner (packages/core/src/jobs.ts), limited to WEBFROST_CONCURRENCY running at once (default 3). Each job opens its own database handle and closes it when it settles, so request handlers can close theirs as soon as the row is inserted. (On SQLite the handles of one process share a connection; on PostgreSQL and MySQL they borrow from the process's pool.)
job_queue.status goes pending, running, then done or failed.
Leases. A job is leased through job_queue.heartbeat_at and job_queue.lease_token. The token is issued when the job is created; the pipeline refreshes the heartbeat every 10 seconds, and the runner refreshes it for everything it has queued or running, including jobs still waiting for a slot.
Recovery. A recovery sweep adopts a snapshot that is pending and whose heartbeat is older than 45 seconds, meaning that no live process is working on it (a crash, deploy or restart). The server sweeps at startup and then every 60 seconds; webfrost schedule worker does the same. Adoption:
- is atomic, so two processes sweeping at once cannot adopt the same job: SQLite runs it in a
BEGIN IMMEDIATEtransaction, PostgreSQL and MySQL lock the job rows withSELECT … FOR UPDATE SKIP LOCKED, which also lets concurrent sweeps split the orphans between them; - issues a new lease token;
- discards partial results of the interrupted run, so a snapshot never ends with duplicate archive results;
- increments
attempts, and marks the snapshotfailedinstead of running it again after 3 attempts.
Because a live job keeps its heartbeat fresh, a server and a worker on one archive never steal each other's work. Right after a crash the dead process's jobs still look alive for up to 45 seconds, hence the periodic sweep.
Fencing. A worker can also be presumed dead while it is still running: a stalled process, a long pause. Every write a pipeline makes (its status, each archive_results row, its heartbeat) names the lease token it started with and is applied only while the job still holds that token. Once another worker has adopted the job, the old worker's writes are dropped and it stops at its next status change, so it can neither overwrite the new run's outcome nor add results to it. Fenced writes lock the job_queue row first, in the same order as recovery, so the two never deadlock.
Per-host spacing
WEBFROST_DOMAIN_DELAY_MS (default 1000, 0 disables) spaces archives of the same hostname by about that many milliseconds. Each call reserves the next free slot for its host: the first starts at once, the next about one delay later, and so on. Measured gaps can come out a few milliseconds shorter than the setting. Different hosts never wait on each other.
The runner waits for the host slot before it takes a concurrency slot. Otherwise a batch of URLs for one host would park every worker in a sleep and stall unrelated hosts behind it.
Timeouts and limits
| What | Limit | Setting |
|---|---|---|
| Page fetch, with redirects | 30 s | WEBFROST_FETCH_TIMEOUT_MS |
| Redirects per fetch | 5 | none |
| Response body | 50 MB | none |
| Browser navigation | 30 s | none |
| Each extractor | 60 s | WEBFROST_EXTRACTOR_TIMEOUT_MS |
Each yt-dlp run | 10 min | WEBFROST_MEDIA_TIMEOUT_MS |
| Media extractor as a whole | three yt-dlp runs | 3 times WEBFROST_MEDIA_TIMEOUT_MS |
| Media file | 500 MB | none |
| Browser shutdown | 5 s | none |
| yt-dlp output collection | gives up 3 s after exit | none |
When an extractor times out, it is recorded as failed and its abort signal fires, so it cannot store a result afterwards.
The browser guard
Fetching with safeFetch protects only the first request. Without more, a page could make Chromium load http://169.254.169.254/ or an intranet host in an iframe or image and get it captured in the screenshot, PDF or DOM. The guard (browser-guard.ts) applies the SSRF policy to everything the browser loads, in three layers:
- Request routing. Every request (navigation, subresources, iframes,
fetch, XHR, WebSockets) is checked and aborted if its host is not public. Service workers are blocked. - Redirect detection. Playwright cannot intercept redirect hops, so they are checked from the request events. A blocked hop cannot be stopped mid-flight, but it is reported, and the pipeline then refuses to capture the page: the four browser extractors are recorded as failed. The request itself may still have been sent (blind SSRF, an accepted risk).
- Seal. After the page has loaded and its network has gone quiet (up to
WEBFROST_SETTLE_MS, so single-page apps can finish loading), every further request is refused before anything is captured. Requests made while waiting still go through the guard. A script cannot wait out the check and then pull an internal URL into a frame. Then the browser scrolls through the page once (one screen at a time, up to 20000 px, a few seconds at most) and back to the top, and the network settles again: sections that sites reveal as they scroll into view, and images loaded only near the viewport, are archived visible.
WEBFROST_ALLOW_PRIVATE_URLS=true turns the address check off everywhere. See Security.
Download flow
How the server serves an archived file.
The extractor name is checked against an allowlist, and the stored reference must lie inside snapshots/ (filesystem) or under the configured prefix (S3) before any read. media-metadata is always streamed by the server.
Import flow
Schedules
The claim advances next_run_at before the pipeline runs, so a slow run is not started again on the next tick, and two processes on one archive cannot both run the same schedule. A run that outlasts its cron period waits until it finishes. A cron expression that no longer parses is skipped without affecting other schedules.