Skip to Content
WebFrostdocs
Architecture

Pipeline

How one snapshot goes from a request to done, including the job runner, leases and recovery, timeouts, per-host spacing and the browser guard.

A snapshot moves through the job runner into the pipeline, which fetches the page, opens it once in Chromium and runs all extractors in parallel. The outcome is a single status on the snapshot row.

One snapshot, request to done

Sequence of one snapshot: the entry point inserts a pending snapshot and its job, the JobRunner starts the pipeline, which fetches the page once through safeFetch, loads it once in a guarded Chromium, runs the text, browser and media extractors in parallel, writes files to storage and results to the database, closes the browser and sets the status to done or failed.
One snapshot, from the request to its final status.Open interactive diagram(opens in a new tab)

In order:

  1. Create. The entry point inserts the snapshot as pending and a job_queue row whose heartbeat starts the job's lease. webfrost add runs the pipeline inline and waits; the server, imports and schedules hand it to the job runner and return at once (202).
  2. Wait for a slot. The runner waits for the host's spacing slot, then for a free concurrency slot.
  3. Fetch. The page is fetched with the SSRF-guarded safeFetch (redirects re-checked on every hop, body capped at 50 MB, 30 s timeout). If the fetch fails or the server answers 400 or higher, no extractor runs and the snapshot is failed.
  4. Browser. Chromium launches once and navigates once. The browser guard checks every request (see below). If the browser or navigation fails, the four browser extractors are recorded as failed and everything else continues.
  5. Extract. All extractors run in parallel, each under its own timeout. Text extractors read the cached response, browser extractors use the open page, and media runs yt-dlp as a subprocess. The browser closes as soon as its extractors finish rather than waiting for a long media download.
  6. Record. Every extractor writes either a result or an error row with its duration. The snapshot becomes done if at least one extractor produced output, otherwise failed. Statuses are only pending, done and failed.
  7. Index. done snapshots are indexed for full-text search from the Readability result. An indexing failure does not fail the snapshot.

Jobs, leases and recovery

Every pipeline the server starts goes through one JobRunner (packages/core/src/jobs.ts), limited to WEBFROST_CONCURRENCY running at once (default 3). Each job opens its own database handle and closes it when it settles, so request handlers can close theirs as soon as the row is inserted. (On SQLite the handles of one process share a connection; on PostgreSQL and MySQL they borrow from the process's pool.)

job_queue.status goes pending, running, then done or failed.

State diagram of a snapshot job: pending, then running, then done. Running goes to failed when the fetch fails or no extractor produced output. A pending or running job whose process dies is orphaned once its heartbeat is over 45 seconds old; recovery adopts it back to pending with a new lease and one more attempt, and after 3 attempts marks it failed.
A job's states. The snapshot row shows only pending, done and failed; running exists on the job alone.Open interactive diagram(opens in a new tab)

Leases. A job is leased through job_queue.heartbeat_at and job_queue.lease_token. The token is issued when the job is created; the pipeline refreshes the heartbeat every 10 seconds, and the runner refreshes it for everything it has queued or running, including jobs still waiting for a slot.

Recovery. A recovery sweep adopts a snapshot that is pending and whose heartbeat is older than 45 seconds, meaning that no live process is working on it (a crash, deploy or restart). The server sweeps at startup and then every 60 seconds; webfrost schedule worker does the same. Adoption:

  • is atomic, so two processes sweeping at once cannot adopt the same job: SQLite runs it in a BEGIN IMMEDIATE transaction, PostgreSQL and MySQL lock the job rows with SELECT … FOR UPDATE SKIP LOCKED, which also lets concurrent sweeps split the orphans between them;
  • issues a new lease token;
  • discards partial results of the interrupted run, so a snapshot never ends with duplicate archive results;
  • increments attempts, and marks the snapshot failed instead of running it again after 3 attempts.

Because a live job keeps its heartbeat fresh, a server and a worker on one archive never steal each other's work. Right after a crash the dead process's jobs still look alive for up to 45 seconds, hence the periodic sweep.

Fencing. A worker can also be presumed dead while it is still running: a stalled process, a long pause. Every write a pipeline makes (its status, each archive_results row, its heartbeat) names the lease token it started with and is applied only while the job still holds that token. Once another worker has adopted the job, the old worker's writes are dropped and it stops at its next status change, so it can neither overwrite the new run's outcome nor add results to it. Fenced writes lock the job_queue row first, in the same order as recovery, so the two never deadlock.

Per-host spacing

WEBFROST_DOMAIN_DELAY_MS (default 1000, 0 disables) spaces archives of the same hostname by about that many milliseconds. Each call reserves the next free slot for its host: the first starts at once, the next about one delay later, and so on. Measured gaps can come out a few milliseconds shorter than the setting. Different hosts never wait on each other.

The runner waits for the host slot before it takes a concurrency slot. Otherwise a batch of URLs for one host would park every worker in a sleep and stall unrelated hosts behind it.

Timeouts and limits

WhatLimitSetting
Page fetch, with redirects30 sWEBFROST_FETCH_TIMEOUT_MS
Redirects per fetch5none
Response body50 MBnone
Browser navigation30 snone
Each extractor60 sWEBFROST_EXTRACTOR_TIMEOUT_MS
Each yt-dlp run10 minWEBFROST_MEDIA_TIMEOUT_MS
Media extractor as a wholethree yt-dlp runs3 times WEBFROST_MEDIA_TIMEOUT_MS
Media file500 MBnone
Browser shutdown5 snone
yt-dlp output collectiongives up 3 s after exitnone

When an extractor times out, it is recorded as failed and its abort signal fires, so it cannot store a result afterwards.

The browser guard

Fetching with safeFetch protects only the first request. Without more, a page could make Chromium load http://169.254.169.254/ or an intranet host in an iframe or image and get it captured in the screenshot, PDF or DOM. The guard (browser-guard.ts) applies the SSRF policy to everything the browser loads, in three layers:

  1. Request routing. Every request (navigation, subresources, iframes, fetch, XHR, WebSockets) is checked and aborted if its host is not public. Service workers are blocked.
  2. Redirect detection. Playwright cannot intercept redirect hops, so they are checked from the request events. A blocked hop cannot be stopped mid-flight, but it is reported, and the pipeline then refuses to capture the page: the four browser extractors are recorded as failed. The request itself may still have been sent (blind SSRF, an accepted risk).
  3. Seal. After the page has loaded and its network has gone quiet (up to WEBFROST_SETTLE_MS, so single-page apps can finish loading), every further request is refused before anything is captured. Requests made while waiting still go through the guard. A script cannot wait out the check and then pull an internal URL into a frame. Then the browser scrolls through the page once (one screen at a time, up to 20000 px, a few seconds at most) and back to the top, and the network settles again: sections that sites reveal as they scroll into view, and images loaded only near the viewport, are archived visible.

WEBFROST_ALLOW_PRIVATE_URLS=true turns the address check off everywhere. See Security.

Download flow

How the server serves an archived file.

Sequence of a download: the auth check admits an API key, a session or public read, the download route looks up the latest successful result and validates its storage reference, then either streams the file as a 200 attachment from the filesystem or redirects with a 302 to a presigned S3 URL.
Serving an archived file from disk or from S3.Open interactive diagram(opens in a new tab)

The extractor name is checked against an allowlist, and the stored reference must lie inside snapshots/ (filesystem) or under the configured prefix (S3) before any read. media-metadata is always streamed by the server.

Import flow

Sequence of a bulk import: the CLI or API has the import parser turn the input into URLs, skips already archived URLs for RSS, adds a pending snapshot and job per URL through the JobRunner, records the import (API only) and answers with queued, skipped and total while the pipelines run in the background.
A bulk import, parsed once and queued per URL.Open interactive diagram(opens in a new tab)

Schedules

Sequence of one scheduler tick: the scheduler reads due, unpaused schedules, claims each by advancing next_run_at only if it is unchanged, fetches feed items for RSS schedules, inserts a snapshot and job, enqueues it on the JobRunner, waits for the pipeline and records last_status and consecutive failures.
One scheduler tick, every 30 seconds.Open interactive diagram(opens in a new tab)

The claim advances next_run_at before the pipeline runs, so a slow run is not started again on the next tick, and two processes on one archive cannot both run the same schedule. A run that outlasts its cron period waits until it finishes. A cron expression that no longer parses is skipped without affecting other schedules.

On this page