Skip to Content
WebFrostdocs
Architecture

Overview

The parts of WebFrost, what runs where, and how they connect, with diagrams.

WebFrost is a Bun monorepo with one shared library and thin layers on top. The server process is self-contained: it serves the API and UI, runs archive jobs and runs due schedules. No separate worker or queue service is required.

System overview

Who talks to WebFrost, what runs inside the server process, and what WebFrost talks to.

The web UI and REST clients call the HTTP API in the server process. The API, the scheduler and the recovery sweep hand snapshots to the JobRunner, which runs the pipeline. The CLI calls the pipeline directly in its own process. The pipeline writes results to the database (SQLite, PostgreSQL or MySQL) and files to the filesystem or S3, and reaches the public web through the guarded HTTP fetch, Chromium and yt-dlp.
Clients, the server process and everything the pipeline reaches.Open interactive diagram(opens in a new tab)

Packages and processes

PackageRoleEntry point
@webfrost/coreLibrary: database, pipeline, extractors, job runner, scheduler, import, auth, storagenot executable
@webfrost/cliThe webfrost command, a thin layer over corepackages/cli/src/index.ts
@webfrost/serverElysia HTTP server: REST API, static UI, embedded job runner and schedulerpackages/server/src/index.ts
@webfrost/uiReact single-page app, built to packages/ui/dist/ and served by the serverpackages/ui/src/main.tsx
@webfrost/sdkTyped REST client, with no dependency on corepackages/sdk/src/index.ts

Key relationships:

  • CLI and server both use core. Pipeline logic lives once, in core.
  • The server embeds the scheduler and the job runner. WEBFROST_CONCURRENCY bounds the pipelines of the whole process. webfrost schedule worker runs the same scheduler for CLI-only setups and can run next to the server on one archive, because due schedules are claimed atomically. WEBFROST_SCHEDULER=false turns off the server's scheduler.
  • The UI talks only to the server, through the same REST API as the SDK.
  • The CLI works on the directory directly. It does not call the server, so it works with no server running. (webfrost serve starts the server itself, in the CLI's process.)

Inside @webfrost/core

Import and the scheduler hand URLs to the JobRunner, which waits for the per-host throttle and runs the pipeline. The pipeline fetches once through the fetch cache and SSRF guard, opens one browser page behind the browser guard, runs the extractors and indexes finished snapshots in search. Extractors write files to storage and results through the data-access layer, which search, tags and auth also use.
The modules of @webfrost/core along one snapshot.Open interactive diagram(opens in a new tab)
ComponentResponsibility
Pipeline (pipeline.ts)Orchestrates one snapshot: fetch, launch the browser once, fan out to the extractors, record results, set the final status.
Job runner (jobs.ts)Runs pipelines in the background, bounded by WEBFROST_CONCURRENCY. Each job gets its own database connection. Leases jobs and recovers orphans.
Scheduler (scheduler.ts)Polls for due schedules, claims each atomically and hands it to the job runner.
Extractors (extractors/)One module per format: warc, readability, html, pdf, screenshot, dom-snapshot, singlefile, media, media-metadata.
Fetch cache (fetch-cache.ts)Per-snapshot, in-memory: the page is fetched over HTTP once and shared by the text extractors.
SSRF guards (ssrf-guard.ts, browser-guard.ts)Block requests to private addresses, for the HTTP fetch and for every browser request.
Throttle (throttle.ts)Spaces archives of the same host.
Database (db.ts)One async data-access layer for SQLite, PostgreSQL and MySQL, and their migrations. SQL through the Kysely query builder, no ORM.
Storage (storage/)Filesystem or S3 backend behind one interface.
Auth, search, tags, importAdmin users, API keys and sessions; full-text search (FTS5, tsvector or FULLTEXT by engine); tag management; text, OPML, Netscape and RSS parsing.

Design principles

One browser navigation per snapshot. Many archivers start a separate browser session per format. WebFrost opens the page once and runs the PDF, screenshot, DOM snapshot and SingleFile extractors against that one load. This is the main source of speed.

One shared HTTP fetch. The page is fetched once. WARC, Readability and HTML read that response.

Open formats. WARC, PDF, PNG, HTML and SQLite. No proprietary binary format.

One directory. With the default SQLite database and filesystem storage, everything lives in one directory and rsync backs it up. PostgreSQL, MySQL and S3 are optional.

Small and replaceable layers. The CLI, server and SDK add no archive logic. A new interface implements against core without touching the pipeline.

In-process queue, durable state. The job runner is in memory, but job state and heartbeats are in the database, so a restart or crash does not strand snapshots. A distributed queue is not needed for self-hosted use.

On this page