Overview
The parts of WebFrost, what runs where, and how they connect, with diagrams.
WebFrost is a Bun monorepo with one shared library and thin layers on top. The server process is self-contained: it serves the API and UI, runs archive jobs and runs due schedules. No separate worker or queue service is required.
System overview
Who talks to WebFrost, what runs inside the server process, and what WebFrost talks to.
Packages and processes
| Package | Role | Entry point |
|---|---|---|
@webfrost/core | Library: database, pipeline, extractors, job runner, scheduler, import, auth, storage | not executable |
@webfrost/cli | The webfrost command, a thin layer over core | packages/cli/src/index.ts |
@webfrost/server | Elysia HTTP server: REST API, static UI, embedded job runner and scheduler | packages/server/src/index.ts |
@webfrost/ui | React single-page app, built to packages/ui/dist/ and served by the server | packages/ui/src/main.tsx |
@webfrost/sdk | Typed REST client, with no dependency on core | packages/sdk/src/index.ts |
Key relationships:
- CLI and server both use
core. Pipeline logic lives once, in core. - The server embeds the scheduler and the job runner.
WEBFROST_CONCURRENCYbounds the pipelines of the whole process.webfrost schedule workerruns the same scheduler for CLI-only setups and can run next to the server on one archive, because due schedules are claimed atomically.WEBFROST_SCHEDULER=falseturns off the server's scheduler. - The UI talks only to the server, through the same REST API as the SDK.
- The CLI works on the directory directly. It does not call the server, so it works with no server running. (
webfrost servestarts the server itself, in the CLI's process.)
Inside @webfrost/core
| Component | Responsibility |
|---|---|
Pipeline (pipeline.ts) | Orchestrates one snapshot: fetch, launch the browser once, fan out to the extractors, record results, set the final status. |
Job runner (jobs.ts) | Runs pipelines in the background, bounded by WEBFROST_CONCURRENCY. Each job gets its own database connection. Leases jobs and recovers orphans. |
Scheduler (scheduler.ts) | Polls for due schedules, claims each atomically and hands it to the job runner. |
Extractors (extractors/) | One module per format: warc, readability, html, pdf, screenshot, dom-snapshot, singlefile, media, media-metadata. |
Fetch cache (fetch-cache.ts) | Per-snapshot, in-memory: the page is fetched over HTTP once and shared by the text extractors. |
SSRF guards (ssrf-guard.ts, browser-guard.ts) | Block requests to private addresses, for the HTTP fetch and for every browser request. |
Throttle (throttle.ts) | Spaces archives of the same host. |
Database (db.ts) | One async data-access layer for SQLite, PostgreSQL and MySQL, and their migrations. SQL through the Kysely query builder, no ORM. |
Storage (storage/) | Filesystem or S3 backend behind one interface. |
| Auth, search, tags, import | Admin users, API keys and sessions; full-text search (FTS5, tsvector or FULLTEXT by engine); tag management; text, OPML, Netscape and RSS parsing. |
Design principles
One browser navigation per snapshot. Many archivers start a separate browser session per format. WebFrost opens the page once and runs the PDF, screenshot, DOM snapshot and SingleFile extractors against that one load. This is the main source of speed.
One shared HTTP fetch. The page is fetched once. WARC, Readability and HTML read that response.
Open formats. WARC, PDF, PNG, HTML and SQLite. No proprietary binary format.
One directory. With the default SQLite database and filesystem storage, everything lives in one directory and rsync backs it up. PostgreSQL, MySQL and S3 are optional.
Small and replaceable layers. The CLI, server and SDK add no archive logic. A new interface implements against core without touching the pipeline.
In-process queue, durable state. The job runner is in memory, but job state and heartbeats are in the database, so a restart or crash does not strand snapshots. A distributed queue is not needed for self-hosted use.