Skip to Content
WebFrostdocs
Architecture

Storage model

How WebFrost stores an archive, with a database (SQLite by default, or PostgreSQL or MySQL) for metadata and plain files or S3 for content, and why.

A WebFrost archive is a directory. A database holds the metadata and search index, and ordinary files hold what was captured. By default the database is a SQLite file in the same directory; with DATABASE_URL it can be PostgreSQL or MySQL instead. Optionally the files can live in an S3-compatible bucket. The two choices are independent: any database works with either storage backend.

One directory

The archive folder: WEBFROST_DIR holds archive.db (with its WAL files while in use) and snapshots/, which has one folder per snapshot id containing archive.warc, page.pdf, screenshot.png, dom.html, singlefile.html and, when yt-dlp found media, a media/ folder with media.mp4 or media.mp3 and media-metadata.json.
One directory per archive, one folder per snapshot. archive.db is absent when DATABASE_URL points at PostgreSQL or MySQL.Open interactive diagram(opens in a new tab)

The html and readability results live in the database. Files exist only for extractors that ran and succeeded. See Disk layout.

What the database holds

  • snapshots: one row per URL, with its status (pending, done or failed) and timestamp.
  • archive_results: per snapshot and extractor, the output (inline content or a storage reference), the duration, and the error when it failed.
  • job_queue: background job state with the lease heartbeat and attempt count.
  • tags and snapshot_tags.
  • schedules: cron definitions with last-run state and failure counts.
  • admin_users, sessions, api_keys: the admin account, login sessions and hashed API keys.
  • imports: history of bulk imports.
  • snapshots_fts (SQLite) or search_index (PostgreSQL, MySQL): the full-text index over URL, title, text and tags.
Entity-relationship diagram of every table and column. snapshots is the hub: archive_results and job_queue reference it one-to-many, snapshot_tags joins it to tags, and search_index (PostgreSQL, MySQL) or the FTS5 table snapshots_fts (SQLite) holds one search row per snapshot. admin_users, api_keys, sessions, schedules and imports stand alone.
Every table and column, grouped by what they serve. Column types are SQLite's, except search_index (PostgreSQL).Open interactive diagram(opens in a new tab)

The tables are the same on every engine; column detail and the per-engine differences are in Database.

Databases

SQLite (default)PostgreSQLMySQL
Setupnonea database and a usera database and a user
Livesin WEBFROST_DIRon the database serveron the database server
SearchFTS5, full query syntaxtsvector + GINFULLTEXT
Backupcopy the file, or Litestreampg_dumpmysqldump
Good forone host, the common casean existing PostgreSQL, managed hosting, replicasan existing MySQL

All three run the same code through one data-access layer (packages/core/src/db.ts, Kysely as the query builder): SQLite on Bun's built-in driver, PostgreSQL on pg, MySQL on mysql2. Queries that differ by engine (full-text search, job claiming, "insert or ignore", the clock) are written per dialect; everything else is shared SQL. Timestamps are stored as UTC text in one format on every engine, so the API output is identical.

Storage backends

Both backends implement one interface (write, read, size, delete, presigned URL), so the pipeline does not care which one is active.

FilesystemS3
Files atWEBFROST_DIR/snapshots/<id>/<prefix>/<id>/<file>
Stored referenceabsolute paths3:<key>
Downloadstreamed by the server302 to a presigned URL
Databaseanyany

Writes from Chromium and yt-dlp go to a temporary file first. With S3 the file is then uploaded and the temporary copy removed.

The reference stored in the database is checked on every read: a filesystem path must lie inside snapshots/ and an S3 key under the configured prefix. This blocks path traversal through a tampered row, and it is also why a backend cannot be switched on a populated archive without breaking old downloads. See Storage.

Why this design

No external services. SQLite and a directory are enough. No database server, no object store required — but both are there when you already run them.

Self-contained backup. With SQLite, copy the directory. Restore by copying it back to the same path. With a database server, dump it alongside the files.

Open formats. WARC is readable by ReplayWeb.page and other tools. PDF, PNG and HTML need nothing special. The database is plain tables in SQLite, PostgreSQL or MySQL.

No lock-in. If WebFrost disappears, the files remain readable. The WARC files alone are enough for long-term preservation.

Concurrency

Several processes can share one archive: the server, CLI commands and a schedule worker. SQLite runs in WAL mode with a 5-second busy timeout; within a process all work shares one connection. PostgreSQL and MySQL use a connection pool per process. Schedules are claimed with an atomic compare-and-set and jobs adopted under row locks (FOR UPDATE SKIP LOCKED on PostgreSQL and MySQL), with lease tokens fencing off a worker that lost its job, so processes do not run the same work twice.

Backing up

rsync -av --progress "$WEBFROST_DIR"/ /backup/webfrost/

With S3, also back up the bucket; with PostgreSQL or MySQL, dump the database. For hot backups, continuous replication and restores, see Backups.

On this page