Storage model
How WebFrost stores an archive, with a database (SQLite by default, or PostgreSQL or MySQL) for metadata and plain files or S3 for content, and why.
A WebFrost archive is a directory. A database holds the metadata and search index, and ordinary files hold what was captured. By default the database is a SQLite file in the same directory; with DATABASE_URL it can be PostgreSQL or MySQL instead. Optionally the files can live in an S3-compatible bucket. The two choices are independent: any database works with either storage backend.
One directory
The html and readability results live in the database. Files exist only for extractors that ran and succeeded. See Disk layout.
What the database holds
- snapshots: one row per URL, with its status (
pending,doneorfailed) and timestamp. - archive_results: per snapshot and extractor, the output (inline content or a storage reference), the duration, and the
errorwhen it failed. - job_queue: background job state with the lease heartbeat and attempt count.
- tags and snapshot_tags.
- schedules: cron definitions with last-run state and failure counts.
- admin_users, sessions, api_keys: the admin account, login sessions and hashed API keys.
- imports: history of bulk imports.
- snapshots_fts (SQLite) or search_index (PostgreSQL, MySQL): the full-text index over URL, title, text and tags.
The tables are the same on every engine; column detail and the per-engine differences are in Database.
Databases
| SQLite (default) | PostgreSQL | MySQL | |
|---|---|---|---|
| Setup | none | a database and a user | a database and a user |
| Lives | in WEBFROST_DIR | on the database server | on the database server |
| Search | FTS5, full query syntax | tsvector + GIN | FULLTEXT |
| Backup | copy the file, or Litestream | pg_dump | mysqldump |
| Good for | one host, the common case | an existing PostgreSQL, managed hosting, replicas | an existing MySQL |
All three run the same code through one data-access layer (packages/core/src/db.ts, Kysely as the query builder): SQLite on Bun's built-in driver, PostgreSQL on pg, MySQL on mysql2. Queries that differ by engine (full-text search, job claiming, "insert or ignore", the clock) are written per dialect; everything else is shared SQL. Timestamps are stored as UTC text in one format on every engine, so the API output is identical.
Storage backends
Both backends implement one interface (write, read, size, delete, presigned URL), so the pipeline does not care which one is active.
| Filesystem | S3 | |
|---|---|---|
| Files at | WEBFROST_DIR/snapshots/<id>/ | <prefix>/<id>/<file> |
| Stored reference | absolute path | s3:<key> |
| Download | streamed by the server | 302 to a presigned URL |
| Database | any | any |
Writes from Chromium and yt-dlp go to a temporary file first. With S3 the file is then uploaded and the temporary copy removed.
The reference stored in the database is checked on every read: a filesystem path must lie inside snapshots/ and an S3 key under the configured prefix. This blocks path traversal through a tampered row, and it is also why a backend cannot be switched on a populated archive without breaking old downloads. See Storage.
Why this design
No external services. SQLite and a directory are enough. No database server, no object store required — but both are there when you already run them.
Self-contained backup. With SQLite, copy the directory. Restore by copying it back to the same path. With a database server, dump it alongside the files.
Open formats. WARC is readable by ReplayWeb.page and other tools. PDF, PNG and HTML need nothing special. The database is plain tables in SQLite, PostgreSQL or MySQL.
No lock-in. If WebFrost disappears, the files remain readable. The WARC files alone are enough for long-term preservation.
Concurrency
Several processes can share one archive: the server, CLI commands and a schedule worker. SQLite runs in WAL mode with a 5-second busy timeout; within a process all work shares one connection. PostgreSQL and MySQL use a connection pool per process. Schedules are claimed with an atomic compare-and-set and jobs adopted under row locks (FOR UPDATE SKIP LOCKED on PostgreSQL and MySQL), with lease tokens fencing off a worker that lost its job, so processes do not run the same work twice.
Backing up
rsync -av --progress "$WEBFROST_DIR"/ /backup/webfrost/With S3, also back up the bucket; with PostgreSQL or MySQL, dump the database. For hot backups, continuous replication and restores, see Backups.