Skip to Content
WebFrostdocs
Reference

Disk layout

Where WebFrost puts the database and each snapshot's files on disk and in S3, and how to back them up.

Everything WebFrost stores lives under WEBFROST_DIR (S3 files aside).

The archive folder: WEBFROST_DIR holds archive.db (with its WAL files while in use) and snapshots/, which has one folder per snapshot id containing archive.warc, page.pdf, screenshot.png, dom.html, singlefile.html and, when yt-dlp found media, a media/ folder with media.mp4 or media.mp3 and media-metadata.json.
Everything under WEBFROST_DIR. The WARC also records redirects; media.mp3 replaces media.mp4 for the audio-only fallback.Open interactive diagram(opens in a new tab)

Next to screenshot.png there is thumbnail.webp: a small WebP of the first screen, which lists and cards load instead of the full page. In media/, poster.jpg is a still from the video, used by lists and as the player's cover. The html and readability results are stored in the database, not as files. A file exists only if its extractor ran and succeeded, so most snapshots lack some of these.

SQLite also creates archive.db-wal and archive.db-shm next to the database while it is in use (write-ahead logging). Keep them with the database.

tmp/ holds snapshot bundles being uploaded to POST /api/v1/snapshots/import (webfrost push), one private folder (bundle-<uuid>) per request, removed when the request ends; folders a crash left behind are removed at the next server start once they are an hour old. It needs no backup.

Snapshot ids

A snapshot's directory is named after its integer id, matching snapshots.id. The directory is created when the first file is written.

File references are paths

The database refers to each file by the path it was written to: an absolute path with the filesystem backend (WEBFROST_DIR/snapshots/<id>/<file>), and s3:<key> with S3. That has two consequences:

  • Use an absolute WEBFROST_DIR. With a relative one (./data) the references are relative too, and break when you start WebFrost from another directory.
  • If you restore or move the archive, put it back at the same absolute path (in Docker, always /data).

S3 layout

With WEBFROST_STORAGE_BACKEND=s3, archive.db stays local and the files go to:

<S3_PATH_PREFIX>/<snapshot-id>/<filename>

The default prefix is snapshots, so a screenshot is at snapshots/42/screenshot.png. See Storage.

Backup

Back up the whole directory. With S3, also back up the bucket: the database is what maps snapshots to their files.

rsync -av --progress "$WEBFROST_DIR"/ /backup/webfrost/
# or
tar -czf webfrost-backup-$(date +%Y%m%d).tar.gz "$WEBFROST_DIR"/

For a backup while the server runs, use SQLite's online backup for the database (sqlite3 archive.db ".backup file.db") and copy snapshots/ separately. Docker specifics are in Deploy.

Deleting a snapshot

webfrost delete <id> and DELETE /api/v1/snapshots/:id remove the database rows, the search entry and the snapshot's files (the directory, or the S3 objects) on a best-effort basis. There is no trash: deletion is permanent.

On this page