Skip to Content
WebFrostdocs
Getting started

Archive your first page

Archive a page, open every format WebFrost saved, and search it, from the CLI and the web UI.

In about ten minutes you archive https://example.com, look at each saved format, and search the text. You need the installed webfrost binary, or a running Docker server for the UI part. Run webfrost setup once if you have no Chromium.

1. Archive a URL

webfrost add https://example.com
# [14:02:11] INFO  Archiving https://example.com...
# [14:02:14] INFO  Snapshot 1 archived.

add runs the whole pipeline in your terminal and returns when it finishes. It exits 0 when the snapshot is done (at least one extractor produced output) and 1 when it ended failed, with the reason on stderr. --json prints the result and each extractor's outcome as JSON.

The archive lives in ~/.webfrost unless you set WEBFROST_DIR. webfrost serve uses the same directory.

What happened: WebFrost fetched the page once, opened it in headless Chromium once, and ran every extractor against those two results in parallel. A snapshot is done when at least one extractor produced output.

2. List snapshots

webfrost list
# 1    done    https://example.com

Newest first, like the web UI and the API. --limit, --page, --status failed and --json narrow it.

3. Look at every format

Files for snapshot 1 are in ~/.webfrost/snapshots/1/:

ls ~/.webfrost/snapshots/1/
# archive.warc  dom.html  page.pdf  screenshot.png  singlefile.html
FormatHow to look at it
screenshot.pngOpen it in any image viewer. It is the full page.
page.pdfOpen it in any PDF viewer.
singlefile.htmlOpen it in a browser. Stylesheets and images are inlined, so it works offline.
dom.htmlThe DOM after the page loaded. Useful when the page builds its content with JavaScript.
archive.warcOpen it in ReplayWeb.page or process it with warcio.
ReadabilityStored in the database. See the query below.
HTMLThe raw response body, also stored in the database.

Read the extracted article text straight from SQLite:

sqlite3 ~/.webfrost/archive.db \
  "SELECT content FROM archive_results WHERE snapshot_id = 1 AND extractor = 'readability';"
# {"title":"Example Domain","byline":null,"textContent":"...","excerpt":"..."}

In the web UI, click the snapshot in Snapshots. It opens on the whole window (its address is /#/snapshots/1) with one view per format: Page (the archived page in a sandbox, which can go full screen or open in a new tab), Reader (the extracted article), Screenshot, Media, Files (downloads of the HTML, WARC, PDF, SingleFile and DOM files) and Details. It opens on Media when the snapshot has a video, else on Page. Details (I) shows the URL, status, capture time, tags and extractors.

A snapshot of a Wikipedia article open in the viewer

Which extractors ran, and why one may be missing, is in Extractors.

4. Search it

When a snapshot finishes, WebFrost adds its extracted text, URL, title and tag names to the full-text index (SQLite FTS5 by default; PostgreSQL and MySQL use their own full-text search).

webfrost search "documentation examples"
# 1	https://example.com
#   This domain is for use in documentation examples without needing permission. ...

--json, --limit and --page are available too.

5. Tag it

webfrost tag add 1 reference
webfrost tag list 1

Tag names are searchable too.

Where next

On this page