Extractors
Every format WebFrost saves for a snapshot, what is in it, the file it lands in, and when it fails or is skipped.
WebFrost runs up to nine extractors for each snapshot. It fetches the page once (shared by the text extractors), opens it once in headless Chromium (shared by the browser extractors), and runs all of them in parallel.
A snapshot is done when at least one extractor produced output and failed otherwise. A single failed extractor never fails the snapshot.
Formats
| Extractor | Output | Needs | What it contains |
|---|---|---|---|
warc | archive.warc | none | A WARC 1.1 file of the HTTP fetch: the response as the server sent it, any redirects before it, and a metadata record. See The WARC file. Subresources are not included. |
readability | JSON in the database | none | The article as found by Mozilla Readability: title, byline, textContent (plain text, paragraph breaks kept) and excerpt. When the page is not an article, title falls back to the page's <title> and the other fields are null. Feeds the search index. Only the first 10 MB of the page are parsed (WEBFROST_READABILITY_MAX_BYTES). |
html | text in the database | none | The HTML response body as text, decoded with the page's charset (the Content-Type charset, then <meta charset>, then UTF-8). |
pdf | page.pdf | Chromium | Full-page PDF rendered by the browser, with backgrounds printed. It follows the archived theme (light or dark). |
thumbnail | thumbnail.webp | Chromium | The first screen (the viewport) as a WebP 640 px wide (half size), quality 70, about 15-20 KB. Lists and cards load it instead of the full-page screenshot, which can be several MB. It is taken before the screenshot, so it exists even when the full page fails. Snapshots archived before it existed have none. |
screenshot | screenshot.png | Chromium | Full-page PNG, cut at 10000 px: Chromium cannot paint a taller image, so a longer page (a long article) is cut there instead of failing. It gets the extractor's time budget (WEBFROST_EXTRACTOR_TIMEOUT_MS, minus 5 s) to paint. |
dom-snapshot | dom.html | Chromium | The serialized DOM after the page loaded, which includes content built by JavaScript. |
singlefile | singlefile.html | Chromium | The rendered page as one HTML file, with external stylesheets, their fonts, img images and images referenced from CSS (stylesheets, <style> blocks and style attributes) inlined as data URIs. The page keeps the theme it was archived in: prefers-color-scheme conditions are fixed to the archived scheme, so a page archived dark shows dark on a light system. Stylesheet media queries are kept, HTML entities in URLs are decoded, and srcset and <picture> sources are dropped so the inlined image is the one shown. Assets that cannot be downloaded, or that no longer fit in the 100 MB budget (WEBFROST_SINGLEFILE_MAX_BYTES), stay as external links. |
media | media/media.mp4 or media/media.mp3 | yt-dlp | The video, as an mp4 a browser plays, up to WEBFROST_MEDIA_MAX_HEIGHT (1080p by default). VP9 or AV1 video with Opus audio comes first, because Chromium builds without the patented codecs cannot play H.264 or AAC; H.264 and AAC come next, and AVI, FLV and WMV, which no browser plays, only when the site offers nothing else (archive.org lists them first). Falls back to an audio-only mp3 when no video can be downloaded. Skipped over 500 MB. For a playlist or channel URL, only the first item. |
media-poster | media/poster.jpg | ffmpeg | A representative still from the downloaded video (a typical frame from about a tenth of the way in, 640 px wide). Lists show it instead of the page's thumbnail, and the player uses it as its cover. Taken only after a successful video download; if no frame can be taken, the video is kept without it. |
media-metadata | media/media-metadata.json | yt-dlp | The yt-dlp metadata (title, uploader, duration, formats). |
The PDF, screenshot, thumbnail, DOM snapshot and SingleFile files come from the browser; the WARC, Readability and HTML results come from the HTTP fetch. When the URL is not text (a PDF or an image linked directly), html and readability do not run and the file is kept, byte for byte, in the WARC. The html and readability results are stored in the database, not as files. File names are the same on the filesystem and in S3. See Disk layout.
When something fails or is skipped
| Situation | Result |
|---|---|
| The page fetch fails or the server answers HTTP 400 or higher | No extractor runs. The snapshot is failed. |
| Chromium is missing, navigation fails, or a request violates the SSRF policy | pdf, thumbnail, screenshot, dom-snapshot and singlefile are recorded as failed with an error. The rest still run. |
| The page redirects to a private address inside the browser | The same four are recorded as failed (blocked: page redirected to a non-public address), so nothing from the internal host is captured. |
An extractor exceeds WEBFROST_EXTRACTOR_TIMEOUT_MS (60 s) | That extractor is recorded as failed. Its downloads are cancelled and nothing it produces afterwards is kept. |
A yt-dlp call exceeds WEBFROST_MEDIA_TIMEOUT_MS (10 min) | yt-dlp is killed together with the processes it started (ffmpeg). |
yt-dlp is not installed, or the URL is not on a site yt-dlp supports | media and media-metadata are skipped silently. No entry, no failure. |
| The site is one yt-dlp supports but it refuses the video (private, removed, or "Sign in to confirm you're not a bot", which video sites often answer to server addresses) | media fails with yt-dlp's reason, shown in the snapshot's Details. |
| The media download fails or exceeds the size cap | media-metadata is kept, and media fails with yt-dlp's reason. |
| Readability finds no article | The readability result exists with null fields. |
Failures show up as status: "failed" with an error in extractorMeta of GET /api/v1/snapshots/:id, and in the snapshot detail view. Fixes are in Troubleshooting.
The WARC file
archive.warc holds, in order:
- A
warcinforecord. - One
responserecord per redirect that was followed (WARC-Target-URIis the URL that answered with the redirect). - The final
responserecord. ItsWARC-Target-URIis the URL that served the page, after redirects. - A
metadatarecord (WARC-Concurrent-Tothe final response) withrequested-url:, the URL you archived, and oneredirect:line per hop.
Each response record holds the HTTP status line, the headers and the body exactly as the server sent it: a gzip or brotli response stays compressed, with its Content-Encoding header, and binary and non-UTF-8 bodies are not altered. The one change is the transfer framing, which the HTTP client removes: Transfer-Encoding is dropped and Content-Length is set to the length of the body that follows. Every record has a WARC-Block-Digest and responses a WARC-Payload-Digest (SHA-1, base32), so warcio check can verify the file.
Header names are lower-case and repeated headers other than Set-Cookie are joined with commas, as the HTTP client reports them. There is no request record.
Limits
| Variable | Default | What it limits |
|---|---|---|
WEBFROST_EXTRACTOR_TIMEOUT_MS | 60000 | Time for each extractor except media. |
WEBFROST_MEDIA_TIMEOUT_MS | 600000 | Time for each yt-dlp call. |
WEBFROST_READABILITY_MAX_BYTES | 10485760 (10 MB) | Bytes of HTML given to Readability; the rest of a bigger page is ignored. 0 turns the cap off. Parsing blocks the server while it runs. |
WEBFROST_SINGLEFILE_MAX_BYTES | 104857600 (100 MB) | Total size of the assets inlined into one singlefile.html. |
WEBFROST_SINGLEFILE_CONCURRENCY | 8 | Asset downloads at once for one singlefile.html. |
WEBFROST_SETTLE_MS | 5000 | Wait for the network to go quiet after the page loads, before the browser extractors capture. 0 skips it. |
WEBFROST_MEDIA_MAX_HEIGHT | 1080 | Tallest video downloaded. 0 for no limit. |
WEBFROST_COLOR_SCHEME | light | Theme the browser archives in when the request does not choose one. |
yt-dlp options
Every yt-dlp call gets --ignore-config (no config file on the host can add options), --no-playlist --playlist-items 1 (one item, never a whole playlist or channel) and --use-extractors default,-generic (only sites yt-dlp knows; the generic extractor would download and scan any page for embedded media). The URL is passed after --. One --dump-json call both detects a media page and reads its metadata.
Requirements
Chromium is needed for four extractors. Run webfrost setup to install one, or set WEBFROST_CHROMIUM_PATH to an existing binary; a system Chrome or Chromium is found on its own. The Docker image bundles it. See Install.
yt-dlp (and ffmpeg for merging streams) is needed for media. Any URL yt-dlp supports qualifies: YouTube, Vimeo, SoundCloud, Twitch clips and many more. The Docker image bundles both.
Reading the results
- Files on disk:
ls $WEBFROST_DIR/snapshots/<snapshot-id>/. - Download over the API:
GET /api/v1/snapshots/:id/archives/:extractor/download, where:extractoris one of the names above. See Archive files. - Database:
archive_resultshas one row per extractor run.
WARC files open in ReplayWeb.page and can be processed with warcio.
Supported sources
- Single URL: any
httporhttpsURL, fromwebfrost add, the UI,POST /api/v1/snapshotsor the SDK. Other schemes are rejected. - Bulk sources: text lists, Netscape bookmarks, OPML and RSS or Atom feeds. See Bulk import.
- Recurring: a URL or an RSS feed on a cron schedule. See Scheduled archiving.
- Not supported: pages behind a login. See Authenticated pages.