Skip to Content
WebFrostdocs
Reference

Extractors

Every format WebFrost saves for a snapshot, what is in it, the file it lands in, and when it fails or is skipped.

WebFrost runs up to nine extractors for each snapshot. It fetches the page once (shared by the text extractors), opens it once in headless Chromium (shared by the browser extractors), and runs all of them in parallel.

A snapshot is done when at least one extractor produced output and failed otherwise. A single failed extractor never fails the snapshot.

Formats

ExtractorOutputNeedsWhat it contains
warcarchive.warcnoneA WARC 1.1 file of the HTTP fetch: the response as the server sent it, any redirects before it, and a metadata record. See The WARC file. Subresources are not included.
readabilityJSON in the databasenoneThe article as found by Mozilla Readability: title, byline, textContent (plain text, paragraph breaks kept) and excerpt. When the page is not an article, title falls back to the page's <title> and the other fields are null. Feeds the search index. Only the first 10 MB of the page are parsed (WEBFROST_READABILITY_MAX_BYTES).
htmltext in the databasenoneThe HTML response body as text, decoded with the page's charset (the Content-Type charset, then <meta charset>, then UTF-8).
pdfpage.pdfChromiumFull-page PDF rendered by the browser, with backgrounds printed. It follows the archived theme (light or dark).
thumbnailthumbnail.webpChromiumThe first screen (the viewport) as a WebP 640 px wide (half size), quality 70, about 15-20 KB. Lists and cards load it instead of the full-page screenshot, which can be several MB. It is taken before the screenshot, so it exists even when the full page fails. Snapshots archived before it existed have none.
screenshotscreenshot.pngChromiumFull-page PNG, cut at 10000 px: Chromium cannot paint a taller image, so a longer page (a long article) is cut there instead of failing. It gets the extractor's time budget (WEBFROST_EXTRACTOR_TIMEOUT_MS, minus 5 s) to paint.
dom-snapshotdom.htmlChromiumThe serialized DOM after the page loaded, which includes content built by JavaScript.
singlefilesinglefile.htmlChromiumThe rendered page as one HTML file, with external stylesheets, their fonts, img images and images referenced from CSS (stylesheets, <style> blocks and style attributes) inlined as data URIs. The page keeps the theme it was archived in: prefers-color-scheme conditions are fixed to the archived scheme, so a page archived dark shows dark on a light system. Stylesheet media queries are kept, HTML entities in URLs are decoded, and srcset and <picture> sources are dropped so the inlined image is the one shown. Assets that cannot be downloaded, or that no longer fit in the 100 MB budget (WEBFROST_SINGLEFILE_MAX_BYTES), stay as external links.
mediamedia/media.mp4 or media/media.mp3yt-dlpThe video, as an mp4 a browser plays, up to WEBFROST_MEDIA_MAX_HEIGHT (1080p by default). VP9 or AV1 video with Opus audio comes first, because Chromium builds without the patented codecs cannot play H.264 or AAC; H.264 and AAC come next, and AVI, FLV and WMV, which no browser plays, only when the site offers nothing else (archive.org lists them first). Falls back to an audio-only mp3 when no video can be downloaded. Skipped over 500 MB. For a playlist or channel URL, only the first item.
media-postermedia/poster.jpgffmpegA representative still from the downloaded video (a typical frame from about a tenth of the way in, 640 px wide). Lists show it instead of the page's thumbnail, and the player uses it as its cover. Taken only after a successful video download; if no frame can be taken, the video is kept without it.
media-metadatamedia/media-metadata.jsonyt-dlpThe yt-dlp metadata (title, uploader, duration, formats).

The PDF, screenshot, thumbnail, DOM snapshot and SingleFile files come from the browser; the WARC, Readability and HTML results come from the HTTP fetch. When the URL is not text (a PDF or an image linked directly), html and readability do not run and the file is kept, byte for byte, in the WARC. The html and readability results are stored in the database, not as files. File names are the same on the filesystem and in S3. See Disk layout.

When something fails or is skipped

SituationResult
The page fetch fails or the server answers HTTP 400 or higherNo extractor runs. The snapshot is failed.
Chromium is missing, navigation fails, or a request violates the SSRF policypdf, thumbnail, screenshot, dom-snapshot and singlefile are recorded as failed with an error. The rest still run.
The page redirects to a private address inside the browserThe same four are recorded as failed (blocked: page redirected to a non-public address), so nothing from the internal host is captured.
An extractor exceeds WEBFROST_EXTRACTOR_TIMEOUT_MS (60 s)That extractor is recorded as failed. Its downloads are cancelled and nothing it produces afterwards is kept.
A yt-dlp call exceeds WEBFROST_MEDIA_TIMEOUT_MS (10 min)yt-dlp is killed together with the processes it started (ffmpeg).
yt-dlp is not installed, or the URL is not on a site yt-dlp supportsmedia and media-metadata are skipped silently. No entry, no failure.
The site is one yt-dlp supports but it refuses the video (private, removed, or "Sign in to confirm you're not a bot", which video sites often answer to server addresses)media fails with yt-dlp's reason, shown in the snapshot's Details.
The media download fails or exceeds the size capmedia-metadata is kept, and media fails with yt-dlp's reason.
Readability finds no articleThe readability result exists with null fields.

Failures show up as status: "failed" with an error in extractorMeta of GET /api/v1/snapshots/:id, and in the snapshot detail view. Fixes are in Troubleshooting.

The WARC file

archive.warc holds, in order:

  1. A warcinfo record.
  2. One response record per redirect that was followed (WARC-Target-URI is the URL that answered with the redirect).
  3. The final response record. Its WARC-Target-URI is the URL that served the page, after redirects.
  4. A metadata record (WARC-Concurrent-To the final response) with requested-url:, the URL you archived, and one redirect: line per hop.

Each response record holds the HTTP status line, the headers and the body exactly as the server sent it: a gzip or brotli response stays compressed, with its Content-Encoding header, and binary and non-UTF-8 bodies are not altered. The one change is the transfer framing, which the HTTP client removes: Transfer-Encoding is dropped and Content-Length is set to the length of the body that follows. Every record has a WARC-Block-Digest and responses a WARC-Payload-Digest (SHA-1, base32), so warcio check can verify the file.

Header names are lower-case and repeated headers other than Set-Cookie are joined with commas, as the HTTP client reports them. There is no request record.

Limits

VariableDefaultWhat it limits
WEBFROST_EXTRACTOR_TIMEOUT_MS60000Time for each extractor except media.
WEBFROST_MEDIA_TIMEOUT_MS600000Time for each yt-dlp call.
WEBFROST_READABILITY_MAX_BYTES10485760 (10 MB)Bytes of HTML given to Readability; the rest of a bigger page is ignored. 0 turns the cap off. Parsing blocks the server while it runs.
WEBFROST_SINGLEFILE_MAX_BYTES104857600 (100 MB)Total size of the assets inlined into one singlefile.html.
WEBFROST_SINGLEFILE_CONCURRENCY8Asset downloads at once for one singlefile.html.
WEBFROST_SETTLE_MS5000Wait for the network to go quiet after the page loads, before the browser extractors capture. 0 skips it.
WEBFROST_MEDIA_MAX_HEIGHT1080Tallest video downloaded. 0 for no limit.
WEBFROST_COLOR_SCHEMElightTheme the browser archives in when the request does not choose one.

yt-dlp options

Every yt-dlp call gets --ignore-config (no config file on the host can add options), --no-playlist --playlist-items 1 (one item, never a whole playlist or channel) and --use-extractors default,-generic (only sites yt-dlp knows; the generic extractor would download and scan any page for embedded media). The URL is passed after --. One --dump-json call both detects a media page and reads its metadata.

Requirements

Chromium is needed for four extractors. Run webfrost setup to install one, or set WEBFROST_CHROMIUM_PATH to an existing binary; a system Chrome or Chromium is found on its own. The Docker image bundles it. See Install.

yt-dlp (and ffmpeg for merging streams) is needed for media. Any URL yt-dlp supports qualifies: YouTube, Vimeo, SoundCloud, Twitch clips and many more. The Docker image bundles both.

Reading the results

  • Files on disk: ls $WEBFROST_DIR/snapshots/<snapshot-id>/.
  • Download over the API: GET /api/v1/snapshots/:id/archives/:extractor/download, where :extractor is one of the names above. See Archive files.
  • Database: archive_results has one row per extractor run.

WARC files open in ReplayWeb.page and can be processed with warcio.

Supported sources

  • Single URL: any http or https URL, from webfrost add, the UI, POST /api/v1/snapshots or the SDK. Other schemes are rejected.
  • Bulk sources: text lists, Netscape bookmarks, OPML and RSS or Atom feeds. See Bulk import.
  • Recurring: a URL or an RSS feed on a cron schedule. See Scheduled archiving.
  • Not supported: pages behind a login. See Authenticated pages.

On this page