Skip to Content
WebFrostdocs
Guides

Bulk import

Import many URLs at once from text files, browser bookmarks, OPML lists or RSS and Atom feeds.

Import turns a file or feed into one snapshot per URL. Pick the format that matches what you have; if you omit the format, WebFrost detects it.

Formats

FormatValueTypical inputWhat is imported
Plain texttext.txtOne URL per line. Blank lines, lines starting with # and non-http(s) lines are ignored.
Netscape bookmarksnetscape.html, .htmEvery href of an a element. Chrome, Firefox and Safari export this format.
OPMLopml.opmlThe xmlUrl (and htmlUrl) attribute of every outline element. These are the feed or site addresses themselves.
RSS or Atomrssa feed URL or .rss, .atom, .xmlThe article link of every item or entry.

OPML imports archive the feed URLs

An OPML import does not open each feed and archive its articles. It archives the URLs listed in the file. To archive the articles of a feed, import the feed itself with the rss format, or follow it with an RSS schedule.

URLs that are not http or https are dropped, and duplicates inside one file are collapsed.

Detection when no format is given: .opml means OPML; .rss and .atom mean RSS; .xml with RSS or Atom content means RSS; .html or .htm with a Netscape bookmark header means Netscape; anything whose content looks like RSS or Atom is RSS; everything else is plain text.

Import

# Plain text, one URL per line
webfrost import urls.txt

# Browser bookmarks
webfrost import bookmarks.html

# OPML subscription list
webfrost import subscriptions.opml

# An RSS feed by URL
webfrost import --format=rss https://example.com/feed.xml

The argument is a file path or an http(s) URL. --format= accepts text, opml, netscape or rss.

The CLI waits for every pipeline to finish and ends with a summary:

Import complete: total=15, done=11, skipped=3, failed=1 (84.2s)

What happens next

The import call returns as soon as the snapshots are created as pending; the server archives them in the background. WEBFROST_CONCURRENCY (default 3) pipelines run at once, and archives of the same host are spaced by about WEBFROST_DOMAIN_DELAY_MS (default 1000 ms), so a large import of one site takes a while on purpose.

WEBFROST_CONCURRENCY=5 webfrost import urls.txt

Each pipeline can launch its own Chromium, so raise concurrency only if the machine has the memory.

Duplicates

Only rss imports skip URLs that already have a snapshot, and they count as skipped. Re-importing the same feed therefore archives only new items. Text, OPML and Netscape imports archive every URL again, each as a new snapshot. A URL that cannot be queued also counts as skipped. Every import is recorded in the imports table.

On this page