Skip to Content
WebFrostdocs
Guides

Scheduled archiving

Re-archive a URL or follow an RSS feed on a cron schedule, run by the server or by a separate worker.

A schedule archives a URL, or the new items of a feed, whenever its cron expression fires. The server runs due schedules by itself, so schedules created in the web UI or API need no extra process.

Create a schedule

# Archive a page every day at midnight
webfrost schedule add https://example.com/news --cron "0 0 * * *"

# Follow an RSS feed every hour
webfrost schedule add --rss https://example.com/feed.xml --cron "0 * * * *"

webfrost schedule list
webfrost schedule remove 1

RSS feeds

schedule add --rss creates a feed schedule. On each run WebFrost fetches the feed, skips items that already have a snapshot, and archives the rest. The API and web UI create URL schedules only; use the CLI to add a feed schedule. It then runs in the server like any other.

A URL schedule creates a new snapshot on every run, even when the page has not changed.

Cron syntax

Five fields: minute hour day-of-month month day-of-week.

FieldValues
minute0-59
hour0-23
day-of-month1-31
month1-12 or jan-dec
day-of-week0-7 (0 and 7 are Sunday) or sun-sat

Each field supports *, a number or name, a range (1-5, mon-fri), a list (1,15) and steps (*/15, 1-10/2, 5/10). The aliases @hourly, @daily (@midnight), @weekly, @monthly and @yearly (@annually) also work. A value out of range, such as 60 * * * *, is rejected when you create the schedule (the API answers 400, the CLI prints the error) and the message names the field.

When both day-of-month and day-of-week are restricted (neither starts with *), a day matches if either one does, as in Vixie cron: 0 0 15 * 1 runs on the 15th and on every Monday. If one of them is *, only the other applies.

ExpressionMeaning
*/15 * * * *Every 15 minutes
0 * * * *Every hour
@dailyEvery day at midnight
0 9 * * mon-friWeekdays at 9:00

Cron is evaluated in the time zone of the process that runs the schedule: TZ, or the system time zone, which is UTC in the Docker image. Set TZ=America/Sao_Paulo to change it. In a zone with daylight saving time, a time that does not exist on the day the clocks jump forward (for example 30 2 * * * in Europe/Berlin on the last Sunday of March) is skipped for that day, and a time that happens twice when the clocks go back runs once, the first time. Hourly and more frequent schedules are not affected. Use UTC if every run must happen.

Who runs the schedules

The server starts a scheduler at boot and polls every 30 seconds. Set WEBFROST_SCHEDULER=false to turn that off.

Without the server (CLI-only use), run a worker as a long-lived process under systemd, Docker or a process manager:

WEBFROST_DIR=$HOME/webfrost-data webfrost schedule worker

The worker uses the same scheduler as the server, runs pipelines with the same concurrency limit, and re-queues snapshots left pending by an interrupted run.

Running a worker next to the server on one archive is safe: a due schedule is claimed atomically, so only one process runs it. Set WEBFROST_SCHEDULER=false on the server if the worker should be the only one. See Deploy.

Failures

Each schedule keeps lastStatus (success or failed) and consecutiveFailures, which resets on success.

  • A URL schedule is success when its snapshot ends done, and failed otherwise.
  • A feed schedule is success when the feed could be read. Failures of individual articles show on their own snapshots.

Failing schedules stay active and retry on the next run. There is no automatic disabling; pause a schedule yourself to stop retries. If a run is still going when the next one is due, the next one waits until it finishes.

On this page