Skip to Content
WebFrostdocs
Contributing

Development

Set up a WebFrost dev environment, run the tests, lint and typecheck, and work on this documentation site.

Prerequisites

ToolPurpose
BunRuntime and package manager. The version is pinned in mise.toml, which CI reads too. Run mise install to get it.
ChromiumThe browser extractors and some tests. Installed by Playwright, or use your own with WEBFROST_CHROMIUM_PATH.
yt-dlp, ffmpegMedia extraction. Optional, and needed by the media end-to-end tests.

Set up

git clone https://github.com/gtkatakura/webfrost.git
cd webfrost
mise run setup    # bun install + bunx playwright install --with-deps chromium

Already have a Chromium? Skip the Playwright download and point WebFrost at it. This also applies to the tests:

export WEBFROST_CHROMIUM_PATH=/usr/bin/chromium

Run the dev stack

mise run dev

Starts the API on port 3000 with AUTH_DISABLED=true and WEBFROST_DIR=./data, and the Vite UI dev server on port 5173, which proxies API calls to the server. The same by hand:

# terminal 1
WEBFROST_DIR=./data AUTH_DISABLED=true bun run packages/server/src/index.ts
# terminal 2
cd packages/ui && bun run dev

Run mise tasks for every task.

Run the CLI from source

export WEBFROST_DIR=./data   # default: ~/.webfrost
bun run webfrost --help
bun run webfrost add https://example.com
bun run webfrost serve       # API, UI (needs `bun run build` once) and scheduler

bun run webfrost <command> is the same program as the installed webfrost binary. Use mise exec -- bun run webfrost ... when your shell's bun is older than the pinned version.

Build

bun run build     # builds every package

The single-file binary

mise run build:binary                     # for this machine: packages/cli/dist-binary/<os>-<arch>/webfrost
bun run build:binary --target all         # darwin-arm64, darwin-x64, linux-x64, linux-arm64
bun run build:binary --version 1.2.3      # the version `webfrost --version` reports

It builds the SDK and the UI, then compiles packages/cli with bun build --compile (packages/cli/scripts/build-binary.ts). Bun cross-compiles, so every target builds on one machine. The binary is 90 to 120 MB (macOS arm64 about 94 MB, Linux x64 about 112 MB, Windows x64 about 118 MB); each release archive is 41 to 54 MB. The build script embeds the UI and works around two jsdom details that a bundle cannot resolve (the default stylesheet it reads from disk, and css-tree's data files); if a dependency upgrade breaks it, it fails with a message naming the file. bun run packages/cli/scripts/package-release.ts --outdir <dir> makes the archives and checksums.txt the release workflow publishes.

To try the installer against local files, serve a directory that holds the archives and checksums.txt and point the script at it:

WEBFROST_DOWNLOAD_BASE=http://127.0.0.1:8765 WEBFROST_INSTALL_DIR=/tmp/wf-bin sh apps/docs/public/install.sh

Tests

bun run test                              # unit and integration tests, every package
cd packages/server && bun run test:e2e    # HTTP API end to end
cd packages/ui && bunx playwright test    # browser end to end (needs Chromium and yt-dlp); same as mise run test:e2e

Notes:

  • Browser tests. The tests that drive a real Chromium (for example packages/core/src/browser-guard.test.ts) are skipped, not failed, when no Chromium is found. Run them at least once if you change the pipeline or the guard: install Chromium with Playwright, or set WEBFROST_CHROMIUM_PATH=/usr/bin/chromium.
  • S3 integration tests run only when MINIO_TEST_ENDPOINT is set (also MINIO_TEST_BUCKET, MINIO_TEST_ACCESS_KEY, MINIO_TEST_SECRET_KEY). docker compose -f docker-compose.dev.yml up -d starts MinIO with a webfrost bucket.
  • PostgreSQL and MySQL tests run the same suites against another engine when WEBFROST_TEST_DATABASE_URL is set to a postgres:// or mysql:// URL (CI does this for both). The account needs CREATE DATABASE: every test archive gets its own database.
  • UI end to end tests run with AUTH_DISABLED=true in CI.
  • Assert the final state (snapshot status, stored files), not only the response: bugs here have mostly been of the "returns 202 but nothing happens afterwards" kind.

Lint, format and typecheck

bun run lint        # Biome check
bun run format      # Biome format, writes files
bun run typecheck   # tsc --noEmit for the packages, then the docs site
mise run ci         # lint + typecheck + test, like CI

Commit hooks

Lefthook runs on every commit, installed by bun install through the prepare script:

  • lint: biome check --staged on staged js, ts, jsx, tsx, json and css files;
  • typecheck: bun run typecheck.

Fix what they report; do not skip them with --no-verify.

Benchmarks

bun run bench                  # microbenchmarks
mise run bench:vs-archivebox   # head-to-head comparison (starts Docker services)

Monorepo conventions

  • Run a script in every package with bun run --filter '*' <script>.
  • New packages go in packages/<name>/. Apps (like the docs site) go in apps/<name>/.
  • Shared logic belongs in @webfrost/core. Keep the CLI, server and SDK thin.
  • The reference pages in this site must match what the code does: new CLI commands, routes, variables and extractors go there. README.md changes only with the quickstart or the big picture; a shipped ROADMAP.md item moves to "Already there".
  • Add a changeset (bun changeset) for anything users will notice. See Releasing.

This documentation site

The docs are in apps/docs: React Router and Vite with Fumadocs, and MDX pages prerendered to static files.

bun run docs:dev       # dev server with hot reload, http://localhost:5174
bun run docs:build     # prerender to apps/docs/build/client
bun run docs:preview   # serve the build, http://localhost:5175

Add a page

Create apps/docs/content/docs/<section>/<name>.mdx with frontmatter:

---
title: Short title
description: One sentence saying what the reader gets from this page.
icon: LucideIconName
---

Start with the point. No H1: the title comes from the frontmatter.

Add the file name (without extension) to pages in that section's meta.json. The order there is the order in the sidebar. A page not listed in meta.json is still built but is not in the navigation.

Run bun run docs:build. It prerenders every page and fails on invalid MDX. Link to other pages with absolute site paths such as /docs/reference/cli.

Components available without an import: Callout (type is info, warn or error), Steps and Step, Tabs and Tab, Cards and Card, Picture for images and ArchDiagram for diagrams (see below).

MDX pitfalls

In prose, write a bare < as &lt; or put it in backticks, and do the same for {. Comments are {/* */}, not <!-- -->. Inside fenced code blocks anything goes.

Add an image

Put the original, as large as you have it, in apps/docs/app/assets/. Import it with a preset and draw it with Picture:

import shot from "@/assets/screens/search.png?preset=content";

<Picture image={shot} sizes="(min-width: 1024px) 720px, 100vw" alt="Search results with highlighted matches" />

A screenshot of the web UI comes in both themes. Keep the two masters in app/assets/screens/ (<name>-light.webp and <name>-dark.webp, captured at 1440×900 with a device scale factor of 2) and draw them with Screenshot, which shows the one matching the site's theme:

import searchLight from "@/assets/screens/search-light.webp?preset=content";
import searchDark from "@/assets/screens/search-dark.webp?preset=content";

<Screenshot light={searchLight} dark={searchDark} alt="Search results with highlighted matches" />

The README shows the same view through .github/assets/screenshot-light.webp and -dark.webp (1280 px wide, WebP quality 82). Capture against a fresh data directory with a few real public pages archived, never with private URLs or usernames other than admin.

The build makes every width in AVIF, WebP and JPG plus a blurred placeholder, and the browser downloads only the one that fits the screen. sizes tells it how wide the image is drawn. The presets are in apps/docs/image-presets.ts:

PresetFor
contentAn image in the text column, such as a screenshot
heroThe large image at the top of a page, with a 4:3 crop for phones
ogThe link preview for social sites (a URL, 1200×630)
touch-iconThe iOS home-screen icon (a URL, 180×180)

After changing the mascot art in app/assets/brand, run mise run brand. It rewrites the images that are not built with the site: the link-preview master, the README banner, the GitHub social preview and the web app's favicon.

Diagrams

The architecture diagrams are made with Archify, which renders a JSON description of a diagram (architecture, sequence, lifecycle, data flow, ERD, tree…) to a checked, standalone HTML viewer. Each diagram has its source in apps/docs/diagrams-src/<name>/candidate.json: its type, nodes and relationships, and the lines of code each node is drawn from, pinned to a commit in meta.repository.revision.

To change one, or add one:

Install the Archify skill, which carries its CLI (Node.js 18 or later):

npx skills add tt-a1i/archify

Below, <archify> is the folder the skill was installed in. A coding agent with the skill loaded can write and repair the candidate for you; its SKILL.md and references/ describe the format.

Edit apps/docs/diagrams-src/<name>/candidate.json and check it against the code. When it cites source lines, point meta.repository.revision at a commit that contains them. Then render and check it from the repository root:

node <archify>/bin/archify.mjs finalize <type> \
  apps/docs/diagrams-src/<name>/candidate.json \
  apps/docs/diagrams-src/<name>/<name>.html \
  --repo-root . --quality showcase --json

<type> is the candidate's diagram_type. The command validates the layout and the cited sources, writes the HTML and opens it in a headless Chrome or Chromium (set ARCHIFY_CHROME to its path if it is not found). A failure names what to fix; fix the candidate and run it again.

Publish it to the site:

mise run docs:diagrams           # every diagram
bun apps/docs/scripts/diagrams.ts <name>   # or one

This exports the light and dark SVG through the viewer's own export, copies the HTML viewer to apps/docs/public/diagrams/ and records the sizes in apps/docs/app/lib/diagrams.json. It uses Playwright's Chromium, or the one in WEBFROST_CHROMIUM_PATH.

Show it on a page:

<ArchDiagram name="system-overview" alt="What the diagram shows, in a sentence or two" caption="One line under it" />

It draws the variant that matches the site's theme, reserves its space before it loads, loads lazily, and links to the interactive viewer.

Commit candidate.json, the files in public/diagrams/ and app/lib/diagrams.json. Archify's receipts next to the candidate are ignored.

The docs site as a container

apps/docs/Dockerfile builds the site into an image: it prerenders every page and resizes the images at build time, and what runs is a small Bun server that hands out the files on port 3000 (health check at /up). Pages and their route data are cacheable for a minute; files under /assets for a year, because their names change when their content does.

docker build -f apps/docs/Dockerfile -t webfrost-docs .
docker run --rm -p 3000:3000 webfrost-docs

On this page