Development
Set up a WebFrost dev environment, run the tests, lint and typecheck, and work on this documentation site.
Prerequisites
| Tool | Purpose |
|---|---|
| Bun | Runtime and package manager. The version is pinned in mise.toml, which CI reads too. Run mise install to get it. |
| Chromium | The browser extractors and some tests. Installed by Playwright, or use your own with WEBFROST_CHROMIUM_PATH. |
| yt-dlp, ffmpeg | Media extraction. Optional, and needed by the media end-to-end tests. |
Set up
git clone https://github.com/gtkatakura/webfrost.git
cd webfrost
mise run setup # bun install + bunx playwright install --with-deps chromiumAlready have a Chromium? Skip the Playwright download and point WebFrost at it. This also applies to the tests:
export WEBFROST_CHROMIUM_PATH=/usr/bin/chromiumRun the dev stack
mise run devStarts the API on port 3000 with AUTH_DISABLED=true and WEBFROST_DIR=./data, and the Vite UI dev server on port 5173, which proxies API calls to the server. The same by hand:
# terminal 1
WEBFROST_DIR=./data AUTH_DISABLED=true bun run packages/server/src/index.ts
# terminal 2
cd packages/ui && bun run devRun mise tasks for every task.
Run the CLI from source
export WEBFROST_DIR=./data # default: ~/.webfrost
bun run webfrost --help
bun run webfrost add https://example.com
bun run webfrost serve # API, UI (needs `bun run build` once) and schedulerbun run webfrost <command> is the same program as the installed webfrost binary. Use mise exec -- bun run webfrost ... when your shell's bun is older than the pinned version.
Build
bun run build # builds every packageThe single-file binary
mise run build:binary # for this machine: packages/cli/dist-binary/<os>-<arch>/webfrost
bun run build:binary --target all # darwin-arm64, darwin-x64, linux-x64, linux-arm64
bun run build:binary --version 1.2.3 # the version `webfrost --version` reportsIt builds the SDK and the UI, then compiles packages/cli with bun build --compile (packages/cli/scripts/build-binary.ts). Bun cross-compiles, so every target builds on one machine. The binary is 90 to 120 MB (macOS arm64 about 94 MB, Linux x64 about 112 MB, Windows x64 about 118 MB); each release archive is 41 to 54 MB. The build script embeds the UI and works around two jsdom details that a bundle cannot resolve (the default stylesheet it reads from disk, and css-tree's data files); if a dependency upgrade breaks it, it fails with a message naming the file. bun run packages/cli/scripts/package-release.ts --outdir <dir> makes the archives and checksums.txt the release workflow publishes.
To try the installer against local files, serve a directory that holds the archives and checksums.txt and point the script at it:
WEBFROST_DOWNLOAD_BASE=http://127.0.0.1:8765 WEBFROST_INSTALL_DIR=/tmp/wf-bin sh apps/docs/public/install.shTests
bun run test # unit and integration tests, every package
cd packages/server && bun run test:e2e # HTTP API end to end
cd packages/ui && bunx playwright test # browser end to end (needs Chromium and yt-dlp); same as mise run test:e2eNotes:
- Browser tests. The tests that drive a real Chromium (for example
packages/core/src/browser-guard.test.ts) are skipped, not failed, when no Chromium is found. Run them at least once if you change the pipeline or the guard: install Chromium with Playwright, or setWEBFROST_CHROMIUM_PATH=/usr/bin/chromium. - S3 integration tests run only when
MINIO_TEST_ENDPOINTis set (alsoMINIO_TEST_BUCKET,MINIO_TEST_ACCESS_KEY,MINIO_TEST_SECRET_KEY).docker compose -f docker-compose.dev.yml up -dstarts MinIO with awebfrostbucket. - PostgreSQL and MySQL tests run the same suites against another engine when
WEBFROST_TEST_DATABASE_URLis set to apostgres://ormysql://URL (CI does this for both). The account needsCREATE DATABASE: every test archive gets its own database. - UI end to end tests run with
AUTH_DISABLED=truein CI. - Assert the final state (snapshot status, stored files), not only the response: bugs here have mostly been of the "returns 202 but nothing happens afterwards" kind.
Lint, format and typecheck
bun run lint # Biome check
bun run format # Biome format, writes files
bun run typecheck # tsc --noEmit for the packages, then the docs site
mise run ci # lint + typecheck + test, like CICommit hooks
Lefthook runs on every commit, installed by bun install through the prepare script:
- lint:
biome check --stagedon stagedjs,ts,jsx,tsx,jsonandcssfiles; - typecheck:
bun run typecheck.
Fix what they report; do not skip them with --no-verify.
Benchmarks
bun run bench # microbenchmarks
mise run bench:vs-archivebox # head-to-head comparison (starts Docker services)Monorepo conventions
- Run a script in every package with
bun run --filter '*' <script>. - New packages go in
packages/<name>/. Apps (like the docs site) go inapps/<name>/. - Shared logic belongs in
@webfrost/core. Keep the CLI, server and SDK thin. - The reference pages in this site must match what the code does: new CLI commands, routes, variables and extractors go there.
README.mdchanges only with the quickstart or the big picture; a shippedROADMAP.mditem moves to "Already there". - Add a changeset (
bun changeset) for anything users will notice. See Releasing.
This documentation site
The docs are in apps/docs: React Router and Vite with Fumadocs, and MDX pages prerendered to static files.
bun run docs:dev # dev server with hot reload, http://localhost:5174
bun run docs:build # prerender to apps/docs/build/client
bun run docs:preview # serve the build, http://localhost:5175Add a page
Create apps/docs/content/docs/<section>/<name>.mdx with frontmatter:
---
title: Short title
description: One sentence saying what the reader gets from this page.
icon: LucideIconName
---
Start with the point. No H1: the title comes from the frontmatter.Add the file name (without extension) to pages in that section's meta.json. The order there is the order in the sidebar. A page not listed in meta.json is still built but is not in the navigation.
Run bun run docs:build. It prerenders every page and fails on invalid MDX. Link to other pages with absolute site paths such as /docs/reference/cli.
Components available without an import: Callout (type is info, warn or error), Steps and Step, Tabs and Tab, Cards and Card, Picture for images and ArchDiagram for diagrams (see below).
MDX pitfalls
In prose, write a bare < as < or put it in backticks, and do the same for {. Comments are {/* */}, not <!-- -->. Inside fenced code blocks anything goes.
Add an image
Put the original, as large as you have it, in apps/docs/app/assets/. Import it with a preset and draw it with Picture:
import shot from "@/assets/screens/search.png?preset=content";
<Picture image={shot} sizes="(min-width: 1024px) 720px, 100vw" alt="Search results with highlighted matches" />A screenshot of the web UI comes in both themes. Keep the two masters in app/assets/screens/ (<name>-light.webp and <name>-dark.webp, captured at 1440×900 with a device scale factor of 2) and draw them with Screenshot, which shows the one matching the site's theme:
import searchLight from "@/assets/screens/search-light.webp?preset=content";
import searchDark from "@/assets/screens/search-dark.webp?preset=content";
<Screenshot light={searchLight} dark={searchDark} alt="Search results with highlighted matches" />The README shows the same view through .github/assets/screenshot-light.webp and -dark.webp (1280 px wide, WebP quality 82). Capture against a fresh data directory with a few real public pages archived, never with private URLs or usernames other than admin.
The build makes every width in AVIF, WebP and JPG plus a blurred placeholder, and the browser downloads only the one that fits the screen. sizes tells it how wide the image is drawn. The presets are in apps/docs/image-presets.ts:
| Preset | For |
|---|---|
content | An image in the text column, such as a screenshot |
hero | The large image at the top of a page, with a 4:3 crop for phones |
og | The link preview for social sites (a URL, 1200×630) |
touch-icon | The iOS home-screen icon (a URL, 180×180) |
After changing the mascot art in app/assets/brand, run mise run brand. It rewrites the images that are not built with the site: the link-preview master, the README banner, the GitHub social preview and the web app's favicon.
Diagrams
The architecture diagrams are made with Archify, which renders a JSON description of a diagram (architecture, sequence, lifecycle, data flow, ERD, tree…) to a checked, standalone HTML viewer. Each diagram has its source in apps/docs/diagrams-src/<name>/candidate.json: its type, nodes and relationships, and the lines of code each node is drawn from, pinned to a commit in meta.repository.revision.
To change one, or add one:
Install the Archify skill, which carries its CLI (Node.js 18 or later):
npx skills add tt-a1i/archifyBelow, <archify> is the folder the skill was installed in. A coding agent with the skill loaded can write and repair the candidate for you; its SKILL.md and references/ describe the format.
Edit apps/docs/diagrams-src/<name>/candidate.json and check it against the code. When it cites source lines, point meta.repository.revision at a commit that contains them. Then render and check it from the repository root:
node <archify>/bin/archify.mjs finalize <type> \
apps/docs/diagrams-src/<name>/candidate.json \
apps/docs/diagrams-src/<name>/<name>.html \
--repo-root . --quality showcase --json<type> is the candidate's diagram_type. The command validates the layout and the cited sources, writes the HTML and opens it in a headless Chrome or Chromium (set ARCHIFY_CHROME to its path if it is not found). A failure names what to fix; fix the candidate and run it again.
Publish it to the site:
mise run docs:diagrams # every diagram
bun apps/docs/scripts/diagrams.ts <name> # or oneThis exports the light and dark SVG through the viewer's own export, copies the HTML viewer to apps/docs/public/diagrams/ and records the sizes in apps/docs/app/lib/diagrams.json. It uses Playwright's Chromium, or the one in WEBFROST_CHROMIUM_PATH.
Show it on a page:
<ArchDiagram name="system-overview" alt="What the diagram shows, in a sentence or two" caption="One line under it" />It draws the variant that matches the site's theme, reserves its space before it loads, loads lazily, and links to the interactive viewer.
Commit candidate.json, the files in public/diagrams/ and app/lib/diagrams.json. Archify's receipts next to the candidate are ignored.
Project structure
Where the code lives and how to add an extractor.
Releasing
Changesets, versions and the release workflow.
The docs site as a container
apps/docs/Dockerfile builds the site into an image: it prerenders every page and resizes the images at
build time, and what runs is a small Bun server that hands out the files on port 3000 (health check at
/up). Pages and their route data are cacheable for a minute; files under /assets for a year, because
their names change when their content does.
docker build -f apps/docs/Dockerfile -t webfrost-docs .
docker run --rm -p 3000:3000 webfrost-docs