Skip to Content
WebFrostdocs
Contributing

Project structure

The monorepo layout, what each package owns, how they depend on each other, and how to add an extractor.

WebFrost is a Bun monorepo. Libraries and services are in packages/, and the documentation site is in apps/.

webfrost/
├── packages/
│   ├── core/       @webfrost/core    library
│   ├── cli/        @webfrost/cli     the webfrost command
│   ├── server/     @webfrost/server  HTTP API and static UI
│   ├── ui/         @webfrost/ui      React SPA
│   └── sdk/        @webfrost/sdk     TypeScript API client
├── apps/
│   └── docs/       @webfrost/docs    this site (React Router, Fumadocs)
├── bench/          benchmark scripts
├── .github/        CI and release workflows
├── .changeset/     pending release notes
├── package.json    workspace root scripts
├── biome.json      lint and format config
├── lefthook.yml    git hooks
└── mise.toml       tool versions and developer tasks

Packages

@webfrost/core

The shared library; nothing in it is user-facing on its own.

packages/core/src/
├── pipeline.ts          archive orchestrator for one snapshot
├── jobs.ts              JobRunner: bounded background pipelines, leases, recovery
├── scheduler.ts         polls due schedules and claims them atomically
├── schedules.ts         schedule storage (add, list, due lookup, status)
├── cron.ts              cron expression parsing and next-run calculation
├── concurrency.ts       concurrency limiter (WEBFROST_CONCURRENCY)
├── db.ts                data access (SQLite, PostgreSQL, MySQL), connections
├── database/            DATABASE_URL parsing, bun:sqlite Kysely driver, migrations
├── results.ts           archive_results writes and lease fencing
├── search-text.ts       tokenizer, query parser and snippets for PostgreSQL/MySQL search
├── fetch-cache.ts       shared per-snapshot HTTP cache
├── throttle.ts          per-host spacing (WEBFROST_DOMAIN_DELAY_MS)
├── ssrf-guard.ts        SSRF protection and safeFetch
├── browser-guard.ts     SSRF protection for browser requests
├── subprocess.ts        process runner with timeouts (yt-dlp)
├── import.ts            import parsing: text, OPML, Netscape, RSS
├── import-history.ts    import history
├── rss.ts               RSS and Atom fetching, URL deduplication
├── auth.ts              admin users, API keys, sessions
├── search.ts            full-text indexing and search (FTS5, tsvector or FULLTEXT)
├── telemetry.ts         optional OpenTelemetry export (telemetry-otel.ts holds the SDK side)
├── tags.ts              tags, kept in sync with the FTS index
├── list.ts, snapshot.ts, stats.ts   queries
├── storage/             filesystem and S3 backends
└── extractors/          warc, readability, html, pdf, screenshot,
                         dom-snapshot, singlefile, media, media-metadata

@webfrost/cli

A thin command-line layer over core, in packages/cli/src/index.ts. It parses arguments and calls core functions.

@webfrost/server

An Elysia server: the REST API routes (src/routes/), rate limiting, security headers, and static hosting of the built UI. It starts the job runner's recovery sweep and the scheduler in the same process (WEBFROST_SCHEDULER=false disables the scheduler).

@webfrost/ui

The React single-page app, built with Vite to packages/ui/dist/, which the server serves. The Docker image copies it to public/. In development Vite proxies API calls to the server. It talks to the server only through the SDK.

@webfrost/sdk

The typed REST client. It wraps fetch and has no dependency on core.

@webfrost/docs

This documentation site in apps/docs. See Development.

Dependencies

Package dependencies: @webfrost/cli imports @webfrost/core and @webfrost/server, @webfrost/server imports @webfrost/core, and @webfrost/ui imports @webfrost/sdk, which reaches the server over REST at /api/v1. @webfrost/docs depends on no other package.
Solid arrows are workspace imports; the dashed arrow is HTTP at runtime.Open interactive diagram(opens in a new tab)

The CLI and server depend on core, and the CLI also depends on the server, which webfrost serve runs inside the CLI's own process. The UI uses the SDK, and at runtime both reach the server only over HTTP. The docs site stands alone.

Add an extractor

  1. Create packages/core/src/extractors/<name>.ts exporting a function that stores its result in archive_results (a storage reference for files, or inline content).
  2. Register it in packages/core/src/pipeline.ts, in the right group: text extractors use the fetch cache, browser extractors use the page, and BROWSER_EXTRACTORS lists those recorded as failed when Chromium is unavailable. Give it a timeout and honour the abort signal.
  3. Add its name to the download allowlist and, for files, to the content type table in packages/server/src/routes/snapshots.ts. Add it to the file-backed sets there and in packages/core/src/stats.ts if it stores a file.
  4. Write tests, then update Extractors, the REST API page, README.md and ROADMAP.md.

On this page