Skip to Content
WebFrostdocs
Guides

Observability

Send traces, metrics and logs to your own OpenTelemetry collector, Jaeger or Grafana stack. Off by default; nothing leaves the process unless you configure it.

WebFrost collects no telemetry. It can, if you ask it to, export traces, metrics and logs over OTLP to a collector you run: an OpenTelemetry Collector, Jaeger, Grafana Tempo/Mimir/Loki, or any OTLP-compatible backend. Export is configured with the standard OTEL_* variables and works the same in the Docker image, webfrost serve and the other CLI commands.

Without an OTLP endpoint in the environment, the OpenTelemetry SDK is not even loaded: no extra memory, no background timers, nothing recorded and nothing sent. The instrumentation points are calls into an empty function.

Turn it on

Set an OTLP/HTTP endpoint. Everything else is optional.

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_SERVICE_NAME=webfrost              # default: webfrost

With the endpoint above, traces go to /v1/traces and metrics to /v1/metrics. Logs are exported only when you also set OTEL_LOGS_EXPORTER=otlp. On shutdown (SIGTERM, Ctrl+C, the end of a CLI command) whatever is still buffered is sent, for at most 5 seconds.

VariableDefaultDescription
OTEL_EXPORTER_OTLP_ENDPOINTnoneBase URL of an OTLP/HTTP receiver, usually port 4318. Setting it (or one of the per-signal endpoints) is what turns export on.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_LOGS_ENDPOINTnoneFull URL for one signal (including /v1/traces etc.). A signal with no endpoint of its own and no shared endpoint is not exported.
OTEL_EXPORTER_OTLP_HEADERSnoneHeaders for every export request, key=value pairs separated by commas, e.g. authorization=Bearer%20<token>. Per-signal OTEL_EXPORTER_OTLP_TRACES_HEADERS etc. also work.
OTEL_EXPORTER_OTLP_PROTOCOLhttp/protobufOnly http/protobuf is supported. grpc and http/json turn the signal off with a warning: point WebFrost at the collector's HTTP port (4318) instead.
OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_COMPRESSION10000, noneExport timeout in ms; gzip to compress. OTEL_EXPORTER_OTLP_CERTIFICATE and the client certificate variables work too.
OTEL_SERVICE_NAMEwebfrostservice.name of everything WebFrost exports. Run several instances? Give each its own name or a service.instance.id.
OTEL_RESOURCE_ATTRIBUTESnoneExtra resource attributes, e.g. deployment.environment.name=home,service.instance.id=nas.
OTEL_TRACES_SAMPLERparentbased_always_onalways_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off or parentbased_traceidratio.
OTEL_TRACES_SAMPLER_ARG1The ratio for the traceidratio samplers, between 0 and 1.
OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTERotlpnone stops exporting that signal.
OTEL_LOGS_EXPORTERnoneotlp exports log lines as well.
OTEL_METRIC_EXPORT_INTERVAL60000How often metrics are sent, in ms.
OTEL_SDK_DISABLEDfalsetrue turns export off even with an endpoint set.
WEBFROST_OTEL_FULL_URLSfalsetrue records the full URL of archived pages (url.full) on spans. See Privacy.

What is exported

Traces

One trace per piece of work. A snapshot queued by an API request is part of that request's trace, even though it runs later in the background; the same goes for a schedule tick and the recovery sweep.

SpanKindAttributes
GET /api/v1/snapshots/:id (method and route template)serverhttp.request.method, http.route, http.response.status_code, url.scheme. An incoming W3C traceparent header is continued.
job_queue.enqueueproducerwebfrost.snapshot.id
snapshot.archive: one pipeline runinternalwebfrost.snapshot.id, server.address (host of the page), webfrost.extractors (the formats run), webfrost.snapshot.status (done or failed), webfrost.browser.failure, error.type when the run failed (fetch_network, fetch_http_status, no_output...)
page.fetch: the one HTTP fetchclienthttp.request.method, server.address, http.response.status_code, http.response.body.size, webfrost.fetch.redirects, webfrost.fetch.text
browser.navigate: Chromium launch and the one navigationinternalserver.address, error.type on failure (timeout, blocked, network...)
extractor.<format> (extractor.pdf, extractor.warc...)internalwebfrost.extractor.format, webfrost.extractor.outcome (ok, failed, timeout), webfrost.extractor.bytes, webfrost.extractor.failure (a short category, not the message)
media.yt-dlp: each yt-dlp runinternalwebfrost.media.step (metadata or download), process.exit.code
throttle.wait: waiting for the per-host politeness slotinternalserver.address, webfrost.throttle.wait_ms
job_queue.recover: the sweep for jobs left behind by a crash or restartinternalwebfrost.jobs.adopted, webfrost.jobs.leases_recovered, webfrost.jobs.abandoned
scheduler.tickinternalwebfrost.schedules.claimed
SELECT snapshots, INSERT archive_results...: database statementsclientdb.system.name (sqlite, postgresql, mysql), db.operation.name, db.collection.name

Database statements are traced only inside another span, so lease heartbeats do not each start a trace of their own. WebFrost does not add trace headers to the pages it archives: a site never learns your trace IDs.

Metrics

MetricTypeUnitAttributes
webfrost.snapshots.finishedcounter{snapshot}webfrost.snapshot.status (done, failed)
webfrost.extractor.durationhistogramswebfrost.extractor.format, webfrost.extractor.outcome (ok, failed, timeout, skipped)
webfrost.archived.bytescounterBywebfrost.extractor.format
webfrost.jobsgauge{job}webfrost.job.state (pending: queued in this process; running)
webfrost.jobs.retriescounter{job}Jobs run again after an interrupted attempt
webfrost.jobs.leases_recoveredcounter{job}Jobs adopted from a stale lease (another process crashed or stalled)
webfrost.jobs.abandonedcounter{job}Jobs given up on after three interrupted attempts
webfrost.throttle.wait.durationhistogramsTime a run waited for its host's politeness slot (WEBFROST_DOMAIN_DELAY_MS)
webfrost.db.transaction.retriescounter{retry}db.system.name: transactions retried after a deadlock or a busy SQLite file
http.server.request.durationhistogramshttp.request.method, http.route, http.response.status_code, url.scheme
db.client.operation.durationhistogramsdb.system.name, db.operation.name, db.collection.name

Logs

When a span is active, every JSON log line of the server carries trace_id, span_id and trace_flags, so a log search leads straight to the trace. This needs only the endpoint, not log export.

With OTEL_LOGS_EXPORTER=otlp the log lines (server and CLI) are also sent over OTLP, with their level, message and scalar fields, linked to the active span. They still go to stdout as before.

Privacy

WebFrost exports to the endpoint you configure and nowhere else. What it records is chosen to be useful without leaking what you archive or who uses it:

  • Recorded: snapshot IDs, the host of archived pages, formats, outcomes, sizes, durations, HTTP methods, route templates and status codes, database operation and table names, short failure categories (timeout, blocked, network...).
  • Not recorded: full URLs and paths of archived pages (unless WEBFROST_OTEL_FULL_URLS=true), query strings, concrete request paths (/api/v1/snapshots/42 is recorded as /api/v1/snapshots/:id), request and response headers (no Authorization, cookies or API keys), request bodies, page content, SQL statements and their parameters, client IP addresses, user agents, error messages (they can quote a URL; only the category is kept).
  • Logs exported with OTEL_LOGS_EXPORTER=otlp are the same lines the server already writes to stdout. They can contain URLs of archived pages: leave log export off if your collector should not see them.

Examples

OpenTelemetry Collector

Send everything to a collector, which forwards it wherever you like. A minimal otel-collector.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch: {}
exporters:
  debug: {}          # replace with otlphttp, prometheusremotewrite, loki...
service:
  pipelines:
    traces: { receivers: [otlp], processors: [batch], exporters: [debug] }
    metrics: { receivers: [otlp], processors: [batch], exporters: [debug] }
    logs: { receivers: [otlp], processors: [batch], exporters: [debug] }

Next to WebFrost in Compose. compose.yml carries the main OTEL_* variables as commented-out lines under environment: uncomment the ones you use and put their values in .env. Add any other variable from the table above to the same block.

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.164.0   # pin a digest in production
    volumes:
      - ./otel-collector.yaml:/etc/otelcol-contrib/config.yaml:ro
# .env
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_LOGS_EXPORTER=otlp

Jaeger

Jaeger accepts OTLP directly and is the quickest way to look at traces:

docker run -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/jaeger:2.23.0
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces webfrost serve

Archive a page and open http://localhost:16686: service webfrost, operation snapshot.archive. Jaeger stores traces only, which is why the example sets the traces endpoint rather than the shared one.

Grafana (Tempo, Mimir or Prometheus, Loki)

Grafana's stack takes OTLP through Grafana Alloy or an OpenTelemetry Collector, or directly with Grafana Cloud's OTLP gateway:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-<region>.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=authorization=Basic%20<base64 of instance-id:token>
OTEL_LOGS_EXPORTER=otlp

Useful panels: rate(webfrost_snapshots_finished_total[5m]) by status, the p95 of webfrost_extractor_duration_seconds by format, webfrost_jobs by state, and http_server_request_duration_seconds by route. (Prometheus-style names; the exact spelling depends on how your backend translates OTLP metric names.)

Troubleshooting

  • Nothing arrives. Check that the endpoint is the HTTP port (4318), not gRPC (4317), and that the WebFrost container can resolve the collector's hostname. OTEL_SDK_DISABLED=true overrides everything.
  • OTEL_EXPORTER_OTLP_PROTOCOL=grpc is not supported in the log: remove the variable or set it to http/protobuf, and use port 4318.
  • Too many traces. Sample them: OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1 keeps one in ten. Metrics are not sampled.

On this page