Go/job-search
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Added the HTTP server and serve command, with proxy, CSRF, TLS and security header config; wsserve is now ws serve, alongside a new ws dial

Browse Source 
The serve command runs internal/server: the API under /api (health, and
the CSRF token endpoint) and the WebSocket hub under /ws, over HTTP or TLS,
with an optional HTTP to HTTPS redirect, the PROXY protocol from trusted
sources and a graceful shutdown that closes WebSockets first. Its middleware
covers the client IP behind trusted proxies, request IDs, request logging,
panic recovery, global and API rate limits, Host and body size checks,
security response headers, compression, CSRF protection (cross-origin
checks and signed double-submit tokens) and JSON-only API bodies. Middleware
that logs takes a WithLogger option.

New server config sections: proxy, csrf, tls, headers and frontend (not
used yet), plus max_header_bytes, max_body_bytes, global and API rate limits
and HTTP read, write and idle timeouts. Every setting documents what 0 or
empty means, and internal/config/README.md is reorganised by section.
server.timeouts.read_header must be greater than 0, and a header or idle
timeout of 0 means no limit rather than the read timeout.

WebSocket upgrades are protected by the Origin check: the server's own
origin, or one of csrf.cross_origin_protection.trusted_origins, which the
hub now takes with WithTrustedOrigins.

wsserve becomes "ws serve". "ws dial" opens a WebSocket on /ws to test it:
--ping sends a ping and waits for the pong, --message sends messages and
prints the replies, --origin tests the Origin check, and with neither
--ping nor --message each line of stdin is sent until Ctrl+C.
This commit is contained in:
bwilliams committed 2026-10-15 17:47:40 +01:00
1 parent f3a91ad915
commit a279405cae
41 files changed
+3876 -78
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

internal/config/README.md
+222 -35
View File
@@ -23,47 +23,195 @@ environment variables, merged through [Viper](https://github.com/spf13/viper).
previous value is kept.
When more than one place sets a key, environment variables win over the
config file, which wins over the defaults. A `debug` or `error` log level
already set on `conf` before `Init` (from the `--debug` and `--quiet` flags)
is kept.
config file, which wins over the defaults.
## Keys
## Configuration Keys
Configuration is managed in sections:
| Key | Section description |
|---|---|
| `logging` | Paths to log files and log level control |
| `database` | Database connection/file details |
| `sources` | Opportunity source settings |
| `server` | Server configuration including CSRF protection, Websockets, Proxy handling and rate limits |
### Logging
| Key | Type | Default | Notes |
|---|---|---|---|
| `logging.level` | string | `warn` | One of `debug`, `info`, `warn`, `error` |
| `logging.file` | string | | Log file for `fetch service` (unless `--foreground`); other commands log to the terminal. Empty: log to the terminal |
| `database.file` | string | | Path to the SQLite database file, used by the `migrate` commands. Empty: commands that need the database fail |
| `logging.file` | string | | Log file (JSON lines) for long-running commands such as `fetch service`. Empty: log to the terminal |
Logging always starts on the terminal. Long-running commands switch to the file
in `logging.file` once the configuration is loaded, unless it is empty (they
then stay on the terminal, with a warning) or `--foreground` is passed.
All other commands log to the terminal.
The `--debug` and `--quiet` flags force the log level to `debug` and `error`
respectively, ignoring the `logging.level` configuration. `--debug` takes
priority over `--quiet`.
### Database
| Key | Type | Default | Notes |
|---|---|---|---|
| `database.file` | string | | Path to the SQLite database file. Empty: commands that need the database fail |
Database is currently restricted to SQLite v3. At this point, no other engine is
supported.
### Opportunity Sources
| Key | Type | Default | Notes |
|---|---|---|---|
| `sources.<source>.enabled` | bool | `false` | For every source |
| `sources.<source>.interval` | duration | `10m` | For every source; e.g. `30s`, `1h`. Must be greater than 0; turn a source off with `enabled` |
| `sources.find_a_tender.api_root` | string | `https://www.find-tender.service.gov.uk/api/1.0` | URL of the Find a Tender API. Empty: the source fails to start if enabled, and `fetch once` fails |
| `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`). Empty: the source fails to start if enabled |
| `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown |
| `server.host` | string | `127.0.0.1` | Address to listen on; empty: every interface |
| `server.port` | int | `9393` | Port to listen on; `0`: any free port |
| `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for clients to close; `0s`: until they have |
| `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's rate limiter is kept after its last request; `0s`: for ever |
| `server.websocket.enable_compression` | bool | `false` | permessage-deflate |
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.*.enabled` | bool | `false` | In a rate limit rule: whether it applies |
| `server.websocket.*.per_second` | int | `50` | In a rate limit rule: sustained rate. `0` or less: the rule is off |
| `server.websocket.*.burst` | int | `100` | In a rate limit rule: largest burst. Below `1`: 1 |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
The current sources are `find_a_tender`, `fts_mock` and `sleeper`. See
"Addition of a new opportunities source" in the top level README for how to add
a new source.
### Server
The `server` section is split into a number of sub-sections:
| Key | Section description |
|---|---|
| `server` | Top level server configuration settings |
| `server.timeouts` | HTTP read, write and idle timeouts, and the shutdown timeout |
| `server.rate_limits` | Global (every request) and API rate limits. Also see `server.websocket.connection_limit` for WebSocket and plain TCP connection limits |
| `server.proxy` | PROXY protocol and proxy header settings |
| `server.csrf` | CSRF and cross-origin protection settings |
| `server.tls` | TLS certificate, minimum version and HTTP redirect |
| `server.headers` | HTTP header settings including policy headers |
| `server.frontend` | How the frontend is served (not used yet) |
| `server.websocket` | Websocket configuration |
#### Server
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.host` | string | `127.0.0.1` | Address to listen on; empty: every interface |
| `server.port` | int | `9393` | Port to listen on; `0`: any free port |
| `server.max_header_bytes` | byte size | `8KB` | Largest request header block; `0B` for 8KB. `-1` (no limit) is rejected, as the HTTP server always has one |
| `server.max_body_bytes` | byte size | `1MB` | Largest request body; `0B` for 1MB, `-1` for no limit |
#### Timeouts
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.timeouts.read_header` | duration | `5s` | Reading a request's headers. Must be greater than 0, as a client could otherwise hold a connection open for ever |
| `server.timeouts.read` | duration | `15s` | Reading a whole request; `0s` for none |
| `server.timeouts.write` | duration | `30s` | Writing a response; `0s` for none |
| `server.timeouts.idle` | duration | `2m` | Waiting for the next request on a keep-alive connection; `0s` for none |
| `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for open requests; `0s`: until they finish |
#### Rate Limits
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.rate_limits.<rule>.enabled` | bool | `false` | `<rule>` is `global` or `api` (see rate limit rules below) |
| `server.rate_limits.<rule>.per_second` | int | `50` | Sustained requests per second per client IP. `0` or less: the rule is off |
| `server.rate_limits.<rule>.burst` | int | `100` | Requests allowed in a burst. Below `1`: 1 |
| `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's limiter is kept after its last request; `0s`: for ever |
#### PROXY Protocol and Proxy Headers
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.proxy.trusted` | list | `127.0.0.1/32` | IP addresses and CIDR ranges of proxies whose forwarding headers are trusted; empty: none |
| `server.proxy.client_ip_headers` | list | `X-Forwarded-For, Forwarded, X-Real-IP` | Headers read for the client IP from a trusted proxy, in priority order; empty: always the connection's address |
| `server.proxy.strip_untrusted` | bool | `true` | Remove forwarding headers from requests not from a trusted proxy |
| `server.proxy.proxy_protocol.enabled` | bool | `false` | Accept the PROXY protocol from trusted sources |
| `server.proxy.proxy_protocol.trusted_sources` | list | | Addresses and CIDR ranges allowed to send a PROXY header; empty: `server.proxy.trusted` |
| `server.proxy.proxy_protocol.required` | bool | `false` | Reject connections from trusted sources without a PROXY header |
| `server.proxy.proxy_protocol.read_header_timeout` | duration | `5s` | How long to wait for the PROXY header; `0s` for none |
#### CSRF & Cross-Origin Protection
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.csrf.cross_origin_protection.enabled` | bool | `true` | Reject cross-origin unsafe requests (`Sec-Fetch-Site`/`Origin`) |
| `server.csrf.cross_origin_protection.trusted_origins` | list | | Extra origins (`https://admin.example.com`) allowed to make them, and to open WebSockets; empty: none |
| `server.csrf.tokens.enabled` | bool | `true` | Require a signed double-submit token on unsafe requests |
| `server.csrf.tokens.key` | string | | Hex HMAC key; empty: a random per-process key, so tokens don't survive a restart |
| `server.csrf.tokens.cookie_name` | string | `js_csrf` | Empty: `js_csrf` |
| `server.csrf.tokens.header_name` | string | `X-CSRF-Token` | Empty: `X-CSRF-Token` |
| `server.csrf.tokens.token_ttl` | duration | `12h` | `0s`: 12h |
| `server.csrf.exempt_paths` | list | | ServeMux-style patterns skipped by both CSRF layers (e.g. `/api/webhooks/`); empty: none |
#### TLS
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.tls.enabled` | bool | `false` | Serve HTTPS |
| `server.tls.cert_file` | string | | Certificate file; must be set when TLS is enabled |
| `server.tls.key_file` | string | | Private key file; must be set when TLS is enabled |
| `server.tls.min_version` | string | `1.3` | `1.2` or `1.3`; anything else, including empty, means 1.2 |
| `server.tls.redirect_http_port` | int | | Port of a plain HTTP listener that redirects to HTTPS; `0`: none |
#### Headers
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.headers.allowed_hosts` | list | | Allowed `Host` headers, `*.example.com` supported; empty disables the check |
| `server.headers.content_security_policy` | string | | Security response headers; empty leaves the header unset |
| `server.headers.frame_options` | string | | |
| `server.headers.referrer_policy` | string | | |
| `server.headers.permissions_policy` | string | | |
| `server.headers.cross_origin_opener_policy` | string | | |
| `server.headers.cross_origin_resource_policy` | string | | |
| `server.headers.extra_response_headers` | list of objects | | Extra headers set on every response, each `{header: ..., value: ...}`. `header` must be a valid header name and `value` must not contain a line break |
| `server.headers.remove_response_headers` | list | | Headers stripped from every response (e.g. `Server`); empty: none |
| `server.headers.hsts.enabled` | bool | `false` | Send `Strict-Transport-Security` on HTTPS requests |
| `server.headers.hsts.max_age` | duration | `8760h` | `0s`: the header isn't sent |
| `server.headers.hsts.include_subdomains` | bool | `true` | |
| `server.headers.hsts.preload` | bool | `false` | |
#### Frontend
Not used yet.
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.frontend.dir` | string | | Serve the frontend from this directory; empty: the embedded build |
| `server.frontend.immutable_prefix` | string | | Where Vite emits content-hashed files |
| `server.frontend.index_cache_control` | string | | `Cache-Control` of HTML and other files without a hash; empty: none |
#### Websockets
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.websocket.enable_compression` | bool | `false` | permessage-deflate |
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
## Complex Types
A rate limit rule (`JSCRateLimitRule`) has `enabled`, `per_second` and `burst`.
It is off unless `enabled` and `per_second` is greater than 0; a `burst` below
1 means 1.
It is used under `server.websocket.connection_limit` and
`server.websocket.message_rate_limit`.
1 means 1. It is used under several keys: `server.rate_limits.global`,
`server.rate_limits.api`, `server.websocket.connection_limit` and
`server.websocket.message_rate_limit`. Rate limits are implemented using the
token bucket algorithm.
Lists can be given as YAML/JSON lists or as a comma-separated string (e.g. in
an environment variable). A list of objects such as `server.headers.extra_response_headers`
is a YAML/JSON list of objects in a config file, or a JSON array in an
environment variable or `config set`:
`'[{"header":"X-Robots-Tag","value":"noindex"}]'`.
## Example Configuration
Example `config.yaml`:
```yaml
@@ -80,6 +228,25 @@ sources:
enabled: false
interval: 1m
api_root: http://localhost:8080/api/1.0
server:
host: 0.0.0.0
port: 9393
max_body_bytes: 2MB
proxy:
trusted: [10.0.0.0/8]
rate_limits:
api:
enabled: true
per_second: 20
burst: 40
headers:
allowed_hosts: [jobs.example.com]
extra_response_headers:
- header: X-Robots-Tag
value: noindex
remove_response_headers: [Server]
hsts:
enabled: true
```
@@ -99,23 +266,42 @@ per key:
`time.Duration` or a duration string in `SetValue`, and are given out as
strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
`YAML` and `String` too.
- `util.ByteSize` fields work the same way, with size strings such as `"32KiB"`
or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers of 1024),
and are given out exactly in the largest unit that divides them (`"1MB"`).
`-1` means no limit (`util.ByteSizeNoLimit`); no other size may be
negative. A setting for which no limit is unsafe rejects `-1`.
- `util.ByteSize` fields work the same way, with size strings such as
`"32KiB"` or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers
of 1024), and are given out exactly in the largest unit that divides them
(`"1MB"`). `-1` means no limit (`util.ByteSizeNoLimit`); no other size may
be negative. A setting for which no limit is unsafe rejects `-1`.
- `[]string` fields report `reflect.Slice` from `TypeOf` and accept either a
`[]string` or a comma-separated string such as `"a, b"` in `SetValue` (and so
in default tags). An empty list is stored as `nil`.
- A list of sections, `[]S` for a section struct `S` (e.g. `[]JSCServerHeadersExtra`),
is a single key reporting `reflect.Slice`. It is given out as a
`[]map[string]any`, one map per item, and `SetValue` accepts a `[]S`, a list
of maps (as Viper reads a YAML/JSON list of objects) or a JSON array string.
Each item starts from its defaults and its keys are set as `SetValue` sets
them. `S` implements `Section` and, like a type used under several keys, has
no `JSCK*` constant.
- A section can reject values by implementing `validate(key, value) error` on
its pointer type. `JSCLogging` does this for `level`, and
`JSCServerWebSocket` for `max_message_size` (rejecting `-1`).
its pointer type. `JSCLogging` does this for `level`, `JSCSourcesSource` for
`interval`, `JSCServer` for `max_header_bytes`, `JSCServerTimeouts` for
`read_header` (rejecting 0 or less) and `JSCServerWebSocket` for
`max_message_size` (rejecting `-1`), and `JSCServerHeadersExtra` for
`header` and `value`. Each item in a list of
sections is validated whole, so a key left out of an item is checked too.
Every section implements the `Section` interface defined in `section.go`:
`Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue`, each a one-line call into
`section.go` (some sections also have `Keys`). The methods sit under a
`// ---------- Section methods (see Section) ----------` banner in each
section's file. `TestSections` walks `JobSearchConfig` and checks every
section's file. A section's `Flatten` keys are prefixed with its `JSCK*`
constant, except for a section type used under several keys, such as
`JSCRateLimitRule`: it has no constant and its keys are unprefixed.
`TestSections` walks `JobSearchConfig` and checks every
section, including embedded ones, implements `Section` and that its
`Flatten` keys match its `cfg` tag, so a new section is checked without being
listed anywhere.
`Flatten` keys match its `cfg` tag, and that its `Map`, `TypeOf`, `ValueOf`
and `SetValue` agree with the whole configuration's view of the same keys (the
items of a list of sections are checked the same way), so a new section is
checked without being listed anywhere.
## Adding a key or section
@@ -126,7 +312,8 @@ else is needed for it to be loaded, printed, and read or written by the
To add a new section:
1. Create a `conf_*.go` file with the section struct, a `JSCK*` constant for
its key, and an implementation of `Section` (copy an existing section's
methods).
methods). A type that will be used under several keys, like
`JSCRateLimitRule`, has no constant and passes `""` to `flattenOf`.
2. Add the struct as a tagged field of its parent: `JobSearchConfig` in
`config.go`, or `JSCSources` in `conf_sources.go` for a source,
etc. The field's `cfg` tag must match the `JSCK*` constant.
@@ -165,7 +352,7 @@ Nothing in `section.go` needs changing.
| `conf_sources.go` | `sources` section and the shared source fields |
| `conf_source_*.go` | One file per source section |
| `conf_server.go` | `server` section |
| `conf_server_*.go` | One file per `server` sub-section (`rate_limits`, `timeouts`, `websocket`) |
| `conf_server_*.go` | One file per `server` sub-section (`csrf`, `frontend`, `headers`, `proxy`, `rate_limits`, `timeouts`, `tls`, `websocket`) |
| `conf_ratelimitrule.go` | `JSCRateLimitRule`, the rate limit rule shared by several keys |
| `print.go`, `debug.go` | Printing and debug logging of the configuration |