Go/job-search
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
2490519c2bfabb057d7ae5c1c0b468b8cbac4934
job-search/internal/config/README.md
T
bwilliams a279405cae Added the HTTP server and serve command, with proxy, CSRF, TLS and security header config; wsserve is now ws serve, alongside a new ws dial 
The serve command runs internal/server: the API under /api (health, and
the CSRF token endpoint) and the WebSocket hub under /ws, over HTTP or TLS,
with an optional HTTP to HTTPS redirect, the PROXY protocol from trusted
sources and a graceful shutdown that closes WebSockets first. Its middleware
covers the client IP behind trusted proxies, request IDs, request logging,
panic recovery, global and API rate limits, Host and body size checks,
security response headers, compression, CSRF protection (cross-origin
checks and signed double-submit tokens) and JSON-only API bodies. Middleware
that logs takes a WithLogger option.

New server config sections: proxy, csrf, tls, headers and frontend (not
used yet), plus max_header_bytes, max_body_bytes, global and API rate limits
and HTTP read, write and idle timeouts. Every setting documents what 0 or
empty means, and internal/config/README.md is reorganised by section.
server.timeouts.read_header must be greater than 0, and a header or idle
timeout of 0 means no limit rather than the read timeout.

WebSocket upgrades are protected by the Origin check: the server's own
origin, or one of csrf.cross_origin_protection.trusted_origins, which the
hub now takes with WithTrustedOrigins.

wsserve becomes "ws serve". "ws dial" opens a WebSocket on /ws to test it:
--ping sends a ping and waits for the pong, --message sends messages and
prints the replies, --origin tests the Origin check, and with neither
--ping nor --message each line of stdin is sent until Ctrl+C.
2026-10-15 17:47:40 +01:00

365 lines
18 KiB
Markdown
Raw Blame History

# config
Package config loads the JobSearch configuration into a `JobSearchConfig`
struct. Values come from struct-tag defaults, a YAML or JSON config file and
environment variables, merged through [Viper](https://github.com/spf13/viper).
## Loading order
`Init(cfgFile, &conf)` builds the configuration in this order:
1. Loads `.env` from the working directory into the environment. A missing
`.env` file is an error.
2. Sets the defaults from the `default` struct tags as Viper defaults.
3. Reads the config file: `cfgFile` if given, otherwise the first
`config.yaml` found in `/etc/job-search` then the working directory. If no
file is found a warning is logged and the defaults are used. A file that
exists but can't be parsed is an error.
4. Reads environment variables prefixed with `JOBSEARCH_`. Dots in a key
become underscores, so `logging.level` is read from
`JOBSEARCH_LOGGING_LEVEL`.
5. Copies every known key from Viper into `conf`. A value that fails
validation (e.g. an unknown log level) is logged as a warning and the
previous value is kept.
When more than one place sets a key, environment variables win over the
config file, which wins over the defaults.
## Configuration Keys
Configuration is managed in sections:
| Key | Section description |
|---|---|
| `logging` | Paths to log files and log level control |
| `database` | Database connection/file details |
| `sources` | Opportunity source settings |
| `server` | Server configuration including CSRF protection, Websockets, Proxy handling and rate limits |
### Logging
| Key | Type | Default | Notes |
|---|---|---|---|
| `logging.level` | string | `warn` | One of `debug`, `info`, `warn`, `error` |
| `logging.file` | string | | Log file (JSON lines) for long-running commands such as `fetch service`. Empty: log to the terminal |
Logging always starts on the terminal. Long-running commands switch to the file
in `logging.file` once the configuration is loaded, unless it is empty (they
then stay on the terminal, with a warning) or `--foreground` is passed.
All other commands log to the terminal.
The `--debug` and `--quiet` flags force the log level to `debug` and `error`
respectively, ignoring the `logging.level` configuration. `--debug` takes
priority over `--quiet`.
### Database
| Key | Type | Default | Notes |
|---|---|---|---|
| `database.file` | string | | Path to the SQLite database file. Empty: commands that need the database fail |
Database is currently restricted to SQLite v3. At this point, no other engine is
supported.
### Opportunity Sources
| Key | Type | Default | Notes |
|---|---|---|---|
| `sources.<source>.enabled` | bool | `false` | For every source |
| `sources.<source>.interval` | duration | `10m` | For every source; e.g. `30s`, `1h`. Must be greater than 0; turn a source off with `enabled` |
| `sources.find_a_tender.api_root` | string | `https://www.find-tender.service.gov.uk/api/1.0` | URL of the Find a Tender API. Empty: the source fails to start if enabled, and `fetch once` fails |
| `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`). Empty: the source fails to start if enabled |
| `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown |
The current sources are `find_a_tender`, `fts_mock` and `sleeper`. See
"Addition of a new opportunities source" in the top level README for how to add
a new source.
### Server
The `server` section is split into a number of sub-sections:
| Key | Section description |
|---|---|
| `server` | Top level server configuration settings |
| `server.timeouts` | HTTP read, write and idle timeouts, and the shutdown timeout |
| `server.rate_limits` | Global (every request) and API rate limits. Also see `server.websocket.connection_limit` for WebSocket and plain TCP connection limits |
| `server.proxy` | PROXY protocol and proxy header settings |
| `server.csrf` | CSRF and cross-origin protection settings |
| `server.tls` | TLS certificate, minimum version and HTTP redirect |
| `server.headers` | HTTP header settings including policy headers |
| `server.frontend` | How the frontend is served (not used yet) |
| `server.websocket` | Websocket configuration |
#### Server
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.host` | string | `127.0.0.1` | Address to listen on; empty: every interface |
| `server.port` | int | `9393` | Port to listen on; `0`: any free port |
| `server.max_header_bytes` | byte size | `8KB` | Largest request header block; `0B` for 8KB. `-1` (no limit) is rejected, as the HTTP server always has one |
| `server.max_body_bytes` | byte size | `1MB` | Largest request body; `0B` for 1MB, `-1` for no limit |
#### Timeouts
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.timeouts.read_header` | duration | `5s` | Reading a request's headers. Must be greater than 0, as a client could otherwise hold a connection open for ever |
| `server.timeouts.read` | duration | `15s` | Reading a whole request; `0s` for none |
| `server.timeouts.write` | duration | `30s` | Writing a response; `0s` for none |
| `server.timeouts.idle` | duration | `2m` | Waiting for the next request on a keep-alive connection; `0s` for none |
| `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for open requests; `0s`: until they finish |
#### Rate Limits
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.rate_limits.<rule>.enabled` | bool | `false` | `<rule>` is `global` or `api` (see rate limit rules below) |
| `server.rate_limits.<rule>.per_second` | int | `50` | Sustained requests per second per client IP. `0` or less: the rule is off |
| `server.rate_limits.<rule>.burst` | int | `100` | Requests allowed in a burst. Below `1`: 1 |
| `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's limiter is kept after its last request; `0s`: for ever |
#### PROXY Protocol and Proxy Headers
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.proxy.trusted` | list | `127.0.0.1/32` | IP addresses and CIDR ranges of proxies whose forwarding headers are trusted; empty: none |
| `server.proxy.client_ip_headers` | list | `X-Forwarded-For, Forwarded, X-Real-IP` | Headers read for the client IP from a trusted proxy, in priority order; empty: always the connection's address |
| `server.proxy.strip_untrusted` | bool | `true` | Remove forwarding headers from requests not from a trusted proxy |
| `server.proxy.proxy_protocol.enabled` | bool | `false` | Accept the PROXY protocol from trusted sources |
| `server.proxy.proxy_protocol.trusted_sources` | list | | Addresses and CIDR ranges allowed to send a PROXY header; empty: `server.proxy.trusted` |
| `server.proxy.proxy_protocol.required` | bool | `false` | Reject connections from trusted sources without a PROXY header |
| `server.proxy.proxy_protocol.read_header_timeout` | duration | `5s` | How long to wait for the PROXY header; `0s` for none |
#### CSRF & Cross-Origin Protection
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.csrf.cross_origin_protection.enabled` | bool | `true` | Reject cross-origin unsafe requests (`Sec-Fetch-Site`/`Origin`) |
| `server.csrf.cross_origin_protection.trusted_origins` | list | | Extra origins (`https://admin.example.com`) allowed to make them, and to open WebSockets; empty: none |
| `server.csrf.tokens.enabled` | bool | `true` | Require a signed double-submit token on unsafe requests |
| `server.csrf.tokens.key` | string | | Hex HMAC key; empty: a random per-process key, so tokens don't survive a restart |
| `server.csrf.tokens.cookie_name` | string | `js_csrf` | Empty: `js_csrf` |
| `server.csrf.tokens.header_name` | string | `X-CSRF-Token` | Empty: `X-CSRF-Token` |
| `server.csrf.tokens.token_ttl` | duration | `12h` | `0s`: 12h |
| `server.csrf.exempt_paths` | list | | ServeMux-style patterns skipped by both CSRF layers (e.g. `/api/webhooks/`); empty: none |
#### TLS
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.tls.enabled` | bool | `false` | Serve HTTPS |
| `server.tls.cert_file` | string | | Certificate file; must be set when TLS is enabled |
| `server.tls.key_file` | string | | Private key file; must be set when TLS is enabled |
| `server.tls.min_version` | string | `1.3` | `1.2` or `1.3`; anything else, including empty, means 1.2 |
| `server.tls.redirect_http_port` | int | | Port of a plain HTTP listener that redirects to HTTPS; `0`: none |
#### Headers
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.headers.allowed_hosts` | list | | Allowed `Host` headers, `*.example.com` supported; empty disables the check |
| `server.headers.content_security_policy` | string | | Security response headers; empty leaves the header unset |
| `server.headers.frame_options` | string | | |
| `server.headers.referrer_policy` | string | | |
| `server.headers.permissions_policy` | string | | |
| `server.headers.cross_origin_opener_policy` | string | | |
| `server.headers.cross_origin_resource_policy` | string | | |
| `server.headers.extra_response_headers` | list of objects | | Extra headers set on every response, each `{header: ..., value: ...}`. `header` must be a valid header name and `value` must not contain a line break |
| `server.headers.remove_response_headers` | list | | Headers stripped from every response (e.g. `Server`); empty: none |
| `server.headers.hsts.enabled` | bool | `false` | Send `Strict-Transport-Security` on HTTPS requests |
| `server.headers.hsts.max_age` | duration | `8760h` | `0s`: the header isn't sent |
| `server.headers.hsts.include_subdomains` | bool | `true` | |
| `server.headers.hsts.preload` | bool | `false` | |
#### Frontend
Not used yet.
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.frontend.dir` | string | | Serve the frontend from this directory; empty: the embedded build |
| `server.frontend.immutable_prefix` | string | | Where Vite emits content-hashed files |
| `server.frontend.index_cache_control` | string | | `Cache-Control` of HTML and other files without a hash; empty: none |
#### Websockets
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.websocket.enable_compression` | bool | `false` | permessage-deflate |
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
## Complex Types
A rate limit rule (`JSCRateLimitRule`) has `enabled`, `per_second` and `burst`.
It is off unless `enabled` and `per_second` is greater than 0; a `burst` below
1 means 1. It is used under several keys: `server.rate_limits.global`,
`server.rate_limits.api`, `server.websocket.connection_limit` and
`server.websocket.message_rate_limit`. Rate limits are implemented using the
token bucket algorithm.
Lists can be given as YAML/JSON lists or as a comma-separated string (e.g. in
an environment variable). A list of objects such as `server.headers.extra_response_headers`
is a YAML/JSON list of objects in a config file, or a JSON array in an
environment variable or `config set`:
`'[{"header":"X-Robots-Tag","value":"noindex"}]'`.
## Example Configuration
Example `config.yaml`:
```yaml
logging:
level: info
file: ""
database:
file: ./jobsearch.db
sources:
find_a_tender:
enabled: true
interval: 10m
fts_mock:
enabled: false
interval: 1m
api_root: http://localhost:8080/api/1.0
server:
host: 0.0.0.0
port: 9393
max_body_bytes: 2MB
proxy:
trusted: [10.0.0.0/8]
rate_limits:
api:
enabled: true
per_second: 20
burst: 40
headers:
allowed_hosts: [jobs.example.com]
extra_response_headers:
- header: X-Robots-Tag
value: noindex
remove_response_headers: [Server]
hsts:
enabled: true
```
## Sections and struct tags
Each section of the configuration is a plain struct. The reflection code in
`section.go` reads its struct tags, so a section needs no hand-written code
per key:
- `cfg:"key"` is the field's configuration key. Fields without it are not part
of the configuration.
- `default:"value"` is the field's default, parsed as if it were a string
passed to `SetValue` (e.g. `"10m"`, `"true"`).
- A tagged field whose type is a struct is a nested section, addressed as
`section.key`. An embedded struct has its fields promoted, as Go does;
every source embeds `JSCSourcesSource` for `enabled` and `interval`.
- `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a
`time.Duration` or a duration string in `SetValue`, and are given out as
strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
`YAML` and `String` too.
- `util.ByteSize` fields work the same way, with size strings such as
`"32KiB"` or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers
of 1024), and are given out exactly in the largest unit that divides them
(`"1MB"`). `-1` means no limit (`util.ByteSizeNoLimit`); no other size may
be negative. A setting for which no limit is unsafe rejects `-1`.
- `[]string` fields report `reflect.Slice` from `TypeOf` and accept either a
`[]string` or a comma-separated string such as `"a, b"` in `SetValue` (and so
in default tags). An empty list is stored as `nil`.
- A list of sections, `[]S` for a section struct `S` (e.g. `[]JSCServerHeadersExtra`),
is a single key reporting `reflect.Slice`. It is given out as a
`[]map[string]any`, one map per item, and `SetValue` accepts a `[]S`, a list
of maps (as Viper reads a YAML/JSON list of objects) or a JSON array string.
Each item starts from its defaults and its keys are set as `SetValue` sets
them. `S` implements `Section` and, like a type used under several keys, has
no `JSCK*` constant.
- A section can reject values by implementing `validate(key, value) error` on
its pointer type. `JSCLogging` does this for `level`, `JSCSourcesSource` for
`interval`, `JSCServer` for `max_header_bytes`, `JSCServerTimeouts` for
`read_header` (rejecting 0 or less) and `JSCServerWebSocket` for
`max_message_size` (rejecting `-1`), and `JSCServerHeadersExtra` for
`header` and `value`. Each item in a list of
sections is validated whole, so a key left out of an item is checked too.
Every section implements the `Section` interface defined in `section.go`:
`Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue`, each a one-line call into
`section.go` (some sections also have `Keys`). The methods sit under a
`// ---------- Section methods (see Section) ----------` banner in each
section's file. A section's `Flatten` keys are prefixed with its `JSCK*`
constant, except for a section type used under several keys, such as
`JSCRateLimitRule`: it has no constant and its keys are unprefixed.
`TestSections` walks `JobSearchConfig` and checks every
section, including embedded ones, implements `Section` and that its
`Flatten` keys match its `cfg` tag, and that its `Map`, `TypeOf`, `ValueOf`
and `SetValue` agree with the whole configuration's view of the same keys (the
items of a list of sections are checked the same way), so a new section is
checked without being listed anywhere.
## Adding a key or section
To add a key to an existing section, add a tagged field to its struct. Nothing
else is needed for it to be loaded, printed, and read or written by the
`config` CLI commands.
To add a new section:
1. Create a `conf_*.go` file with the section struct, a `JSCK*` constant for
its key, and an implementation of `Section` (copy an existing section's
methods). A type that will be used under several keys, like
`JSCRateLimitRule`, has no constant and passes `""` to `flattenOf`.
2. Add the struct as a tagged field of its parent: `JobSearchConfig` in
`config.go`, or `JSCSources` in `conf_sources.go` for a source,
etc. The field's `cfg` tag must match the `JSCK*` constant.
Nothing in `section.go` needs changing.
## API
- `Init(cfgFile, &conf)`: load the configuration, as described above.
- `conf.TypeOf(key)`, `conf.ValueOf(key)`, `conf.SetValue(key, value)`: read
or set one dotted key on the loaded struct. `SetValue` converts and
validates the value.
- `conf.Map()` (nested) and `conf.Flatten()` (dotted keys): the whole
configuration as a map.
- `conf.String()`, `conf.JSON()`, `conf.YAML()` and the matching `Print*`
methods: the configuration as text, with keys sorted.
- `StoreValue(&conf, key, value)` / `PurgeValue(&conf, key)`: set or remove a
key in the config file Viper loaded. Used by `config set` and
`config set --unset`. `StoreValue` validates the value before writing it,
and stores it as the field's own type. Only the file's own settings are
written back; defaults and environment variables never are.
- `LoadDefaultConfig`, `LoadToViper`, `LoadFromViper`: move values between
the struct defaults, Viper and a `JobSearchConfig`.
## Files
| File | Contents |
|---|---|
| `config.go` | `JobSearchConfig`, defaults, Viper loading, text output |
| `conf_database.go` | `database` section |
| `load.go` | `Init`, `.env` and config file loading |
| `modify.go` | `StoreValue` and `PurgeValue` |
| `section.go` | The `Section` interface and the struct tag reflection shared by every section |
| `conf_logging.go` | `logging` section and log level helpers |
| `conf_sources.go` | `sources` section and the shared source fields |
| `conf_source_*.go` | One file per source section |
| `conf_server.go` | `server` section |
| `conf_server_*.go` | One file per `server` sub-section (`csrf`, `frontend`, `headers`, `proxy`, `rate_limits`, `timeouts`, `tls`, `websocket`) |
| `conf_ratelimitrule.go` | `JSCRateLimitRule`, the rate limit rule shared by several keys |
| `print.go`, `debug.go` | Printing and debug logging of the configuration |
## Testing
Viper keeps its state in package globals, so tests that use it call
`resetViper(t)` to reset it before and after the test. Tests of `Init`
use `initDir`, which changes into a temp directory with its own `.env` and
`config.yaml`, so the real files in the repository are never read.
Reference in new issue View Git Blame Copy Permalink