Go/job-search
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
main
job-search/internal/config/README.md
T
bwilliams b0f93fd437 config init now leaves out the test sources unless given --all 
A section tagged init:"hidden" isn't written to the file config init
generates; fts_mock and sleeper are. config init --all writes them too.
They still load, and the config commands read and set them as before.
2026-10-21 16:22:45 +01:00

378 lines
20 KiB
Markdown
Raw Permalink Blame History

# config
Package config loads the JobSearch configuration into a `JobSearchConfig`
struct. Values come from struct-tag defaults, a YAML or JSON config file and
environment variables, merged through [Viper](https://github.com/spf13/viper).
## Loading order
`Init(cfgFile, &conf)` builds the configuration in this order:
1. Loads `.env` from the working directory into the environment, if there
is one. A `.env` file that can't be read or parsed is an error.
2. Sets the defaults from the `default` struct tags as Viper defaults.
3. Reads the config file: `cfgFile` if given, otherwise the first
`config.yaml` found in `/etc/job-search` then the working directory. If no
file is found a warning is logged and the defaults are used. A file that
exists but can't be parsed is an error.
4. Reads environment variables prefixed with `JOBSEARCH_`. Dots in a key
become underscores, so `logging.level` is read from
`JOBSEARCH_LOGGING_LEVEL`.
5. Copies every known key from Viper into `conf`. A value that fails
validation (e.g. an unknown log level) is logged as a warning and the
previous value is kept.
When more than one place sets a key, environment variables win over the
config file, which wins over the defaults.
## Configuration Keys
Configuration is managed in sections:
| Key | Section description |
|---|---|
| `logging` | Paths to log files and log level control |
| `database` | Database connection/file details |
| `sources` | Opportunity source settings |
| `server` | Server configuration including CSRF protection, Websockets, Proxy handling and rate limits |
### Logging
| Key | Type | Default | Notes |
|---|---|---|---|
| `logging.level` | string | `warn` | One of `debug`, `info`, `warn`, `error` |
| `logging.file` | string | | Log file (JSON lines) for long-running commands such as `fetcher service`. Empty: log to the terminal |
Logging always starts on the terminal. Long-running commands switch to the file
in `logging.file` once the configuration is loaded, unless it is empty (they
then stay on the terminal, with a warning) or `--foreground` is passed.
All other commands log to the terminal.
The `--debug` and `--quiet` flags force the log level to `debug` and `error`
respectively, ignoring the `logging.level` configuration. `--debug` takes
priority over `--quiet`.
### Database
| Key | Type | Default | Notes |
|---|---|---|---|
| `database.file` | string | | Path to the SQLite database file. Empty: commands that need the database fail |
| `database.change_batch_retention` | duration | `720h` | How long a batch of changed opportunities is kept for clients to catch up from (`?changed_after=`); older ones are deleted as new ones are recorded. A client whose batch has gone reloads the whole list. 0 or less: kept for ever |
Database is currently restricted to SQLite v3. At this point, no other engine is
supported.
### Opportunity Sources
| Key | Type | Default | Notes |
|---|---|---|---|
| `sources.skip_rules_file` | string | | JSON file of rules that mark new opportunities Skipped (see `internal/skiprules`); read again when it changes. Empty: none. Missing or invalid: `fetcher service` and `serve` refuse to start |
| `sources.<source>.enabled` | bool | `false` | For every source |
| `sources.<source>.interval` | duration | `10m` | For every source; e.g. `30s`, `1h`. Must be greater than 0; turn a source off with `enabled` |
| `sources.find_a_tender.api_root` | string | `https://www.find-tender.service.gov.uk/api/1.0` | URL of the Find a Tender API. Empty: the source fails to start if enabled, and `fetcher fts-once` fails |
| `sources.find_a_tender.rate_limit_retries` | int | `5` | How many times a request refused for the API's rate limit is waited out (for its `Retry-After`) and retried. 0 fails at the first; can't be negative. `fetcher fts-once --ignore-rate-limit` uses 0 |
| `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`). Empty: the source fails to start if enabled |
| `sources.fts_mock.rate_limit_retries` | int | `5` | As for `find_a_tender` |
| `sources.eu_supply.base_url` | string | `https://uk.eu-supply.com` | Root of EU Supply's UK portal (see `internal/eusupply`) |
| `sources.eu_supply.request_min_gap` | duration | `2s` | The shortest wait between two requests to the portal. Must be greater than 0: the site publishes no rate limit |
| `sources.eu_supply.request_max_gap` | duration | `10s` | The longest wait between two requests; each wait is random, from `request_min_gap`. Must be greater than 0; less than `request_min_gap` is read as `request_min_gap` |
| `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown |
The current sources are `find_a_tender`, `fts_mock`, `eu_supply` and
`sleeper`. See
"Addition of a new opportunities source" in the top level README for how to add
a new source.
### Server
The `server` section is split into a number of sub-sections:
| Key | Section description |
|---|---|
| `server` | Top level server configuration settings |
| `server.timeouts` | HTTP read, write and idle timeouts, and the shutdown timeout |
| `server.rate_limits` | Global (every request) and API rate limits. Also see `server.websocket.connection_limit` for WebSocket and plain TCP connection limits |
| `server.proxy` | PROXY protocol and proxy header settings |
| `server.csrf` | CSRF and cross-origin protection settings |
| `server.tls` | TLS certificate, minimum version and HTTP redirect |
| `server.headers` | HTTP header settings including policy headers |
| `server.frontend` | Where the frontend's build output is and how it is cached |
| `server.websocket` | Websocket configuration |
#### Server
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.host` | string | `127.0.0.1` | Address to listen on; empty: every interface |
| `server.port` | int | `9393` | Port to listen on; `0`: any free port |
| `server.max_header_bytes` | byte size | `8KB` | Largest request header block; `0B` for 8KB. `-1` (no limit) is rejected, as the HTTP server always has one |
| `server.max_body_bytes` | byte size | `1MB` | Largest request body; `0B` for 1MB, `-1` for no limit |
| `server.access_log` | string | | File the access log is appended to, one line per request in the Combined Log Format; `-`: stdout; empty: no access log. `serve --access-log` overrides it |
#### Timeouts
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.timeouts.read_header` | duration | `5s` | Reading a request's headers. Must be greater than 0, as a client could otherwise hold a connection open for ever |
| `server.timeouts.read` | duration | `15s` | Reading a whole request; `0s` for none |
| `server.timeouts.write` | duration | `30s` | Writing a response; `0s` for none |
| `server.timeouts.idle` | duration | `2m` | Waiting for the next request on a keep-alive connection; `0s` for none |
| `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for open requests; `0s`: until they finish |
#### Rate Limits
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.rate_limits.<rule>.enabled` | bool | `false` | `<rule>` is `global` or `api` (see rate limit rules below) |
| `server.rate_limits.<rule>.per_second` | int | `50` | Sustained requests per second per client IP. `0` or less: the rule is off |
| `server.rate_limits.<rule>.burst` | int | `100` | Requests allowed in a burst. Below `1`: 1 |
| `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's limiter is kept after its last request; `0s`: for ever |
#### PROXY Protocol and Proxy Headers
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.proxy.trusted` | list | `127.0.0.1/32` | IP addresses and CIDR ranges of proxies whose forwarding headers are trusted; empty: none |
| `server.proxy.client_ip_headers` | list | `X-Forwarded-For, Forwarded, X-Real-IP` | Headers read for the client IP from a trusted proxy, in priority order; empty: always the connection's address |
| `server.proxy.strip_untrusted` | bool | `true` | Remove forwarding headers from requests not from a trusted proxy |
| `server.proxy.proxy_protocol.enabled` | bool | `false` | Accept the PROXY protocol from trusted sources |
| `server.proxy.proxy_protocol.trusted_sources` | list | | Addresses and CIDR ranges allowed to send a PROXY header; empty: `server.proxy.trusted` |
| `server.proxy.proxy_protocol.required` | bool | `false` | Reject connections from trusted sources without a PROXY header |
| `server.proxy.proxy_protocol.read_header_timeout` | duration | `5s` | How long to wait for the PROXY header; `0s` for none |
#### CSRF & Cross-Origin Protection
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.csrf.cross_origin_protection.enabled` | bool | `true` | Reject cross-origin unsafe requests (`Sec-Fetch-Site`/`Origin`) |
| `server.csrf.cross_origin_protection.trusted_origins` | list | | Extra origins (`https://admin.example.com`) allowed to make them, and to open WebSockets; empty: none |
| `server.csrf.tokens.enabled` | bool | `true` | Require a signed double-submit token on unsafe requests |
| `server.csrf.tokens.key` | string | | Hex HMAC key; empty: a random per-process key, so tokens don't survive a restart. `job-search config generate-csrf-key` stores one |
| `server.csrf.tokens.cookie_name` | string | `js_csrf` | Empty: `js_csrf` |
| `server.csrf.tokens.header_name` | string | `X-CSRF-Token` | Empty: `X-CSRF-Token` |
| `server.csrf.tokens.token_ttl` | duration | `12h` | `0s`: 12h |
| `server.csrf.exempt_paths` | list | | ServeMux-style patterns skipped by both CSRF layers (e.g. `/api/webhooks/`); empty: none |
#### TLS
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.tls.enabled` | bool | `false` | Serve HTTPS |
| `server.tls.cert_file` | string | | Certificate file; must be set when TLS is enabled |
| `server.tls.key_file` | string | | Private key file; must be set when TLS is enabled |
| `server.tls.min_version` | string | `1.3` | `1.2` or `1.3`; anything else, including empty, means 1.2 |
| `server.tls.redirect_http_port` | int | | Port of a plain HTTP listener that redirects to HTTPS; `0`: none |
#### Headers
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.headers.allowed_hosts` | list | | Allowed `Host` headers, `*.example.com` supported; empty disables the check |
| `server.headers.content_security_policy` | string | | Security response headers; empty leaves the header unset |
| `server.headers.frame_options` | string | | |
| `server.headers.referrer_policy` | string | | |
| `server.headers.permissions_policy` | string | | |
| `server.headers.cross_origin_opener_policy` | string | | |
| `server.headers.cross_origin_resource_policy` | string | | |
| `server.headers.extra_response_headers` | list of objects | | Extra headers set on every response, each `{header: ..., value: ...}`. `header` must be a valid header name and `value` must not contain a line break |
| `server.headers.remove_response_headers` | list | | Headers stripped from every response (e.g. `Server`); empty: none |
| `server.headers.hsts.enabled` | bool | `false` | Send `Strict-Transport-Security` on HTTPS requests |
| `server.headers.hsts.max_age` | duration | `8760h` | `0s`: the header isn't sent |
| `server.headers.hsts.include_subdomains` | bool | `true` | |
| `server.headers.hsts.preload` | bool | `false` | |
#### Frontend
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.frontend.dir` | string | | Directory of the frontend's build output, holding `index.html`; empty: the build embedded in the binary (see `npm run sync` in `web/`), or no frontend if it has none |
| `server.frontend.immutable_prefix` | string | | Where Vite emits content-hashed files, cached for a year; empty: `/assets/` |
| `server.frontend.index_cache_control` | string | | `Cache-Control` of HTML and other files without a hash; empty: `no-cache` |
#### Websockets
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.websocket.enable_compression` | bool | `false` | permessage-deflate |
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
## Complex Types
A rate limit rule (`JSCRateLimitRule`) has `enabled`, `per_second` and `burst`.
It is off unless `enabled` and `per_second` is greater than 0; a `burst` below
1 means 1. It is used under several keys: `server.rate_limits.global`,
`server.rate_limits.api`, `server.websocket.connection_limit` and
`server.websocket.message_rate_limit`. Rate limits are implemented using the
token bucket algorithm.
Lists can be given as YAML/JSON lists or as a comma-separated string (e.g. in
an environment variable). A list of objects such as `server.headers.extra_response_headers`
is a YAML/JSON list of objects in a config file, or a JSON array in an
environment variable or `config set`:
`'[{"header":"X-Robots-Tag","value":"noindex"}]'`.
## Example Configuration
Example `config.yaml`:
```yaml
logging:
level: info
file: ""
database:
file: ./jobsearch.db
change_batch_retention: 720h
sources:
find_a_tender:
enabled: true
interval: 10m
fts_mock:
enabled: false
interval: 1m
api_root: http://localhost:8080/api/1.0
server:
host: 0.0.0.0
port: 9393
max_body_bytes: 2MB
proxy:
trusted: [10.0.0.0/8]
rate_limits:
api:
enabled: true
per_second: 20
burst: 40
headers:
allowed_hosts: [jobs.example.com]
extra_response_headers:
- header: X-Robots-Tag
value: noindex
remove_response_headers: [Server]
hsts:
enabled: true
```
## Sections and struct tags
Each section of the configuration is a plain struct. The reflection code in
`section.go` reads its struct tags, so a section needs no hand-written code
per key:
- `cfg:"key"` is the field's configuration key. Fields without it are not part
of the configuration.
- `default:"value"` is the field's default, parsed as if it were a string
passed to `SetValue` (e.g. `"10m"`, `"true"`).
- `init:"hidden"` on a nested section's field leaves the section out of the
file `config init` writes, unless `--all` is given (`InitDefaults`). The
test sources, `fts_mock` and `sleeper`, have it. The section is otherwise
as any other: loaded, and read and set by the `config` commands.
- A tagged field whose type is a struct is a nested section, addressed as
`section.key`. An embedded struct has its fields promoted, as Go does;
every source embeds `JSCSourcesSource` for `enabled` and `interval`.
- `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a
`time.Duration` or a duration string in `SetValue`, and are given out as
strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
`YAML` and `String` too.
- `util.ByteSize` fields work the same way, with size strings such as
`"32KiB"` or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers
of 1024), and are given out exactly in the largest unit that divides them
(`"1MB"`). `-1` means no limit (`util.ByteSizeNoLimit`); no other size may
be negative. A setting for which no limit is unsafe rejects `-1`.
- `[]string` fields report `reflect.Slice` from `TypeOf` and accept either a
`[]string` or a comma-separated string such as `"a, b"` in `SetValue` (and so
in default tags). An empty list is stored as `nil`.
- A list of sections, `[]S` for a section struct `S` (e.g. `[]JSCServerHeadersExtra`),
is a single key reporting `reflect.Slice`. It is given out as a
`[]map[string]any`, one map per item, and `SetValue` accepts a `[]S`, a list
of maps (as Viper reads a YAML/JSON list of objects) or a JSON array string.
Each item starts from its defaults and its keys are set as `SetValue` sets
them. `S` implements `Section` and, like a type used under several keys, has
no `JSCK*` constant.
- A section can reject values by implementing `validate(key, value) error` on
its pointer type. `JSCLogging` does this for `level`, `JSCSourcesSource` for
`interval`, `JSCServer` for `max_header_bytes`, `JSCServerTimeouts` for
`read_header` (rejecting 0 or less) and `JSCServerWebSocket` for
`max_message_size` (rejecting `-1`), and `JSCServerHeadersExtra` for
`header` and `value`. Each item in a list of
sections is validated whole, so a key left out of an item is checked too.
Every section implements the `Section` interface defined in `section.go`:
`Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue`, each a one-line call into
`section.go` (some sections also have `Keys`). The methods sit under a
`// ---------- Section methods (see Section) ----------` banner in each
section's file. A section's `Flatten` keys are prefixed with its `JSCK*`
constant, except for a section type used under several keys, such as
`JSCRateLimitRule`: it has no constant and its keys are unprefixed.
`TestSections` walks `JobSearchConfig` and checks every
section, including embedded ones, implements `Section` and that its
`Flatten` keys match its `cfg` tag, and that its `Map`, `TypeOf`, `ValueOf`
and `SetValue` agree with the whole configuration's view of the same keys (the
items of a list of sections are checked the same way), so a new section is
checked without being listed anywhere.
## Adding a key or section
To add a key to an existing section, add a tagged field to its struct. Nothing
else is needed for it to be loaded, printed, and read or written by the
`config` CLI commands.
To add a new section:
1. Create a `conf_*.go` file with the section struct, a `JSCK*` constant for
its key, and an implementation of `Section` (copy an existing section's
methods). A type that will be used under several keys, like
`JSCRateLimitRule`, has no constant and passes `""` to `flattenOf`.
2. Add the struct as a tagged field of its parent: `JobSearchConfig` in
`config.go`, or `JSCSources` in `conf_sources.go` for a source,
etc. The field's `cfg` tag must match the `JSCK*` constant. Add
`init:"hidden"` if it's only for testing.
Nothing in `section.go` needs changing.
## API
- `Init(cfgFile, &conf)`: load the configuration, as described above.
- `conf.TypeOf(key)`, `conf.ValueOf(key)`, `conf.SetValue(key, value)`: read
or set one dotted key on the loaded struct. `SetValue` converts and
validates the value.
- `conf.Map()` (nested) and `conf.Flatten()` (dotted keys): the whole
configuration as a map.
- `conf.String()`, `conf.JSON()`, `conf.YAML()` and the matching `Print*`
methods: the configuration as text, with keys sorted.
- `StoreValue(&conf, key, value)` / `PurgeValue(&conf, key)`: set or remove a
key in the config file Viper loaded. Used by `config set` and
`config set --unset`. `StoreValue` validates the value before writing it,
and stores it as the field's own type. Only the file's own settings are
written back; defaults and environment variables never are.
- `LoadDefaultConfig`, `LoadToViper`, `LoadFromViper`: move values between
the struct defaults, Viper and a `JobSearchConfig`.
## Files
| File | Contents |
|---|---|
| `config.go` | `JobSearchConfig`, defaults, Viper loading, text output |
| `conf_database.go` | `database` section |
| `load.go` | `Init`, `.env` and config file loading |
| `modify.go` | `StoreValue` and `PurgeValue` |
| `section.go` | The `Section` interface and the struct tag reflection shared by every section |
| `conf_logging.go` | `logging` section and log level helpers |
| `conf_sources.go` | `sources` section and the shared source fields |
| `conf_source_*.go` | One file per source section |
| `conf_server.go` | `server` section |
| `conf_server_*.go` | One file per `server` sub-section (`csrf`, `frontend`, `headers`, `proxy`, `rate_limits`, `timeouts`, `tls`, `websocket`) |
| `conf_ratelimitrule.go` | `JSCRateLimitRule`, the rate limit rule shared by several keys |
| `print.go`, `debug.go` | Printing and debug logging of the configuration |
## Testing
Viper keeps its state in package globals, so tests that use it call
`resetViper(t)` to reset it before and after the test. Tests of `Init`
use `initDir`, which changes into a temp directory with its own `.env` and
`config.yaml`, so the real files in the repository are never read.
Reference in new issue View Git Blame Copy Permalink