Go/job-search
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
ff2aa316b5513e436a81244927f2fef2f8558bc3
job-search/internal/config/README.md
T
bwilliams 0543762330 Added skip rules, which mark new opportunities Skipped when they're fetched 
sources.skip_rules_file names a JSON file of filters, each with a
comment, an active flag, a logic mode ("or" by default, or "and") and
rules by buyer, title words or CPV codes. The fetcher applies them
before saving, so only new opportunities are marked; the marks of
stored ones are never touched. CPV rules match when the opportunity's
codes are all in the list, or any of them with exclusive false, and
by prefix with exact false.

The file is read again when its time or size changes: an invalid edit
is logged and the last good rules kept, and a deleted file means no
rules until it's back. fetcher service and serve refuse to start if it
is define but missing or invalid.

job-search skip-rules check validates a rules file, and skip-rules
apply marks the stored opportunities that match Skipped, or only lists
them with --dry-run.
2026-10-20 08:55:19 +01:00

367 lines
19 KiB
Markdown
Raw Blame History

# config
Package config loads the JobSearch configuration into a `JobSearchConfig`
struct. Values come from struct-tag defaults, a YAML or JSON config file and
environment variables, merged through [Viper](https://github.com/spf13/viper).
## Loading order
`Init(cfgFile, &conf)` builds the configuration in this order:
1. Loads `.env` from the working directory into the environment. A missing
`.env` file is an error.
2. Sets the defaults from the `default` struct tags as Viper defaults.
3. Reads the config file: `cfgFile` if given, otherwise the first
`config.yaml` found in `/etc/job-search` then the working directory. If no
file is found a warning is logged and the defaults are used. A file that
exists but can't be parsed is an error.
4. Reads environment variables prefixed with `JOBSEARCH_`. Dots in a key
become underscores, so `logging.level` is read from
`JOBSEARCH_LOGGING_LEVEL`.
5. Copies every known key from Viper into `conf`. A value that fails
validation (e.g. an unknown log level) is logged as a warning and the
previous value is kept.
When more than one place sets a key, environment variables win over the
config file, which wins over the defaults.
## Configuration Keys
Configuration is managed in sections:
| Key | Section description |
|---|---|
| `logging` | Paths to log files and log level control |
| `database` | Database connection/file details |
| `sources` | Opportunity source settings |
| `server` | Server configuration including CSRF protection, Websockets, Proxy handling and rate limits |
### Logging
| Key | Type | Default | Notes |
|---|---|---|---|
| `logging.level` | string | `warn` | One of `debug`, `info`, `warn`, `error` |
| `logging.file` | string | | Log file (JSON lines) for long-running commands such as `fetcher service`. Empty: log to the terminal |
Logging always starts on the terminal. Long-running commands switch to the file
in `logging.file` once the configuration is loaded, unless it is empty (they
then stay on the terminal, with a warning) or `--foreground` is passed.
All other commands log to the terminal.
The `--debug` and `--quiet` flags force the log level to `debug` and `error`
respectively, ignoring the `logging.level` configuration. `--debug` takes
priority over `--quiet`.
### Database
| Key | Type | Default | Notes |
|---|---|---|---|
| `database.file` | string | | Path to the SQLite database file. Empty: commands that need the database fail |
| `database.change_batch_retention` | duration | `720h` | How long a batch of changed opportunities is kept for clients to catch up from (`?changed_after=`); older ones are deleted as new ones are recorded. A client whose batch has gone reloads the whole list. 0 or less: kept for ever |
Database is currently restricted to SQLite v3. At this point, no other engine is
supported.
### Opportunity Sources
| Key | Type | Default | Notes |
|---|---|---|---|
| `sources.skip_rules_file` | string | | JSON file of rules that mark new opportunities Skipped (see `internal/skiprules`); read again when it changes. Empty: none. Missing or invalid: `fetcher service` and `serve` refuse to start |
| `sources.<source>.enabled` | bool | `false` | For every source |
| `sources.<source>.interval` | duration | `10m` | For every source; e.g. `30s`, `1h`. Must be greater than 0; turn a source off with `enabled` |
| `sources.find_a_tender.api_root` | string | `https://www.find-tender.service.gov.uk/api/1.0` | URL of the Find a Tender API. Empty: the source fails to start if enabled, and `fetcher fts-once` fails |
| `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`). Empty: the source fails to start if enabled |
| `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown |
The current sources are `find_a_tender`, `fts_mock` and `sleeper`. See
"Addition of a new opportunities source" in the top level README for how to add
a new source.
### Server
The `server` section is split into a number of sub-sections:
| Key | Section description |
|---|---|
| `server` | Top level server configuration settings |
| `server.timeouts` | HTTP read, write and idle timeouts, and the shutdown timeout |
| `server.rate_limits` | Global (every request) and API rate limits. Also see `server.websocket.connection_limit` for WebSocket and plain TCP connection limits |
| `server.proxy` | PROXY protocol and proxy header settings |
| `server.csrf` | CSRF and cross-origin protection settings |
| `server.tls` | TLS certificate, minimum version and HTTP redirect |
| `server.headers` | HTTP header settings including policy headers |
| `server.frontend` | Where the frontend's build output is and how it is cached |
| `server.websocket` | Websocket configuration |
#### Server
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.host` | string | `127.0.0.1` | Address to listen on; empty: every interface |
| `server.port` | int | `9393` | Port to listen on; `0`: any free port |
| `server.max_header_bytes` | byte size | `8KB` | Largest request header block; `0B` for 8KB. `-1` (no limit) is rejected, as the HTTP server always has one |
| `server.max_body_bytes` | byte size | `1MB` | Largest request body; `0B` for 1MB, `-1` for no limit |
| `server.access_log` | string | | File the access log is appended to, one line per request in the Combined Log Format; `-`: stdout; empty: no access log. `serve --access-log` overrides it |
#### Timeouts
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.timeouts.read_header` | duration | `5s` | Reading a request's headers. Must be greater than 0, as a client could otherwise hold a connection open for ever |
| `server.timeouts.read` | duration | `15s` | Reading a whole request; `0s` for none |
| `server.timeouts.write` | duration | `30s` | Writing a response; `0s` for none |
| `server.timeouts.idle` | duration | `2m` | Waiting for the next request on a keep-alive connection; `0s` for none |
| `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for open requests; `0s`: until they finish |
#### Rate Limits
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.rate_limits.<rule>.enabled` | bool | `false` | `<rule>` is `global` or `api` (see rate limit rules below) |
| `server.rate_limits.<rule>.per_second` | int | `50` | Sustained requests per second per client IP. `0` or less: the rule is off |
| `server.rate_limits.<rule>.burst` | int | `100` | Requests allowed in a burst. Below `1`: 1 |
| `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's limiter is kept after its last request; `0s`: for ever |
#### PROXY Protocol and Proxy Headers
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.proxy.trusted` | list | `127.0.0.1/32` | IP addresses and CIDR ranges of proxies whose forwarding headers are trusted; empty: none |
| `server.proxy.client_ip_headers` | list | `X-Forwarded-For, Forwarded, X-Real-IP` | Headers read for the client IP from a trusted proxy, in priority order; empty: always the connection's address |
| `server.proxy.strip_untrusted` | bool | `true` | Remove forwarding headers from requests not from a trusted proxy |
| `server.proxy.proxy_protocol.enabled` | bool | `false` | Accept the PROXY protocol from trusted sources |
| `server.proxy.proxy_protocol.trusted_sources` | list | | Addresses and CIDR ranges allowed to send a PROXY header; empty: `server.proxy.trusted` |
| `server.proxy.proxy_protocol.required` | bool | `false` | Reject connections from trusted sources without a PROXY header |
| `server.proxy.proxy_protocol.read_header_timeout` | duration | `5s` | How long to wait for the PROXY header; `0s` for none |
#### CSRF & Cross-Origin Protection
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.csrf.cross_origin_protection.enabled` | bool | `true` | Reject cross-origin unsafe requests (`Sec-Fetch-Site`/`Origin`) |
| `server.csrf.cross_origin_protection.trusted_origins` | list | | Extra origins (`https://admin.example.com`) allowed to make them, and to open WebSockets; empty: none |
| `server.csrf.tokens.enabled` | bool | `true` | Require a signed double-submit token on unsafe requests |
| `server.csrf.tokens.key` | string | | Hex HMAC key; empty: a random per-process key, so tokens don't survive a restart. `job-search config generate-csrf-key` stores one |
| `server.csrf.tokens.cookie_name` | string | `js_csrf` | Empty: `js_csrf` |
| `server.csrf.tokens.header_name` | string | `X-CSRF-Token` | Empty: `X-CSRF-Token` |
| `server.csrf.tokens.token_ttl` | duration | `12h` | `0s`: 12h |
| `server.csrf.exempt_paths` | list | | ServeMux-style patterns skipped by both CSRF layers (e.g. `/api/webhooks/`); empty: none |
#### TLS
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.tls.enabled` | bool | `false` | Serve HTTPS |
| `server.tls.cert_file` | string | | Certificate file; must be set when TLS is enabled |
| `server.tls.key_file` | string | | Private key file; must be set when TLS is enabled |
| `server.tls.min_version` | string | `1.3` | `1.2` or `1.3`; anything else, including empty, means 1.2 |
| `server.tls.redirect_http_port` | int | | Port of a plain HTTP listener that redirects to HTTPS; `0`: none |
#### Headers
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.headers.allowed_hosts` | list | | Allowed `Host` headers, `*.example.com` supported; empty disables the check |
| `server.headers.content_security_policy` | string | | Security response headers; empty leaves the header unset |
| `server.headers.frame_options` | string | | |
| `server.headers.referrer_policy` | string | | |
| `server.headers.permissions_policy` | string | | |
| `server.headers.cross_origin_opener_policy` | string | | |
| `server.headers.cross_origin_resource_policy` | string | | |
| `server.headers.extra_response_headers` | list of objects | | Extra headers set on every response, each `{header: ..., value: ...}`. `header` must be a valid header name and `value` must not contain a line break |
| `server.headers.remove_response_headers` | list | | Headers stripped from every response (e.g. `Server`); empty: none |
| `server.headers.hsts.enabled` | bool | `false` | Send `Strict-Transport-Security` on HTTPS requests |
| `server.headers.hsts.max_age` | duration | `8760h` | `0s`: the header isn't sent |
| `server.headers.hsts.include_subdomains` | bool | `true` | |
| `server.headers.hsts.preload` | bool | `false` | |
#### Frontend
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.frontend.dir` | string | | Directory of the frontend's build output, holding `index.html`; empty: the build embedded in the binary (see `npm run sync` in `web/`), or no frontend if it has none |
| `server.frontend.immutable_prefix` | string | | Where Vite emits content-hashed files, cached for a year; empty: `/assets/` |
| `server.frontend.index_cache_control` | string | | `Cache-Control` of HTML and other files without a hash; empty: `no-cache` |
#### Websockets
| Key | Type | Default | Notes |
|---|---|---|---|
| `server.websocket.enable_compression` | bool | `false` | permessage-deflate |
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
## Complex Types
A rate limit rule (`JSCRateLimitRule`) has `enabled`, `per_second` and `burst`.
It is off unless `enabled` and `per_second` is greater than 0; a `burst` below
1 means 1. It is used under several keys: `server.rate_limits.global`,
`server.rate_limits.api`, `server.websocket.connection_limit` and
`server.websocket.message_rate_limit`. Rate limits are implemented using the
token bucket algorithm.
Lists can be given as YAML/JSON lists or as a comma-separated string (e.g. in
an environment variable). A list of objects such as `server.headers.extra_response_headers`
is a YAML/JSON list of objects in a config file, or a JSON array in an
environment variable or `config set`:
`'[{"header":"X-Robots-Tag","value":"noindex"}]'`.
## Example Configuration
Example `config.yaml`:
```yaml
logging:
level: info
file: ""
database:
file: ./jobsearch.db
change_batch_retention: 720h
sources:
find_a_tender:
enabled: true
interval: 10m
fts_mock:
enabled: false
interval: 1m
api_root: http://localhost:8080/api/1.0
server:
host: 0.0.0.0
port: 9393
max_body_bytes: 2MB
proxy:
trusted: [10.0.0.0/8]
rate_limits:
api:
enabled: true
per_second: 20
burst: 40
headers:
allowed_hosts: [jobs.example.com]
extra_response_headers:
- header: X-Robots-Tag
value: noindex
remove_response_headers: [Server]
hsts:
enabled: true
```
## Sections and struct tags
Each section of the configuration is a plain struct. The reflection code in
`section.go` reads its struct tags, so a section needs no hand-written code
per key:
- `cfg:"key"` is the field's configuration key. Fields without it are not part
of the configuration.
- `default:"value"` is the field's default, parsed as if it were a string
passed to `SetValue` (e.g. `"10m"`, `"true"`).
- A tagged field whose type is a struct is a nested section, addressed as
`section.key`. An embedded struct has its fields promoted, as Go does;
every source embeds `JSCSourcesSource` for `enabled` and `interval`.
- `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a
`time.Duration` or a duration string in `SetValue`, and are given out as
strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
`YAML` and `String` too.
- `util.ByteSize` fields work the same way, with size strings such as
`"32KiB"` or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers
of 1024), and are given out exactly in the largest unit that divides them
(`"1MB"`). `-1` means no limit (`util.ByteSizeNoLimit`); no other size may
be negative. A setting for which no limit is unsafe rejects `-1`.
- `[]string` fields report `reflect.Slice` from `TypeOf` and accept either a
`[]string` or a comma-separated string such as `"a, b"` in `SetValue` (and so
in default tags). An empty list is stored as `nil`.
- A list of sections, `[]S` for a section struct `S` (e.g. `[]JSCServerHeadersExtra`),
is a single key reporting `reflect.Slice`. It is given out as a
`[]map[string]any`, one map per item, and `SetValue` accepts a `[]S`, a list
of maps (as Viper reads a YAML/JSON list of objects) or a JSON array string.
Each item starts from its defaults and its keys are set as `SetValue` sets
them. `S` implements `Section` and, like a type used under several keys, has
no `JSCK*` constant.
- A section can reject values by implementing `validate(key, value) error` on
its pointer type. `JSCLogging` does this for `level`, `JSCSourcesSource` for
`interval`, `JSCServer` for `max_header_bytes`, `JSCServerTimeouts` for
`read_header` (rejecting 0 or less) and `JSCServerWebSocket` for
`max_message_size` (rejecting `-1`), and `JSCServerHeadersExtra` for
`header` and `value`. Each item in a list of
sections is validated whole, so a key left out of an item is checked too.
Every section implements the `Section` interface defined in `section.go`:
`Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue`, each a one-line call into
`section.go` (some sections also have `Keys`). The methods sit under a
`// ---------- Section methods (see Section) ----------` banner in each
section's file. A section's `Flatten` keys are prefixed with its `JSCK*`
constant, except for a section type used under several keys, such as
`JSCRateLimitRule`: it has no constant and its keys are unprefixed.
`TestSections` walks `JobSearchConfig` and checks every
section, including embedded ones, implements `Section` and that its
`Flatten` keys match its `cfg` tag, and that its `Map`, `TypeOf`, `ValueOf`
and `SetValue` agree with the whole configuration's view of the same keys (the
items of a list of sections are checked the same way), so a new section is
checked without being listed anywhere.
## Adding a key or section
To add a key to an existing section, add a tagged field to its struct. Nothing
else is needed for it to be loaded, printed, and read or written by the
`config` CLI commands.
To add a new section:
1. Create a `conf_*.go` file with the section struct, a `JSCK*` constant for
its key, and an implementation of `Section` (copy an existing section's
methods). A type that will be used under several keys, like
`JSCRateLimitRule`, has no constant and passes `""` to `flattenOf`.
2. Add the struct as a tagged field of its parent: `JobSearchConfig` in
`config.go`, or `JSCSources` in `conf_sources.go` for a source,
etc. The field's `cfg` tag must match the `JSCK*` constant.
Nothing in `section.go` needs changing.
## API
- `Init(cfgFile, &conf)`: load the configuration, as described above.
- `conf.TypeOf(key)`, `conf.ValueOf(key)`, `conf.SetValue(key, value)`: read
or set one dotted key on the loaded struct. `SetValue` converts and
validates the value.
- `conf.Map()` (nested) and `conf.Flatten()` (dotted keys): the whole
configuration as a map.
- `conf.String()`, `conf.JSON()`, `conf.YAML()` and the matching `Print*`
methods: the configuration as text, with keys sorted.
- `StoreValue(&conf, key, value)` / `PurgeValue(&conf, key)`: set or remove a
key in the config file Viper loaded. Used by `config set` and
`config set --unset`. `StoreValue` validates the value before writing it,
and stores it as the field's own type. Only the file's own settings are
written back; defaults and environment variables never are.
- `LoadDefaultConfig`, `LoadToViper`, `LoadFromViper`: move values between
the struct defaults, Viper and a `JobSearchConfig`.
## Files
| File | Contents |
|---|---|
| `config.go` | `JobSearchConfig`, defaults, Viper loading, text output |
| `conf_database.go` | `database` section |
| `load.go` | `Init`, `.env` and config file loading |
| `modify.go` | `StoreValue` and `PurgeValue` |
| `section.go` | The `Section` interface and the struct tag reflection shared by every section |
| `conf_logging.go` | `logging` section and log level helpers |
| `conf_sources.go` | `sources` section and the shared source fields |
| `conf_source_*.go` | One file per source section |
| `conf_server.go` | `server` section |
| `conf_server_*.go` | One file per `server` sub-section (`csrf`, `frontend`, `headers`, `proxy`, `rate_limits`, `timeouts`, `tls`, `websocket`) |
| `conf_ratelimitrule.go` | `JSCRateLimitRule`, the rate limit rule shared by several keys |
| `print.go`, `debug.go` | Printing and debug logging of the configuration |
## Testing
Viper keeps its state in package globals, so tests that use it call
`resetViper(t)` to reset it before and after the test. Tests of `Init`
use `initDir`, which changes into a temp directory with its own `.env` and
`config.yaml`, so the real files in the repository are never read.
Reference in new issue View Git Blame Copy Permalink