# config Package config loads the JobSearch configuration into a `JobSearchConfig` struct. Values come from struct-tag defaults, a YAML or JSON config file and environment variables, merged through [Viper](https://github.com/spf13/viper). ## Loading order `Init(cfgFile, &conf)` builds the configuration in this order: 1. Loads `.env` from the working directory into the environment. A missing `.env` file is an error. 2. Sets the defaults from the `default` struct tags as Viper defaults. 3. Reads the config file: `cfgFile` if given, otherwise the first `config.yaml` found in `/etc/job-search` then the working directory. If no file is found a warning is logged and the defaults are used. A file that exists but can't be parsed is an error. 4. Reads environment variables prefixed with `JOBSEARCH_`. Dots in a key become underscores, so `logging.level` is read from `JOBSEARCH_LOGGING_LEVEL`. 5. Copies every known key from Viper into `conf`. A value that fails validation (e.g. an unknown log level) is logged as a warning and the previous value is kept. When more than one place sets a key, environment variables win over the config file, which wins over the defaults. A `debug` or `error` log level already set on `conf` before `Init` (from the `--debug` and `--quiet` flags) is kept. ## Keys | Key | Type | Default | Notes | |---|---|---|---| | `logging.level` | string | `warn` | One of `debug`, `info`, `warn`, `error` | | `logging.file` | string | | Log file for `fetch service` (unless `--foreground`); other commands log to the terminal. Empty: log to the terminal | | `database.file` | string | | Path to the SQLite database file, used by the `migrate` commands. Empty: commands that need the database fail | | `sources..enabled` | bool | `false` | For every source | | `sources..interval` | duration | `10m` | For every source; e.g. `30s`, `1h`. Must be greater than 0; turn a source off with `enabled` | | `sources.find_a_tender.api_root` | string | `https://www.find-tender.service.gov.uk/api/1.0` | URL of the Find a Tender API | | `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`); must be set when the source is enabled | | `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown | | `server.host` | string | `127.0.0.1` | Address to listen on; empty: every interface | | `server.port` | int | `9393` | Port to listen on; `0`: any free port | | `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for clients to close; `0s`: until they have | | `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's rate limiter is kept after its last request; `0s`: for ever | | `server.websocket.enable_compression` | bool | `false` | permessage-deflate | | `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory | | `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` | | `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection | | `server.websocket.*.enabled` | bool | `false` | In a rate limit rule: whether it applies | | `server.websocket.*.per_second` | int | `50` | In a rate limit rule: sustained rate. `0` or less: the rule is off | | `server.websocket.*.burst` | int | `100` | In a rate limit rule: largest burst. Below `1`: 1 | | `server.websocket.write_timeout` | duration | `10s` | `0s` for none | | `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings | | `server.websocket.pong_timeout` | duration | `10s` | `0s` for none | The current sources are `find_a_tender`, `fts_mock` and `sleeper`. See "Addition of a new opportunities source" in the top level README for how to add a new source. A rate limit rule (`JSCRateLimitRule`) has `enabled`, `per_second` and `burst`. It is off unless `enabled` and `per_second` is greater than 0; a `burst` below 1 means 1. It is used under `server.websocket.connection_limit` and `server.websocket.message_rate_limit`. Example `config.yaml`: ```yaml logging: level: info file: "" database: file: ./jobsearch.db sources: find_a_tender: enabled: true interval: 10m fts_mock: enabled: false interval: 1m api_root: http://localhost:8080/api/1.0 ``` ## Sections and struct tags Each section of the configuration is a plain struct. The reflection code in `section.go` reads its struct tags, so a section needs no hand-written code per key: - `cfg:"key"` is the field's configuration key. Fields without it are not part of the configuration. - `default:"value"` is the field's default, parsed as if it were a string passed to `SetValue` (e.g. `"10m"`, `"true"`). - A tagged field whose type is a struct is a nested section, addressed as `section.key`. An embedded struct has its fields promoted, as Go does; every source embeds `JSCSourcesSource` for `enabled` and `interval`. - `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a `time.Duration` or a duration string in `SetValue`, and are given out as strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`, `YAML` and `String` too. - `util.ByteSize` fields work the same way, with size strings such as `"32KiB"` or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers of 1024), and are given out exactly in the largest unit that divides them (`"1MB"`). `-1` means no limit (`util.ByteSizeNoLimit`); no other size may be negative. A setting for which no limit is unsafe rejects `-1`. - A section can reject values by implementing `validate(key, value) error` on its pointer type. `JSCLogging` does this for `level`, and `JSCServerWebSocket` for `max_message_size` (rejecting `-1`). Every section implements the `Section` interface defined in `section.go`: `Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue`, each a one-line call into `section.go` (some sections also have `Keys`). The methods sit under a `// ---------- Section methods (see Section) ----------` banner in each section's file. `TestSections` walks `JobSearchConfig` and checks every section, including embedded ones, implements `Section` and that its `Flatten` keys match its `cfg` tag, so a new section is checked without being listed anywhere. ## Adding a key or section To add a key to an existing section, add a tagged field to its struct. Nothing else is needed for it to be loaded, printed, and read or written by the `config` CLI commands. To add a new section: 1. Create a `conf_*.go` file with the section struct, a `JSCK*` constant for its key, and an implementation of `Section` (copy an existing section's methods). 2. Add the struct as a tagged field of its parent: `JobSearchConfig` in `config.go`, or `JSCSources` in `conf_sources.go` for a source, etc. The field's `cfg` tag must match the `JSCK*` constant. Nothing in `section.go` needs changing. ## API - `Init(cfgFile, &conf)`: load the configuration, as described above. - `conf.TypeOf(key)`, `conf.ValueOf(key)`, `conf.SetValue(key, value)`: read or set one dotted key on the loaded struct. `SetValue` converts and validates the value. - `conf.Map()` (nested) and `conf.Flatten()` (dotted keys): the whole configuration as a map. - `conf.String()`, `conf.JSON()`, `conf.YAML()` and the matching `Print*` methods: the configuration as text, with keys sorted. - `StoreValue(&conf, key, value)` / `PurgeValue(&conf, key)`: set or remove a key in the config file Viper loaded. Used by `config set` and `config set --unset`. `StoreValue` validates the value before writing it, and stores it as the field's own type. Only the file's own settings are written back; defaults and environment variables never are. - `LoadDefaultConfig`, `LoadToViper`, `LoadFromViper`: move values between the struct defaults, Viper and a `JobSearchConfig`. ## Files | File | Contents | |---|---| | `config.go` | `JobSearchConfig`, defaults, Viper loading, text output | | `conf_database.go` | `database` section | | `load.go` | `Init`, `.env` and config file loading | | `modify.go` | `StoreValue` and `PurgeValue` | | `section.go` | The `Section` interface and the struct tag reflection shared by every section | | `conf_logging.go` | `logging` section and log level helpers | | `conf_sources.go` | `sources` section and the shared source fields | | `conf_source_*.go` | One file per source section | | `conf_server.go` | `server` section | | `conf_server_*.go` | One file per `server` sub-section (`rate_limits`, `timeouts`, `websocket`) | | `conf_ratelimitrule.go` | `JSCRateLimitRule`, the rate limit rule shared by several keys | | `print.go`, `debug.go` | Printing and debug logging of the configuration | ## Testing Viper keeps its state in package globals, so tests that use it call `resetViper(t)` to reset it before and after the test. Tests of `Init` use `initDir`, which changes into a temp directory with its own `.env` and `config.yaml`, so the real files in the repository are never read.