job-search/internal/config/README.md

# config

Package config loads the JobSearch configuration into a `JobSearchConfig`
struct. Values come from struct-tag defaults, a YAML or JSON config file and
environment variables, merged through [Viper](https://github.com/spf13/viper).


## Loading order
`Init(cfgFile, &conf)` builds the configuration in this order:

1. Loads `.env` from the working directory into the environment. A missing
   `.env` file is an error.
2. Sets the defaults from the `default` struct tags as Viper defaults.
3. Reads the config file: `cfgFile` if given, otherwise the first
   `config.yaml` found in `/etc/job-search` then the working directory. If no
   file is found a warning is logged and the defaults are used. A file that
   exists but can't be parsed is an error.
4. Reads environment variables prefixed with `JOBSEARCH_`. Dots in a key
   become underscores, so `logging.level` is read from
   `JOBSEARCH_LOGGING_LEVEL`.
5. Copies every known key from Viper into `conf`. A value that fails
   validation (e.g. an unknown log level) is logged as a warning and the
   previous value is kept.

When more than one place sets a key, environment variables win over the
config file, which wins over the defaults. A `debug` or `error` log level
already set on `conf` before `Init` (from the `--debug` and `--quiet` flags)
is kept.


## Keys

| Key | Type | Default | Notes |
|---|---|---|---|
| `logging.level` | string | `warn` | One of `debug`, `info`, `warn`, `error` |
| `logging.file` | string | | Log file for `fetch service` (unless `--foreground`); other commands log to the terminal |
| `database.file` | string | | Path to the SQLite database file, used by the `migrate` commands |
| `sources.<source>.enabled` | bool | `false` | For every source |
| `sources.<source>.interval` | duration | `10m` | For every source; e.g. `30s`, `1h` |
| `sources.find_a_tender.api_root` | string | `https://www.find-tender.service.gov.uk/api/1.0` | URL of the Find a Tender API |
| `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`) |
| `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown |
| `server.host` | string | `127.0.0.1` | Address to listen on |
| `server.port` | int | `9393` | Port to listen on |
| `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for clients to close |
| `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's rate limiter is kept after its last request |
| `server.websocket.enable_compression` | bool | `false` | permessage-deflate |
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |

The current sources are `find_a_tender`, `fts_mock` and `sleeper`. See 
"Addition of a new opportunities source" in the top level README for how to add
a new source.

A rate limit rule (`JSCRateLimitRule`) has `enabled`, `per_second` and `burst`.
It is used under `server.websocket.connection_limit` and
`server.websocket.message_rate_limit`.

Example `config.yaml`:
```yaml
logging:
    level: info
    file: ""
database:
    file: ./jobsearch.db
sources:
    find_a_tender:
        enabled: true
        interval: 10m
    fts_mock:
        enabled: false
        interval: 1m
        api_root: http://localhost:8080/api/1.0
```


## Sections and struct tags
Each section of the configuration is a plain struct. The reflection code in
`section.go` reads its struct tags, so a section needs no hand-written code
per key:

- `cfg:"key"` is the field's configuration key. Fields without it are not part
  of the configuration.
- `default:"value"` is the field's default, parsed as if it were a string
  passed to `SetValue` (e.g. `"10m"`, `"true"`).
- A tagged field whose type is a struct is a nested section, addressed as
  `section.key`. An embedded struct has its fields promoted, as Go does;
  every source embeds `JobSearchConfigSource` for `enabled` and `interval`.
- `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a
  `time.Duration` or a duration string in `SetValue`, and are given out as
  strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
  `YAML` and `String` too.
- `util.ByteSize` fields work the same way, with size strings such as `"32KiB"`
  or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers of 1024),
  and are given out exactly in the largest unit that divides them (`"1MB"`).
- A section can reject values by implementing `validate(key, value) error` on
  its pointer type. `JobSearchConfigLogging` does this for `level`.

Every section implements the `Section` interface defined in `section.go`:
`Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue`, each a one-line call into
`section.go` (some sections also have `Keys`). The methods sit under a
`// ---------- Section methods (see Section) ----------` banner in each
section's file. `TestSections` walks `JobSearchConfig` and checks every
section, including embedded ones, implements `Section` and that its
`Flatten` keys match its `cfg` tag, so a new section is checked without being
listed anywhere.


## Adding a key or section
To add a key to an existing section, add a tagged field to its struct. Nothing
else is needed for it to be loaded, printed, and read or written by the
`config` CLI commands.

To add a new section:
1. Create a `conf_*.go` file with the section struct, a `JSCK*` constant for
   its key, and an implementation of `Section` (copy an existing section's
   methods).
2. Add the struct as a tagged field of its parent: `JobSearchConfig` in
   `config.go`, or `JobSearchConfigSources` in `conf_sources.go` for a source,
   etc. The field's `cfg` tag must match the `JSCK*` constant.

Nothing in `section.go` needs changing.


## API
- `Init(cfgFile, &conf)`: load the configuration, as described above.
- `conf.TypeOf(key)`, `conf.ValueOf(key)`, `conf.SetValue(key, value)`: read
  or set one dotted key on the loaded struct. `SetValue` converts and
  validates the value.
- `conf.Map()` (nested) and `conf.Flatten()` (dotted keys): the whole
  configuration as a map.
- `conf.String()`, `conf.JSON()`, `conf.YAML()` and the matching `Print*`
  methods: the configuration as text, with keys sorted.
- `StoreValue(&conf, key, value)` / `PurgeValue(&conf, key)`: set or remove a
  key in the config file Viper loaded. Used by `config set` and
  `config set --unset`. `StoreValue` validates the value before writing it,
  and stores it as the field's own type. Only the file's own settings are
  written back; defaults and environment variables never are.
- `LoadDefaultConfig`, `LoadToViper`, `LoadFromViper`: move values between
  the struct defaults, Viper and a `JobSearchConfig`.


## Files

| File | Contents |
|---|---|
| `config.go` | `JobSearchConfig`, defaults, Viper loading, text output |
| `conf_database.go` | `database` section |
| `load.go` | `Init`, `.env` and config file loading |
| `modify.go` | `StoreValue` and `PurgeValue` |
| `section.go` | The `Section` interface and the struct tag reflection shared by every section |
| `conf_logging.go` | `logging` section and log level helpers |
| `conf_sources.go` | `sources` section and the shared source fields |
| `conf_source_*.go` | One file per source section |
| `conf_server.go` | `server` section |
| `conf_server_*.go` | One file per `server` sub-section (`rate_limits`, `timeouts`, `websocket`) |
| `conf_ratelimitrule.go` | `JSCRateLimitRule`, the rate limit rule shared by several keys |
| `print.go`, `debug.go` | Printing and debug logging of the configuration |


## Testing
Viper keeps its state in package globals, so tests that use it call
`resetViper(t)` to reset it before and after the test. Tests of `Init`
use `initDir`, which changes into a temp directory with its own `.env` and
`config.yaml`, so the real files in the repository are never read.