bwilliams
67215a6aaa
CODE_CONVENTIONS.md sets out the conventions for Go code in this repository: naming, layout, comments, errors, logging, configuration and tests. .golangci.json enables the linters that check the parts that can be checked (each limited to the rules the conventions use); the rest is applied by hand. Applied across the codebase: - Naming: full-word names in application code, initialisms in one case (BuyerID, PrintJSON, PrintYAML, ocdsGetBuyerID), no names shadowing built-ins (min/max are minGap/maxGap), mustRaw renamed rawJSON as it returns nil rather than panicking, and config.ConfigLevelToLogLevel renamed LevelToLogLevel. - Layout: lines kept to 80 characters where possible, assignments taken out of if and for statements that don't fit, wrapped arguments all on one line or one per line, steps separated by blank lines, no else after a return, function and struct literals wrapped. - Comments: doc comments on every exported name (including each section's Map, Flatten, TypeOf, ValueOf and SetValue), section banners, British spelling. - Errors: a created error starts with its package name, a wrapping error adds only what was being done, sentinels in one commented var block, errors.Is and errors.As in place of == and type assertions, and no err := shadowing of a named result. That fixes the mock server losing its access log's close error. - Logging: the fetcher passes its workers an untagged logger, so findatender's lines are tagged with their own component rather than with fetcher's as well; cmd/mock-fts logs "stopped" with slog. - Configuration: util.WithTimeout (moved from the websocket package) applies a timeout of 0 or less as no limit, used for the websocket write and ping, TCP close and tcpserver shutdown timeouts; every setting documents what 0 or empty means, including the rate limit rule's keys in internal/config/README.md. - Tests: related tests merged into table-driven ones with named cases. Separately from CODE_CONVENTIONS.md, internal/store/README.md gains a soft SQL style convention for the queries in internal/store/queries.go, and the queries are reformatted to follow it.
178 lines
9.1 KiB
Markdown
178 lines
9.1 KiB
Markdown
# config
|
|
|
|
Package config loads the JobSearch configuration into a `JobSearchConfig`
|
|
struct. Values come from struct-tag defaults, a YAML or JSON config file and
|
|
environment variables, merged through [Viper](https://github.com/spf13/viper).
|
|
|
|
|
|
## Loading order
|
|
`Init(cfgFile, &conf)` builds the configuration in this order:
|
|
|
|
1. Loads `.env` from the working directory into the environment. A missing
|
|
`.env` file is an error.
|
|
2. Sets the defaults from the `default` struct tags as Viper defaults.
|
|
3. Reads the config file: `cfgFile` if given, otherwise the first
|
|
`config.yaml` found in `/etc/job-search` then the working directory. If no
|
|
file is found a warning is logged and the defaults are used. A file that
|
|
exists but can't be parsed is an error.
|
|
4. Reads environment variables prefixed with `JOBSEARCH_`. Dots in a key
|
|
become underscores, so `logging.level` is read from
|
|
`JOBSEARCH_LOGGING_LEVEL`.
|
|
5. Copies every known key from Viper into `conf`. A value that fails
|
|
validation (e.g. an unknown log level) is logged as a warning and the
|
|
previous value is kept.
|
|
|
|
When more than one place sets a key, environment variables win over the
|
|
config file, which wins over the defaults. A `debug` or `error` log level
|
|
already set on `conf` before `Init` (from the `--debug` and `--quiet` flags)
|
|
is kept.
|
|
|
|
|
|
## Keys
|
|
|
|
| Key | Type | Default | Notes |
|
|
|---|---|---|---|
|
|
| `logging.level` | string | `warn` | One of `debug`, `info`, `warn`, `error` |
|
|
| `logging.file` | string | | Log file for `fetch service` (unless `--foreground`); other commands log to the terminal. Empty: log to the terminal |
|
|
| `database.file` | string | | Path to the SQLite database file, used by the `migrate` commands. Empty: commands that need the database fail |
|
|
| `sources.<source>.enabled` | bool | `false` | For every source |
|
|
| `sources.<source>.interval` | duration | `10m` | For every source; e.g. `30s`, `1h`. Must be greater than 0; turn a source off with `enabled` |
|
|
| `sources.find_a_tender.api_root` | string | `https://www.find-tender.service.gov.uk/api/1.0` | URL of the Find a Tender API |
|
|
| `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`); must be set when the source is enabled |
|
|
| `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown |
|
|
| `server.host` | string | `127.0.0.1` | Address to listen on; empty: every interface |
|
|
| `server.port` | int | `9393` | Port to listen on; `0`: any free port |
|
|
| `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for clients to close; `0s`: until they have |
|
|
| `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's rate limiter is kept after its last request; `0s`: for ever |
|
|
| `server.websocket.enable_compression` | bool | `false` | permessage-deflate |
|
|
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory |
|
|
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
|
|
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
|
|
| `server.websocket.*.enabled` | bool | `false` | In a rate limit rule: whether it applies |
|
|
| `server.websocket.*.per_second` | int | `50` | In a rate limit rule: sustained rate. `0` or less: the rule is off |
|
|
| `server.websocket.*.burst` | int | `100` | In a rate limit rule: largest burst. Below `1`: 1 |
|
|
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
|
|
| `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings |
|
|
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
|
|
|
|
The current sources are `find_a_tender`, `fts_mock` and `sleeper`. See
|
|
"Addition of a new opportunities source" in the top level README for how to add
|
|
a new source.
|
|
|
|
A rate limit rule (`JSCRateLimitRule`) has `enabled`, `per_second` and `burst`.
|
|
It is off unless `enabled` and `per_second` is greater than 0; a `burst` below
|
|
1 means 1.
|
|
It is used under `server.websocket.connection_limit` and
|
|
`server.websocket.message_rate_limit`.
|
|
|
|
Example `config.yaml`:
|
|
```yaml
|
|
logging:
|
|
level: info
|
|
file: ""
|
|
database:
|
|
file: ./jobsearch.db
|
|
sources:
|
|
find_a_tender:
|
|
enabled: true
|
|
interval: 10m
|
|
fts_mock:
|
|
enabled: false
|
|
interval: 1m
|
|
api_root: http://localhost:8080/api/1.0
|
|
```
|
|
|
|
|
|
## Sections and struct tags
|
|
Each section of the configuration is a plain struct. The reflection code in
|
|
`section.go` reads its struct tags, so a section needs no hand-written code
|
|
per key:
|
|
|
|
- `cfg:"key"` is the field's configuration key. Fields without it are not part
|
|
of the configuration.
|
|
- `default:"value"` is the field's default, parsed as if it were a string
|
|
passed to `SetValue` (e.g. `"10m"`, `"true"`).
|
|
- A tagged field whose type is a struct is a nested section, addressed as
|
|
`section.key`. An embedded struct has its fields promoted, as Go does;
|
|
every source embeds `JSCSourcesSource` for `enabled` and `interval`.
|
|
- `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a
|
|
`time.Duration` or a duration string in `SetValue`, and are given out as
|
|
strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
|
|
`YAML` and `String` too.
|
|
- `util.ByteSize` fields work the same way, with size strings such as `"32KiB"`
|
|
or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers of 1024),
|
|
and are given out exactly in the largest unit that divides them (`"1MB"`).
|
|
`-1` means no limit (`util.ByteSizeNoLimit`); no other size may be
|
|
negative. A setting for which no limit is unsafe rejects `-1`.
|
|
- A section can reject values by implementing `validate(key, value) error` on
|
|
its pointer type. `JSCLogging` does this for `level`, and
|
|
`JSCServerWebSocket` for `max_message_size` (rejecting `-1`).
|
|
|
|
Every section implements the `Section` interface defined in `section.go`:
|
|
`Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue`, each a one-line call into
|
|
`section.go` (some sections also have `Keys`). The methods sit under a
|
|
`// ---------- Section methods (see Section) ----------` banner in each
|
|
section's file. `TestSections` walks `JobSearchConfig` and checks every
|
|
section, including embedded ones, implements `Section` and that its
|
|
`Flatten` keys match its `cfg` tag, so a new section is checked without being
|
|
listed anywhere.
|
|
|
|
|
|
## Adding a key or section
|
|
To add a key to an existing section, add a tagged field to its struct. Nothing
|
|
else is needed for it to be loaded, printed, and read or written by the
|
|
`config` CLI commands.
|
|
|
|
To add a new section:
|
|
1. Create a `conf_*.go` file with the section struct, a `JSCK*` constant for
|
|
its key, and an implementation of `Section` (copy an existing section's
|
|
methods).
|
|
2. Add the struct as a tagged field of its parent: `JobSearchConfig` in
|
|
`config.go`, or `JSCSources` in `conf_sources.go` for a source,
|
|
etc. The field's `cfg` tag must match the `JSCK*` constant.
|
|
|
|
Nothing in `section.go` needs changing.
|
|
|
|
|
|
## API
|
|
- `Init(cfgFile, &conf)`: load the configuration, as described above.
|
|
- `conf.TypeOf(key)`, `conf.ValueOf(key)`, `conf.SetValue(key, value)`: read
|
|
or set one dotted key on the loaded struct. `SetValue` converts and
|
|
validates the value.
|
|
- `conf.Map()` (nested) and `conf.Flatten()` (dotted keys): the whole
|
|
configuration as a map.
|
|
- `conf.String()`, `conf.JSON()`, `conf.YAML()` and the matching `Print*`
|
|
methods: the configuration as text, with keys sorted.
|
|
- `StoreValue(&conf, key, value)` / `PurgeValue(&conf, key)`: set or remove a
|
|
key in the config file Viper loaded. Used by `config set` and
|
|
`config set --unset`. `StoreValue` validates the value before writing it,
|
|
and stores it as the field's own type. Only the file's own settings are
|
|
written back; defaults and environment variables never are.
|
|
- `LoadDefaultConfig`, `LoadToViper`, `LoadFromViper`: move values between
|
|
the struct defaults, Viper and a `JobSearchConfig`.
|
|
|
|
|
|
## Files
|
|
|
|
| File | Contents |
|
|
|---|---|
|
|
| `config.go` | `JobSearchConfig`, defaults, Viper loading, text output |
|
|
| `conf_database.go` | `database` section |
|
|
| `load.go` | `Init`, `.env` and config file loading |
|
|
| `modify.go` | `StoreValue` and `PurgeValue` |
|
|
| `section.go` | The `Section` interface and the struct tag reflection shared by every section |
|
|
| `conf_logging.go` | `logging` section and log level helpers |
|
|
| `conf_sources.go` | `sources` section and the shared source fields |
|
|
| `conf_source_*.go` | One file per source section |
|
|
| `conf_server.go` | `server` section |
|
|
| `conf_server_*.go` | One file per `server` sub-section (`rate_limits`, `timeouts`, `websocket`) |
|
|
| `conf_ratelimitrule.go` | `JSCRateLimitRule`, the rate limit rule shared by several keys |
|
|
| `print.go`, `debug.go` | Printing and debug logging of the configuration |
|
|
|
|
|
|
## Testing
|
|
Viper keeps its state in package globals, so tests that use it call
|
|
`resetViper(t)` to reset it before and after the test. Tests of `Init`
|
|
use `initDir`, which changes into a temp directory with its own `.env` and
|
|
`config.yaml`, so the real files in the repository are never read.
|