job-search/internal/config/README.md

config

Package config loads the JobSearch configuration into a JobSearchConfig struct. Values come from struct-tag defaults, a YAML or JSON config file and environment variables, merged through Viper.

Loading order

Init(cfgFile, &conf) builds the configuration in this order:

1. Loads .env from the working directory into the environment. A missing .env file is an error.
2. Sets the defaults from the default struct tags as Viper defaults.
3. Reads the config file: cfgFile if given, otherwise the first config.yaml found in /etc/job-search then the working directory. If no file is found a warning is logged and the defaults are used. A file that exists but can't be parsed is an error.
4. Reads environment variables prefixed with JOBSEARCH_. Dots in a key become underscores, so logging.level is read from JOBSEARCH_LOGGING_LEVEL.
5. Copies every known key from Viper into conf. A value that fails validation (e.g. an unknown log level) is logged as a warning and the previous value is kept.

When more than one place sets a key, environment variables win over the config file, which wins over the defaults. A debug or error log level already set on conf before Init (from the --debug and --quiet flags) is kept.

Keys

Key	Type	Default	Notes
logging.level	string	warn	One of debug, info, warn, error
logging.file	string		Log file for fetch service (unless --foreground); other commands log to the terminal. Empty: log to the terminal
database.file	string		Path to the SQLite database file, used by the migrate commands. Empty: commands that need the database fail
sources.<source>.enabled	bool	false	For every source
sources.<source>.interval	duration	10m	For every source; e.g. 30s, 1h. Must be greater than 0; turn a source off with enabled
sources.find_a_tender.api_root	string	https://www.find-tender.service.gov.uk/api/1.0	URL of the Find a Tender API
sources.fts_mock.api_root	string		URL of the mock FTS server (see internal/mock); must be set when the source is enabled
sources.sleeper.block_shutdown	bool	false	Test source: if true, its sleep ignores shutdown
server.host	string	127.0.0.1	Address to listen on; empty: every interface
server.port	int	9393	Port to listen on; 0: any free port
server.timeouts.shutdown	duration	15s	How long shutdown waits for clients to close; 0s: until they have
server.rate_limits.idle_ttl	duration	10m	How long a client's rate limiter is kept after its last request; 0s: for ever
server.websocket.enable_compression	bool	false	permessage-deflate
server.websocket.max_message_size	byte size	32KiB	Largest message read from a client; 0B for 32KiB. -1 (no limit) is rejected, as a client could then exhaust the server's memory
server.websocket.connection_limit.*			A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for server.rate_limits.idle_ttl
server.websocket.message_rate_limit.*			A rate limit rule for messages from each connection
server.websocket.*.enabled	bool	false	In a rate limit rule: whether it applies
server.websocket.*.per_second	int	50	In a rate limit rule: sustained rate. 0 or less: the rule is off
server.websocket.*.burst	int	100	In a rate limit rule: largest burst. Below 1: 1
server.websocket.write_timeout	duration	10s	0s for none
server.websocket.ping_interval	duration	30s	How long a connection may be idle before it is pinged; 0s: no pings
server.websocket.pong_timeout	duration	10s	0s for none

The current sources are find_a_tender, fts_mock and sleeper. See "Addition of a new opportunities source" in the top level README for how to add a new source.

A rate limit rule (JSCRateLimitRule) has enabled, per_second and burst. It is off unless enabled and per_second is greater than 0; a burst below 1 means 1. It is used under server.websocket.connection_limit and server.websocket.message_rate_limit.

Example config.yaml:

logging:
    level: info
    file: ""
database:
    file: ./jobsearch.db
sources:
    find_a_tender:
        enabled: true
        interval: 10m
    fts_mock:
        enabled: false
        interval: 1m
        api_root: http://localhost:8080/api/1.0

Sections and struct tags

Each section of the configuration is a plain struct. The reflection code in section.go reads its struct tags, so a section needs no hand-written code per key:

• cfg:"key" is the field's configuration key. Fields without it are not part of the configuration.
• default:"value" is the field's default, parsed as if it were a string passed to SetValue (e.g. "10m", "true").
• A tagged field whose type is a struct is a nested section, addressed as section.key. An embedded struct has its fields promoted, as Go does; every source embeds JSCSourcesSource for enabled and interval.
• time.Duration fields report reflect.String from TypeOf, accept either a time.Duration or a duration string in SetValue, and are given out as strings ("10m0s") by Map, Flatten and ValueOf, and so by JSON, YAML and String too.
• util.ByteSize fields work the same way, with size strings such as "32KiB" or "1MB" (KB, MB, ... are powers of 1000; KiB, MiB, ... powers of 1024), and are given out exactly in the largest unit that divides them ("1MB"). -1 means no limit (util.ByteSizeNoLimit); no other size may be negative. A setting for which no limit is unsafe rejects -1.
• A section can reject values by implementing validate(key, value) error on its pointer type. JSCLogging does this for level, and JSCServerWebSocket for max_message_size (rejecting -1).

Every section implements the Section interface defined in section.go: Map, Flatten, TypeOf, ValueOf and SetValue, each a one-line call into section.go (some sections also have Keys). The methods sit under a // ---------- Section methods (see Section) ---------- banner in each section's file. TestSections walks JobSearchConfig and checks every section, including embedded ones, implements Section and that its Flatten keys match its cfg tag, so a new section is checked without being listed anywhere.

Adding a key or section

To add a key to an existing section, add a tagged field to its struct. Nothing else is needed for it to be loaded, printed, and read or written by the config CLI commands.

To add a new section:

1. Create a conf_*.go file with the section struct, a JSCK* constant for its key, and an implementation of Section (copy an existing section's methods).
2. Add the struct as a tagged field of its parent: JobSearchConfig in config.go, or JSCSources in conf_sources.go for a source, etc. The field's cfg tag must match the JSCK* constant.

Nothing in section.go needs changing.

API

• Init(cfgFile, &conf): load the configuration, as described above.
• conf.TypeOf(key), conf.ValueOf(key), conf.SetValue(key, value): read or set one dotted key on the loaded struct. SetValue converts and validates the value.
• conf.Map() (nested) and conf.Flatten() (dotted keys): the whole configuration as a map.
• conf.String(), conf.JSON(), conf.YAML() and the matching Print* methods: the configuration as text, with keys sorted.
• StoreValue(&conf, key, value) / PurgeValue(&conf, key): set or remove a key in the config file Viper loaded. Used by config set and config set --unset. StoreValue validates the value before writing it, and stores it as the field's own type. Only the file's own settings are written back; defaults and environment variables never are.
• LoadDefaultConfig, LoadToViper, LoadFromViper: move values between the struct defaults, Viper and a JobSearchConfig.

Files

File	Contents
config.go	JobSearchConfig, defaults, Viper loading, text output
conf_database.go	database section
load.go	Init, .env and config file loading
modify.go	StoreValue and PurgeValue
section.go	The Section interface and the struct tag reflection shared by every section
conf_logging.go	logging section and log level helpers
conf_sources.go	sources section and the shared source fields
conf_source_*.go	One file per source section
conf_server.go	server section
conf_server_*.go	One file per server sub-section (rate_limits, timeouts, websocket)
conf_ratelimitrule.go	JSCRateLimitRule, the rate limit rule shared by several keys
print.go, debug.go	Printing and debug logging of the configuration

Testing

Viper keeps its state in package globals, so tests that use it call resetViper(t) to reset it before and after the test. Tests of Init use initDir, which changes into a temp directory with its own .env and config.yaml, so the real files in the repository are never read.