job-search/internal/config/README.md

config

Package config loads the JobSearch configuration into a JobSearchConfig struct. Values come from struct-tag defaults, a YAML or JSON config file and environment variables, merged through Viper.

Loading order

Init(cfgFile, &conf) builds the configuration in this order:

1. Loads .env from the working directory into the environment. A missing .env file is an error.
2. Sets the defaults from the default struct tags as Viper defaults.
3. Reads the config file: cfgFile if given, otherwise the first config.yaml found in /etc/job-search then the working directory. If no file is found a warning is logged and the defaults are used. A file that exists but can't be parsed is an error.
4. Reads environment variables prefixed with JOBSEARCH_. Dots in a key become underscores, so logging.level is read from JOBSEARCH_LOGGING_LEVEL.
5. Copies every known key from Viper into conf. A value that fails validation (e.g. an unknown log level) is logged as a warning and the previous value is kept.

When more than one place sets a key, environment variables win over the config file, which wins over the defaults.

Configuration Keys

Configuration is managed in sections:

Key	Section description
logging	Paths to log files and log level control
database	Database connection/file details
sources	Opportunity source settings
server	Server configuration including CSRF protection, Websockets, Proxy handling and rate limits

Logging

Key	Type	Default	Notes
logging.level	string	warn	One of debug, info, warn, error
logging.file	string		Log file (JSON lines) for long-running commands such as fetcher service. Empty: log to the terminal

Logging always starts on the terminal. Long-running commands switch to the file in logging.file once the configuration is loaded, unless it is empty (they then stay on the terminal, with a warning) or --foreground is passed. All other commands log to the terminal.

The --debug and --quiet flags force the log level to debug and error respectively, ignoring the logging.level configuration. --debug takes priority over --quiet.

Database

Key	Type	Default	Notes
database.file	string		Path to the SQLite database file. Empty: commands that need the database fail
database.change_batch_retention	duration	720h	How long a batch of changed opportunities is kept for clients to catch up from (?changed_after=); older ones are deleted as new ones are recorded. A client whose batch has gone reloads the whole list. 0 or less: kept for ever

Database is currently restricted to SQLite v3. At this point, no other engine is supported.

Opportunity Sources

Key	Type	Default	Notes
sources.skip_rules_file	string		JSON file of rules that mark new opportunities Skipped (see internal/skiprules); read again when it changes. Empty: none. Missing or invalid: fetcher service and serve refuse to start
sources.<source>.enabled	bool	false	For every source
sources.<source>.interval	duration	10m	For every source; e.g. 30s, 1h. Must be greater than 0; turn a source off with enabled
sources.find_a_tender.api_root	string	https://www.find-tender.service.gov.uk/api/1.0	URL of the Find a Tender API. Empty: the source fails to start if enabled, and fetcher fts-once fails
sources.find_a_tender.rate_limit_retries	int	5	How many times a request refused for the API's rate limit is waited out (for its Retry-After) and retried. 0 fails at the first; can't be negative. fetcher fts-once --ignore-rate-limit uses 0
sources.fts_mock.api_root	string		URL of the mock FTS server (see internal/mock). Empty: the source fails to start if enabled
sources.fts_mock.rate_limit_retries	int	5	As for find_a_tender
sources.sleeper.block_shutdown	bool	false	Test source: if true, its sleep ignores shutdown

The current sources are find_a_tender, fts_mock and sleeper. See "Addition of a new opportunities source" in the top level README for how to add a new source.

Server

The server section is split into a number of sub-sections:

Key	Section description
server	Top level server configuration settings
server.timeouts	HTTP read, write and idle timeouts, and the shutdown timeout
server.rate_limits	Global (every request) and API rate limits. Also see server.websocket.connection_limit for WebSocket and plain TCP connection limits
server.proxy	PROXY protocol and proxy header settings
server.csrf	CSRF and cross-origin protection settings
server.tls	TLS certificate, minimum version and HTTP redirect
server.headers	HTTP header settings including policy headers
server.frontend	Where the frontend's build output is and how it is cached
server.websocket	Websocket configuration

Server

Key	Type	Default	Notes
server.host	string	127.0.0.1	Address to listen on; empty: every interface
server.port	int	9393	Port to listen on; 0: any free port
server.max_header_bytes	byte size	8KB	Largest request header block; 0B for 8KB. -1 (no limit) is rejected, as the HTTP server always has one
server.max_body_bytes	byte size	1MB	Largest request body; 0B for 1MB, -1 for no limit
server.access_log	string		File the access log is appended to, one line per request in the Combined Log Format; -: stdout; empty: no access log. serve --access-log overrides it

Timeouts

Key	Type	Default	Notes
server.timeouts.read_header	duration	5s	Reading a request's headers. Must be greater than 0, as a client could otherwise hold a connection open for ever
server.timeouts.read	duration	15s	Reading a whole request; 0s for none
server.timeouts.write	duration	30s	Writing a response; 0s for none
server.timeouts.idle	duration	2m	Waiting for the next request on a keep-alive connection; 0s for none
server.timeouts.shutdown	duration	15s	How long shutdown waits for open requests; 0s: until they finish

Rate Limits

Key	Type	Default	Notes
server.rate_limits.<rule>.enabled	bool	false	<rule> is global or api (see rate limit rules below)
server.rate_limits.<rule>.per_second	int	50	Sustained requests per second per client IP. 0 or less: the rule is off
server.rate_limits.<rule>.burst	int	100	Requests allowed in a burst. Below 1: 1
server.rate_limits.idle_ttl	duration	10m	How long a client's limiter is kept after its last request; 0s: for ever

PROXY Protocol and Proxy Headers

Key	Type	Default	Notes
server.proxy.trusted	list	127.0.0.1/32	IP addresses and CIDR ranges of proxies whose forwarding headers are trusted; empty: none
server.proxy.client_ip_headers	list	X-Forwarded-For, Forwarded, X-Real-IP	Headers read for the client IP from a trusted proxy, in priority order; empty: always the connection's address
server.proxy.strip_untrusted	bool	true	Remove forwarding headers from requests not from a trusted proxy
server.proxy.proxy_protocol.enabled	bool	false	Accept the PROXY protocol from trusted sources
server.proxy.proxy_protocol.trusted_sources	list		Addresses and CIDR ranges allowed to send a PROXY header; empty: server.proxy.trusted
server.proxy.proxy_protocol.required	bool	false	Reject connections from trusted sources without a PROXY header
server.proxy.proxy_protocol.read_header_timeout	duration	5s	How long to wait for the PROXY header; 0s for none

CSRF & Cross-Origin Protection

Key	Type	Default	Notes
server.csrf.cross_origin_protection.enabled	bool	true	Reject cross-origin unsafe requests (Sec-Fetch-Site/Origin)
server.csrf.cross_origin_protection.trusted_origins	list		Extra origins (https://admin.example.com) allowed to make them, and to open WebSockets; empty: none
server.csrf.tokens.enabled	bool	true	Require a signed double-submit token on unsafe requests
server.csrf.tokens.key	string		Hex HMAC key; empty: a random per-process key, so tokens don't survive a restart. job-search config generate-csrf-key stores one
server.csrf.tokens.cookie_name	string	js_csrf	Empty: js_csrf
server.csrf.tokens.header_name	string	X-CSRF-Token	Empty: X-CSRF-Token
server.csrf.tokens.token_ttl	duration	12h	0s: 12h
server.csrf.exempt_paths	list		ServeMux-style patterns skipped by both CSRF layers (e.g. /api/webhooks/); empty: none

TLS

Key	Type	Default	Notes
server.tls.enabled	bool	false	Serve HTTPS
server.tls.cert_file	string		Certificate file; must be set when TLS is enabled
server.tls.key_file	string		Private key file; must be set when TLS is enabled
server.tls.min_version	string	1.3	1.2 or 1.3; anything else, including empty, means 1.2
server.tls.redirect_http_port	int		Port of a plain HTTP listener that redirects to HTTPS; 0: none

Headers

Key	Type	Default	Notes
server.headers.allowed_hosts	list		Allowed Host headers, *.example.com supported; empty disables the check
server.headers.content_security_policy	string		Security response headers; empty leaves the header unset
server.headers.frame_options	string
server.headers.referrer_policy	string
server.headers.permissions_policy	string
server.headers.cross_origin_opener_policy	string
server.headers.cross_origin_resource_policy	string
server.headers.extra_response_headers	list of objects		Extra headers set on every response, each {header: ..., value: ...}. header must be a valid header name and value must not contain a line break
server.headers.remove_response_headers	list		Headers stripped from every response (e.g. Server); empty: none
server.headers.hsts.enabled	bool	false	Send Strict-Transport-Security on HTTPS requests
server.headers.hsts.max_age	duration	8760h	0s: the header isn't sent
server.headers.hsts.include_subdomains	bool	true
server.headers.hsts.preload	bool	false

Frontend

Key	Type	Default	Notes
server.frontend.dir	string		Directory of the frontend's build output, holding index.html; empty: the build embedded in the binary (see npm run sync in web/), or no frontend if it has none
server.frontend.immutable_prefix	string		Where Vite emits content-hashed files, cached for a year; empty: /assets/
server.frontend.index_cache_control	string		Cache-Control of HTML and other files without a hash; empty: no-cache

Websockets

Key	Type	Default	Notes
server.websocket.enable_compression	bool	false	permessage-deflate
server.websocket.max_message_size	byte size	32KiB	Largest message read from a client; 0B for 32KiB. -1 (no limit) is rejected, as a client could then exhaust the server's memory
server.websocket.connection_limit.*			A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for server.rate_limits.idle_ttl
server.websocket.message_rate_limit.*			A rate limit rule for messages from each connection
server.websocket.write_timeout	duration	10s	0s for none
server.websocket.ping_interval	duration	30s	How long a connection may be idle before it is pinged; 0s: no pings
server.websocket.pong_timeout	duration	10s	0s for none

Complex Types

A rate limit rule (JSCRateLimitRule) has enabled, per_second and burst. It is off unless enabled and per_second is greater than 0; a burst below 1 means 1. It is used under several keys: server.rate_limits.global, server.rate_limits.api, server.websocket.connection_limit and server.websocket.message_rate_limit. Rate limits are implemented using the token bucket algorithm.

Lists can be given as YAML/JSON lists or as a comma-separated string (e.g. in an environment variable). A list of objects such as server.headers.extra_response_headers is a YAML/JSON list of objects in a config file, or a JSON array in an environment variable or config set: '[{"header":"X-Robots-Tag","value":"noindex"}]'.

Example Configuration

Example config.yaml:

logging:
    level: info
    file: ""
database:
    file: ./jobsearch.db
    change_batch_retention: 720h
sources:
    find_a_tender:
        enabled: true
        interval: 10m
    fts_mock:
        enabled: false
        interval: 1m
        api_root: http://localhost:8080/api/1.0
server:
    host: 0.0.0.0
    port: 9393
    max_body_bytes: 2MB
    proxy:
        trusted: [10.0.0.0/8]
    rate_limits:
        api:
            enabled: true
            per_second: 20
            burst: 40
    headers:
        allowed_hosts: [jobs.example.com]
        extra_response_headers:
            - header: X-Robots-Tag
              value: noindex
        remove_response_headers: [Server]
        hsts:
            enabled: true

Sections and struct tags

Each section of the configuration is a plain struct. The reflection code in section.go reads its struct tags, so a section needs no hand-written code per key:

• cfg:"key" is the field's configuration key. Fields without it are not part of the configuration.
• default:"value" is the field's default, parsed as if it were a string passed to SetValue (e.g. "10m", "true").
• A tagged field whose type is a struct is a nested section, addressed as section.key. An embedded struct has its fields promoted, as Go does; every source embeds JSCSourcesSource for enabled and interval.
• time.Duration fields report reflect.String from TypeOf, accept either a time.Duration or a duration string in SetValue, and are given out as strings ("10m0s") by Map, Flatten and ValueOf, and so by JSON, YAML and String too.
• util.ByteSize fields work the same way, with size strings such as "32KiB" or "1MB" (KB, MB, ... are powers of 1000; KiB, MiB, ... powers of 1024), and are given out exactly in the largest unit that divides them ("1MB"). -1 means no limit (util.ByteSizeNoLimit); no other size may be negative. A setting for which no limit is unsafe rejects -1.
• []string fields report reflect.Slice from TypeOf and accept either a []string or a comma-separated string such as "a, b" in SetValue (and so in default tags). An empty list is stored as nil.
• A list of sections, []S for a section struct S (e.g. []JSCServerHeadersExtra), is a single key reporting reflect.Slice. It is given out as a []map[string]any, one map per item, and SetValue accepts a []S, a list of maps (as Viper reads a YAML/JSON list of objects) or a JSON array string. Each item starts from its defaults and its keys are set as SetValue sets them. S implements Section and, like a type used under several keys, has no JSCK* constant.
• A section can reject values by implementing validate(key, value) error on its pointer type. JSCLogging does this for level, JSCSourcesSource for interval, JSCServer for max_header_bytes, JSCServerTimeouts for read_header (rejecting 0 or less) and JSCServerWebSocket for max_message_size (rejecting -1), and JSCServerHeadersExtra for header and value. Each item in a list of sections is validated whole, so a key left out of an item is checked too.

Every section implements the Section interface defined in section.go: Map, Flatten, TypeOf, ValueOf and SetValue, each a one-line call into section.go (some sections also have Keys). The methods sit under a // ---------- Section methods (see Section) ---------- banner in each section's file. A section's Flatten keys are prefixed with its JSCK* constant, except for a section type used under several keys, such as JSCRateLimitRule: it has no constant and its keys are unprefixed. TestSections walks JobSearchConfig and checks every section, including embedded ones, implements Section and that its Flatten keys match its cfg tag, and that its Map, TypeOf, ValueOf and SetValue agree with the whole configuration's view of the same keys (the items of a list of sections are checked the same way), so a new section is checked without being listed anywhere.

Adding a key or section

To add a key to an existing section, add a tagged field to its struct. Nothing else is needed for it to be loaded, printed, and read or written by the config CLI commands.

To add a new section:

1. Create a conf_*.go file with the section struct, a JSCK* constant for its key, and an implementation of Section (copy an existing section's methods). A type that will be used under several keys, like JSCRateLimitRule, has no constant and passes "" to flattenOf.
2. Add the struct as a tagged field of its parent: JobSearchConfig in config.go, or JSCSources in conf_sources.go for a source, etc. The field's cfg tag must match the JSCK* constant.

Nothing in section.go needs changing.

API

• Init(cfgFile, &conf): load the configuration, as described above.
• conf.TypeOf(key), conf.ValueOf(key), conf.SetValue(key, value): read or set one dotted key on the loaded struct. SetValue converts and validates the value.
• conf.Map() (nested) and conf.Flatten() (dotted keys): the whole configuration as a map.
• conf.String(), conf.JSON(), conf.YAML() and the matching Print* methods: the configuration as text, with keys sorted.
• StoreValue(&conf, key, value) / PurgeValue(&conf, key): set or remove a key in the config file Viper loaded. Used by config set and config set --unset. StoreValue validates the value before writing it, and stores it as the field's own type. Only the file's own settings are written back; defaults and environment variables never are.
• LoadDefaultConfig, LoadToViper, LoadFromViper: move values between the struct defaults, Viper and a JobSearchConfig.

Files

File	Contents
config.go	JobSearchConfig, defaults, Viper loading, text output
conf_database.go	database section
load.go	Init, .env and config file loading
modify.go	StoreValue and PurgeValue
section.go	The Section interface and the struct tag reflection shared by every section
conf_logging.go	logging section and log level helpers
conf_sources.go	sources section and the shared source fields
conf_source_*.go	One file per source section
conf_server.go	server section
conf_server_*.go	One file per server sub-section (csrf, frontend, headers, proxy, rate_limits, timeouts, tls, websocket)
conf_ratelimitrule.go	JSCRateLimitRule, the rate limit rule shared by several keys
print.go, debug.go	Printing and debug logging of the configuration

Testing

Viper keeps its state in package globals, so tests that use it call resetViper(t) to reset it before and after the test. Tests of Init use initDir, which changes into a temp directory with its own .env and config.yaml, so the real files in the repository are never read.