Go/job-search
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Added CODE_CONVENTIONS.md and linters to enforce it, and applied it across the codebase

Browse Source 
CODE_CONVENTIONS.md sets out the conventions for Go code in this
repository: naming, layout, comments, errors, logging, configuration and
tests. .golangci.json enables the linters that check the parts that can be
checked (each limited to the rules the conventions use); the rest is
applied by hand.

Applied across the codebase:
- Naming: full-word names in application code, initialisms in one case
  (BuyerID, PrintJSON, PrintYAML, ocdsGetBuyerID), no names shadowing
  built-ins (min/max are minGap/maxGap), mustRaw renamed rawJSON as it
  returns nil rather than panicking, and config.ConfigLevelToLogLevel
  renamed LevelToLogLevel.
- Layout: lines kept to 80 characters where possible, assignments taken
  out of if and for statements that don't fit, wrapped arguments all on
  one line or one per line, steps separated by blank lines, no else after
  a return, function and struct literals wrapped.
- Comments: doc comments on every exported name (including each section's
  Map, Flatten, TypeOf, ValueOf and SetValue), section banners, British
  spelling.
- Errors: a created error starts with its package name, a wrapping error
  adds only what was being done, sentinels in one commented var block,
  errors.Is and errors.As in place of == and type assertions, and no
  err := shadowing of a named result. That fixes the mock server losing
  its access log's close error.
- Logging: the fetcher passes its workers an untagged logger, so
  findatender's lines are tagged with their own component rather than
  with fetcher's as well; cmd/mock-fts logs "stopped" with slog.
- Configuration: util.WithTimeout (moved from the websocket package)
  applies a timeout of 0 or less as no limit, used for the websocket
  write and ping, TCP close and tcpserver shutdown timeouts; every setting
  documents what 0 or empty means, including the rate limit rule's keys
  in internal/config/README.md.
- Tests: related tests merged into table-driven ones with named cases.

Separately from CODE_CONVENTIONS.md, internal/store/README.md gains a soft
SQL style convention for the queries in internal/store/queries.go, and the
queries are reformatted to follow it.
This commit is contained in:
bwilliams committed 2026-10-15 16:26:42 +01:00
1 parent 465358609a
commit 67215a6aaa
104 files changed
+3189 -1206
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

internal/config/README.md
+5 -2
View File
@@ -48,6 +48,9 @@ is kept.
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.*.enabled` | bool | `false` | In a rate limit rule: whether it applies |
| `server.websocket.*.per_second` | int | `50` | In a rate limit rule: sustained rate. `0` or less: the rule is off |
| `server.websocket.*.burst` | int | `100` | In a rate limit rule: largest burst. Below `1`: 1 |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
@@ -91,7 +94,7 @@ per key:
passed to `SetValue` (e.g. `"10m"`, `"true"`).
- A tagged field whose type is a struct is a nested section, addressed as
`section.key`. An embedded struct has its fields promoted, as Go does;
every source embeds `JobSearchConfigSource` for `enabled` and `interval`.
every source embeds `JSCSourcesSource` for `enabled` and `interval`.
- `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a
`time.Duration` or a duration string in `SetValue`, and are given out as
strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
@@ -125,7 +128,7 @@ To add a new section:
its key, and an implementation of `Section` (copy an existing section's
methods).
2. Add the struct as a tagged field of its parent: `JobSearchConfig` in
`config.go`, or `JobSearchConfigSources` in `conf_sources.go` for a source,
`config.go`, or `JSCSources` in `conf_sources.go` for a source,
etc. The field's `cfg` tag must match the `JSCK*` constant.
Nothing in `section.go` needs changing.
internal/config/conf_database.go
+5
View File
@@ -17,20 +17,25 @@ type JSCDatabase struct {
// Keys returns the keys directly in the section.
func (c *JSCDatabase) Keys() []string { return keysOf(c) }
// Map implements Section.
func (c JSCDatabase) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCDatabase) Flatten() map[string]any {
return flattenOf(JSCKDatabase, c)
}
// TypeOf implements Section.
func (c JSCDatabase) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCDatabase) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCDatabase) SetValue(key string, value any) error {
return setValue(c, key, value)
}
internal/config/conf_logging.go
+9 -4
View File
@@ -19,20 +19,25 @@ type JSCLogging struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCLogging) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCLogging) Flatten() map[string]any {
return flattenOf(JSCKLogging, c)
}
// TypeOf implements Section.
func (c JSCLogging) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCLogging) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCLogging) SetValue(key string, value any) error {
return setValue(c, key, value)
}
@@ -50,16 +55,16 @@ func (c *JSCLogging) validate(key string, value any) error {
case "debug", "info", "warn", "error":
default:
return errors.New(
"invalid log level. Must be one of: debug, info, warn, error",
"config: invalid log level: must be one of debug, info, warn, error",
)
}
}
return nil
}
// ConfigLevelToLogLevel converts a configured level name to an slog level. An
// LevelToLogLevel converts a configured level name to an slog level. An
// unknown name gives 99, above every real level, so nothing is logged.
func ConfigLevelToLogLevel(level string) slog.Level {
func LevelToLogLevel(level string) slog.Level {
switch level {
case "debug":
return slog.LevelDebug
@@ -77,5 +82,5 @@ func ConfigLevelToLogLevel(level string) slog.Level {
// LogLevel returns the configured level as an slog level. An unknown level
// gives 99, above every real level, so nothing is logged.
func (c *JSCLogging) LogLevel() slog.Level {
return ConfigLevelToLogLevel(c.Level)
return LevelToLogLevel(c.Level)
}
internal/config/conf_ratelimitrule.go
+5
View File
@@ -14,18 +14,23 @@ type JSCRateLimitRule struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCRateLimitRule) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCRateLimitRule) Flatten() map[string]any { return flattenOf("", c) }
// TypeOf implements Section.
func (c JSCRateLimitRule) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCRateLimitRule) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCRateLimitRule) SetValue(key string, value any) error {
return setValue(c, key, value)
}
internal/config/conf_sections_test.go
+89 -56
View File
@@ -25,9 +25,10 @@ func captureLogs(t *testing.T, level slog.Level) *bytes.Buffer {
t.Helper()
var buf bytes.Buffer
def := slog.Default()
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{
handler := slog.NewTextHandler(&buf, &slog.HandlerOptions{
Level: level,
})))
})
slog.SetDefault(slog.New(handler))
t.Cleanup(func() {
slog.SetDefault(def)
})
@@ -48,7 +49,8 @@ func TestDefaults(t *testing.T) {
if conf.Logging.Level != "warn" {
t.Errorf("Logging.Level = %q; want warn", conf.Logging.Level)
}
if conf.Sources.FindATender.APIRoot != "https://www.find-tender.service.gov.uk/api/1.0" {
const apiRoot = "https://www.find-tender.service.gov.uk/api/1.0"
if conf.Sources.FindATender.APIRoot != apiRoot {
t.Errorf("FindATender.APIRoot = %q", conf.Sources.FindATender.APIRoot)
}
if conf.Sources.FTSMock.APIRoot != "" {
@@ -71,25 +73,22 @@ func TestLoggingSection(t *testing.T) {
File: "/tmp/x.log",
}
if got, want := c.Keys(), []string{"level", "file"}; !reflect.DeepEqual(got, want) {
if got, want := c.Keys(), []string{
"level",
"file",
}; !reflect.DeepEqual(got, want) {
t.Errorf("Keys = %v; want %v", got, want)
}
if got, want := c.Map(), map[string]any{
"level": "info",
"file": "/tmp/x.log",
}; !reflect.DeepEqual(
got,
want,
) {
}; !reflect.DeepEqual(got, want) {
t.Errorf("Map = %v; want %v", got, want)
}
if got, want := c.Flatten(), map[string]any{
"logging.level": "info",
"logging.file": "/tmp/x.log",
}; !reflect.DeepEqual(
got,
want,
) {
}; !reflect.DeepEqual(got, want) {
t.Errorf("Flatten = %v; want %v", got, want)
}
if k, err := c.TypeOf("file"); err != nil || k != reflect.String {
@@ -101,7 +100,8 @@ func TestLoggingSection(t *testing.T) {
if _, err := c.TypeOf("nope"); err == nil {
t.Error("TypeOf(nope) succeeded")
}
if err := c.SetValue("file", "/var/log/js.log"); err != nil || c.File != "/var/log/js.log" {
if err := c.SetValue("file", "/var/log/js.log"); err != nil ||
c.File != "/var/log/js.log" {
t.Errorf("SetValue(file) = %v, File = %q", err, c.File)
}
}
@@ -114,10 +114,7 @@ func TestLoggingLevelValidation(t *testing.T) {
}
}
for _, lvl := range []string{"", "trace", "WARN", "fatal"} {
if err := c.SetValue(
"level",
lvl,
); err == nil ||
if err := c.SetValue("level", lvl); err == nil ||
!strings.Contains(err.Error(), "invalid log level") {
t.Errorf("SetValue(level, %q) = %v; want invalid log level", lvl, err)
}
@@ -136,7 +133,7 @@ func TestLoggingLevelValidation(t *testing.T) {
}
}
func TestConfigLevelToLogLevel(t *testing.T) {
func TestLevelToLogLevel(t *testing.T) {
tests := map[string]slog.Level{
"debug": slog.LevelDebug,
"info": slog.LevelInfo,
@@ -146,8 +143,8 @@ func TestConfigLevelToLogLevel(t *testing.T) {
"bogus": 99,
}
for in, want := range tests {
if got := ConfigLevelToLogLevel(in); got != want {
t.Errorf("ConfigLevelToLogLevel(%q) = %v; want %v", in, got, want)
if got := LevelToLogLevel(in); got != want {
t.Errorf("LevelToLogLevel(%q) = %v; want %v", in, got, want)
}
c := JSCLogging{
Level: in,
@@ -164,7 +161,10 @@ func TestSourceSection(t *testing.T) {
Interval: time.Minute,
}
if got, want := c.Keys(), []string{"enabled", "interval"}; !reflect.DeepEqual(got, want) {
if got, want := c.Keys(), []string{
"enabled",
"interval",
}; !reflect.DeepEqual(got, want) {
t.Errorf("Keys = %v; want %v", got, want)
}
want := map[string]any{"enabled": true, "interval": "1m0s"}
@@ -184,7 +184,8 @@ func TestSourceSection(t *testing.T) {
if v, err := c.ValueOf("interval"); err != nil || v != "1m0s" {
t.Errorf("ValueOf(interval) = %v, %v", v, err)
}
if err := c.SetValue("interval", "5s"); err != nil || c.Interval != 5*time.Second {
if err := c.SetValue("interval", "5s"); err != nil ||
c.Interval != 5*time.Second {
t.Errorf("SetValue(interval, 5s) = %v, Interval = %v", err, c.Interval)
}
if err := c.SetValue("interval", "never"); err == nil {
@@ -229,7 +230,8 @@ func TestConcreteSourceSections(t *testing.T) {
if err := s.SetValue("api_root", "http://localhost:8080"); err != nil {
t.Fatalf("SetValue(api_root): %v", err)
}
if v, err := s.ValueOf("api_root"); err != nil || v != "http://localhost:8080" {
if v, err := s.ValueOf("api_root"); err != nil ||
v != "http://localhost:8080" {
t.Errorf("ValueOf(api_root) = %v, %v", v, err)
}
wantMap["api_root"] = "http://localhost:8080"
@@ -241,7 +243,8 @@ func TestConcreteSourceSections(t *testing.T) {
if err := s.SetValue("block_shutdown", "true"); err != nil {
t.Fatalf("SetValue(block_shutdown): %v", err)
}
if k, err := s.TypeOf("block_shutdown"); err != nil || k != reflect.Bool {
if k, err := s.TypeOf("block_shutdown"); err != nil ||
k != reflect.Bool {
t.Errorf("TypeOf(block_shutdown) = %v, %v", k, err)
}
wantMap["block_shutdown"] = true
@@ -271,7 +274,8 @@ func TestSourcesSection(t *testing.T) {
if c.FTSMock.APIRoot != "http://mock" {
t.Errorf("FTSMock.APIRoot = %q", c.FTSMock.APIRoot)
}
if v, err := c.ValueOf("fts_mock.api_root"); err != nil || v != "http://mock" {
if v, err := c.ValueOf("fts_mock.api_root"); err != nil ||
v != "http://mock" {
t.Errorf("ValueOf = %v, %v", v, err)
}
if k, err := c.TypeOf("sleeper.enabled"); err != nil || k != reflect.Bool {
@@ -299,6 +303,7 @@ func TestSourcesSection(t *testing.T) {
func TestJobSearchConfigMap(t *testing.T) {
conf := defaultConfig(t)
m := conf.Map()
logging, ok := m["logging"].(map[string]any)
if !ok || logging["level"] != "warn" {
t.Errorf("Map()[logging] = %#v", m["logging"])
@@ -330,31 +335,33 @@ func TestString(t *testing.T) {
}
}
func TestJSON(t *testing.T) {
// JSON and YAML give the configuration as nested sections.
func TestSerialised(t *testing.T) {
conf := defaultConfig(t)
var m map[string]map[string]any
if err := json.Unmarshal([]byte(conf.JSON()), &m); err != nil {
t.Fatalf("JSON() is not valid JSON: %v\n%s", err, conf.JSON())
tests := []struct {
name string
output string
unmarshal func([]byte, any) error
}{
{"JSON", conf.JSON(), json.Unmarshal},
{"YAML", conf.YAML(), yaml.Unmarshal},
}
if m["logging"]["level"] != "warn" {
t.Errorf("JSON logging.level = %#v", m["logging"]["level"])
}
if _, ok := m["sources"]["fts_mock"]; !ok {
t.Errorf("JSON missing sources.fts_mock: %v", m["sources"])
}
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var m map[string]map[string]any
if err := tt.unmarshal([]byte(tt.output), &m); err != nil {
t.Fatalf("%s() doesn't parse: %v\n%s", tt.name, err, tt.output)
}
func TestYAML(t *testing.T) {
conf := defaultConfig(t)
var m map[string]map[string]any
if err := yaml.Unmarshal([]byte(conf.YAML()), &m); err != nil {
t.Fatalf("YAML() is not valid YAML: %v\n%s", err, conf.YAML())
}
if m["logging"]["level"] != "warn" {
t.Errorf("YAML logging.level = %#v", m["logging"]["level"])
}
if _, ok := m["sources"]["sleeper"]; !ok {
t.Errorf("YAML missing sources.sleeper: %v", m["sources"])
if m["logging"]["level"] != "warn" {
t.Errorf("logging.level = %#v", m["logging"]["level"])
}
for _, source := range []string{"fts_mock", "sleeper"} {
if _, ok := m["sources"][source]; !ok {
t.Errorf("missing sources.%s: %v", source, m["sources"])
}
}
})
}
}
@@ -365,6 +372,7 @@ func TestSerialisedRoundTrip(t *testing.T) {
orig.Sources.FTSMock.Enabled = true
orig.Sources.FTSMock.APIRoot = "http://mock"
orig.Server.WebSocket.MaxMessageSize = 64 << 10
for _, format := range []string{"json", "yaml"} {
t.Run(format, func(t *testing.T) {
resetViper(t)
@@ -372,10 +380,12 @@ func TestSerialisedRoundTrip(t *testing.T) {
if format == "yaml" {
out = orig.YAML()
}
viper.SetConfigType(format)
if err := viper.ReadConfig(strings.NewReader(out)); err != nil {
t.Fatalf("ReadConfig: %v", err)
}
var got JobSearchConfig
LoadFromViper(&got)
if !reflect.DeepEqual(got, orig) {
@@ -402,7 +412,9 @@ func TestLoadDefaultConfig(t *testing.T) {
if got := viper.GetString("logging.level"); got != "warn" {
t.Errorf("logging.level = %q; want warn", got)
}
if got := viper.GetDuration("sources.sleeper.interval"); got != 10*time.Minute {
if got := viper.GetDuration(
"sources.sleeper.interval",
); got != 10*time.Minute {
t.Errorf("sources.sleeper.interval = %v; want 10m", got)
}
}
@@ -460,7 +472,9 @@ func TestLoadFromViperInvalidValues(t *testing.T) {
t.Errorf("Logging.Level = %q; want warn kept", conf.Logging.Level)
}
if conf.Sources.Sleeper.Interval != 10*time.Minute {
t.Errorf("Sleeper.Interval = %v; want 10m kept", conf.Sources.Sleeper.Interval)
t.Errorf(
"Sleeper.Interval = %v; want 10m kept", conf.Sources.Sleeper.Interval,
)
}
for _, key := range []string{"logging.level", "sources.sleeper.interval"} {
if !strings.Contains(logs.String(), "key="+key) {
@@ -477,7 +491,9 @@ func TestDatabaseSection(t *testing.T) {
if got, want := c.Keys(), []string{"file"}; !reflect.DeepEqual(got, want) {
t.Errorf("Keys = %v; want %v", got, want)
}
if got, want := c.Map(), map[string]any{"file": "./jobsearch.db"}; !reflect.DeepEqual(got, want) {
if got, want := c.Map(), map[string]any{
"file": "./jobsearch.db",
}; !reflect.DeepEqual(got, want) {
t.Errorf("Map = %v; want %v", got, want)
}
want := map[string]any{"database.file": "./jobsearch.db"}
@@ -496,7 +512,8 @@ func TestDatabaseSection(t *testing.T) {
if _, err := c.ValueOf("nope"); err == nil {
t.Error("ValueOf(nope) succeeded")
}
if err := c.SetValue("file", "/var/lib/js.db"); err != nil || c.File != "/var/lib/js.db" {
if err := c.SetValue("file", "/var/lib/js.db"); err != nil ||
c.File != "/var/lib/js.db" {
t.Errorf("SetValue(file) = %v, File = %q", err, c.File)
}
if err := c.SetValue("file", 42); err == nil {
@@ -532,10 +549,18 @@ func TestViperValue(t *testing.T) {
} {
got, ok := viperValue(tt.key, tt.kind)
if !ok || !reflect.DeepEqual(got, tt.want) {
t.Errorf("viperValue(%q, %v) = %#v, %v; want %#v", tt.key, tt.kind, got, ok, tt.want)
t.Errorf(
"viperValue(%q, %v) = %#v, %v; want %#v",
tt.key,
tt.kind,
got,
ok,
tt.want,
)
}
}
for _, kind := range []reflect.Kind{reflect.Float64, reflect.Int8, reflect.Map} {
unhandled := []reflect.Kind{reflect.Float64, reflect.Int8, reflect.Map}
for _, kind := range unhandled {
if got, ok := viperValue("i", kind); ok {
t.Errorf("viperValue(i, %v) = %#v, true; want unhandled", kind, got)
}
@@ -553,14 +578,20 @@ func TestWebSocketMaxMessageSizeNoLimit(t *testing.T) {
if c.MaxMessageSize != 0 {
t.Errorf("rejected size was stored: %v", c.MaxMessageSize)
}
for _, text := range []string{"0B", "64KiB"} {
if err := c.SetValue("max_message_size", text); err != nil {
t.Errorf("SetValue(max_message_size, %s): %v", text, err)
}
}
conf := defaultConfig(t)
if err := conf.SetValue("server.websocket.max_message_size", "-1"); err == nil {
t.Error("JobSearchConfig.SetValue(server.websocket.max_message_size, -1) succeeded")
if err := conf.SetValue(
"server.websocket.max_message_size", "-1",
); err == nil {
t.Error(
"JobSearchConfig.SetValue(server.websocket.max_message_size, -1) succeeded",
)
}
}
@@ -584,6 +615,8 @@ func TestSourceIntervalMustBePositive(t *testing.T) {
}
}
if conf.Sources.FindATender.Interval.Minutes() != 1 {
t.Errorf("FindATender.Interval = %v; want 1m", conf.Sources.FindATender.Interval)
t.Errorf(
"FindATender.Interval = %v; want 1m", conf.Sources.FindATender.Interval,
)
}
}
internal/config/conf_server.go
+5
View File
@@ -20,20 +20,25 @@ type JSCServer struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCServer) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCServer) Flatten() map[string]any {
return flattenOf(JSCKServer, c)
}
// TypeOf implements Section.
func (c JSCServer) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCServer) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCServer) SetValue(key string, value any) error {
return setValue(c, key, value)
}
internal/config/conf_server_ratelimits.go
+5
View File
@@ -17,20 +17,25 @@ type JSCServerRateLimits struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCServerRateLimits) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCServerRateLimits) Flatten() map[string]any {
return flattenOf(JSCKServerRateLimits, c)
}
// TypeOf implements Section.
func (c JSCServerRateLimits) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCServerRateLimits) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCServerRateLimits) SetValue(key string, value any) error {
return setValue(c, key, value)
}
internal/config/conf_server_timeouts.go
+5
View File
@@ -17,20 +17,25 @@ type JSCServerTimeouts struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCServerTimeouts) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCServerTimeouts) Flatten() map[string]any {
return flattenOf(JSCKServerTimeouts, c)
}
// TypeOf implements Section.
func (c JSCServerTimeouts) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCServerTimeouts) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCServerTimeouts) SetValue(key string, value any) error {
return setValue(c, key, value)
}
internal/config/conf_server_websocket.go
+8 -2
View File
@@ -28,20 +28,25 @@ type JSCServerWebSocket struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCServerWebSocket) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCServerWebSocket) Flatten() map[string]any {
return flattenOf(JSCKServerWebSocket, c)
}
// TypeOf implements Section.
func (c JSCServerWebSocket) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCServerWebSocket) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCServerWebSocket) SetValue(key string, value any) error {
return setValue(c, key, value)
}
@@ -52,9 +57,10 @@ func (c *JSCServerWebSocket) SetValue(key string, value any) error {
// max_message_size can't be -1 (no limit): a client could then send a
// message of any size, and the server holds each message in memory.
func (c *JSCServerWebSocket) validate(key string, value any) error {
if key == "max_message_size" && value.(util.ByteSize) == util.ByteSizeNoLimit {
if key == "max_message_size" &&
value.(util.ByteSize) == util.ByteSizeNoLimit {
return errors.New(
"max_message_size can't be -1 (no limit): set an explicit size, or 0 for the default",
"config: max_message_size can't be -1 (no limit): set an explicit size, or 0 for the default",
)
}
return nil
internal/config/conf_source_findatender.go
+5
View File
@@ -14,20 +14,25 @@ type JSCSourceFindATender struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSourceFindATender) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSourceFindATender) Flatten() map[string]any {
return flattenOf(JSCKSourceFindATender, c)
}
// TypeOf implements Section.
func (c JSCSourceFindATender) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCSourceFindATender) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCSourceFindATender) SetValue(key string, value any) error {
return setValue(c, key, value)
}
internal/config/conf_source_ftsmock.go
+5
View File
@@ -15,20 +15,25 @@ type JSCSourceFTSMock struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSourceFTSMock) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSourceFTSMock) Flatten() map[string]any {
return flattenOf(JSCKSourceFTSMock, c)
}
// TypeOf implements Section.
func (c JSCSourceFTSMock) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCSourceFTSMock) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCSourceFTSMock) SetValue(key string, value any) error {
return setValue(c, key, value)
}
internal/config/conf_source_sleeper.go
+5
View File
@@ -15,20 +15,25 @@ type JSCSourceSleeper struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSourceSleeper) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSourceSleeper) Flatten() map[string]any {
return flattenOf(JSCKSourceSleeper, c)
}
// TypeOf implements Section.
func (c JSCSourceSleeper) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCSourceSleeper) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCSourceSleeper) SetValue(key string, value any) error {
return setValue(c, key, value)
}
internal/config/conf_sources.go
+13 -1
View File
@@ -6,6 +6,8 @@ import (
"time"
)
// ========== Sources ==========
// JSCKSources is the sources section's configuration key.
const JSCKSources string = "sources"
@@ -19,20 +21,25 @@ type JSCSources struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSources) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSources) Flatten() map[string]any {
return flattenOf(JSCKSources, c)
}
// TypeOf implements Section.
func (c JSCSources) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCSources) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCSources) SetValue(key string, value any) error {
return setValue(c, key, value)
}
@@ -50,20 +57,25 @@ type JSCSourcesSource struct {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSourcesSource) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSourcesSource) Flatten() map[string]any {
return flattenOf("", c)
}
// TypeOf implements Section.
func (c JSCSourcesSource) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JSCSourcesSource) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JSCSourcesSource) SetValue(key string, value any) error {
return setValue(c, key, value)
}
@@ -76,7 +88,7 @@ func (c *JSCSourcesSource) Keys() []string { return keysOf(c) }
func (c *JSCSourcesSource) validate(key string, value any) error {
if key == "interval" && value.(time.Duration) <= 0 {
return errors.New(
"interval must be greater than 0; set enabled to false to turn the source off",
"config: interval must be greater than 0; set enabled to false to turn the source off",
)
}
return nil
internal/config/config.go
+17 -8
View File
@@ -42,18 +42,23 @@ func LoadDefaultConfig() {
// ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JobSearchConfig) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JobSearchConfig) Flatten() map[string]any { return flattenOf("", c) }
// TypeOf implements Section.
func (c JobSearchConfig) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
// ValueOf implements Section.
func (c JobSearchConfig) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
// SetValue implements Section.
func (c *JobSearchConfig) SetValue(key string, value any) error {
return setValue(c, key, value)
}
@@ -70,7 +75,11 @@ func LoadFromViper(conf *JobSearchConfig) {
}
value, ok := viperValue(key, keyType)
if !ok {
slog.Error("unknown config key type", "key", key, "kind", keyType.String())
slog.Error(
"unknown config key type",
"key", key,
"kind", keyType.String(),
)
continue
}
if err := conf.SetValue(key, value); err != nil {
@@ -79,9 +88,9 @@ func LoadFromViper(conf *JobSearchConfig) {
}
}
// viperValue reads key from Viper as a value of kind. ok is false for a kind
// LoadFromViper doesn't handle.
func viperValue(key string, kind reflect.Kind) (value any, ok bool) {
// viperValue reads key from Viper as a value of kind. Its bool is false for a
// kind LoadFromViper doesn't handle.
func viperValue(key string, kind reflect.Kind) (any, bool) {
switch kind {
case reflect.String:
return viper.GetString(key), true
@@ -134,13 +143,13 @@ func (c JobSearchConfig) String() string {
// JSON formats the configuration as indented JSON.
func (c JobSearchConfig) JSON() string {
confMap := c.Map()
jbytes, _ := json.MarshalIndent(confMap, "", " ")
return string(jbytes)
encoded, _ := json.MarshalIndent(confMap, "", " ")
return string(encoded)
}
// YAML formats the configuration as YAML.
func (c JobSearchConfig) YAML() string {
confMap := c.Map()
ybytes, _ := yaml.Marshal(confMap)
return string(ybytes)
encoded, _ := yaml.Marshal(confMap)
return string(encoded)
}
internal/config/config_test.go
+29 -13
View File
@@ -10,7 +10,10 @@ import (
// Kinds that LoadFromViper knows how to read.
var viperKinds = map[reflect.Kind]bool{
reflect.String: true, reflect.Int: true, reflect.Bool: true, reflect.Slice: true,
reflect.String: true,
reflect.Int: true,
reflect.Bool: true,
reflect.Slice: true,
}
// Every key round-trips through TypeOf/ValueOf/SetValue and has a type that
@@ -25,7 +28,9 @@ func TestKeysRoundTrip(t *testing.T) {
if err != nil {
t.Errorf("TypeOf(%q): %v", key, err)
} else if !viperKinds[kind] {
t.Errorf("TypeOf(%q) = %v, which LoadFromViper does not handle", key, kind)
t.Errorf(
"TypeOf(%q) = %v, which LoadFromViper does not handle", key, kind,
)
}
got, err := conf.ValueOf(key)
if err != nil || !reflect.DeepEqual(got, want) {
@@ -53,6 +58,7 @@ func TestSections(t *testing.T) {
if !reflect.PointerTo(reflect.TypeOf(conf)).Implements(sectionType) {
t.Error("*JobSearchConfig does not implement Section")
}
all := conf.Flatten()
uses := map[reflect.Type]int{}
countSections(uses, reflect.TypeOf(conf))
@@ -64,8 +70,8 @@ func TestSections(t *testing.T) {
func countSections(uses map[reflect.Type]int, section reflect.Type) {
for _, f := range fieldsOf(section) {
if f.section {
uses[f.typ]++
countSections(uses, f.typ)
uses[f.fieldType]++
countSections(uses, f.fieldType)
}
}
}
@@ -78,14 +84,16 @@ func checkSections(
prefix string,
section reflect.Value,
) {
t.Helper()
for _, f := range fieldsOf(section.Type()) {
key := joinKey(prefix, f.key)
if isSectionList(f.typ) {
if isSectionList(f.fieldType) {
checkSectionList(t, all, key, section.FieldByIndex(f.index))
}
if !f.section {
continue
}
fv := section.FieldByIndex(f.index)
name := fv.Type().Name()
if !fv.Addr().Type().Implements(sectionType) {
@@ -93,18 +101,16 @@ func checkSections(
continue
}
checkEmbedded(t, fv.Type())
keys := fv.Addr().Interface().(Section).Flatten()
keyPrefix := prefix
if uses[f.typ] > 1 {
if uses[f.fieldType] > 1 {
keyPrefix = key
}
for k := range keys {
if _, ok := all[joinKey(keyPrefix, k)]; !ok {
t.Errorf(
"%s.Flatten() key %q does not match its cfg tag %q",
name,
k,
key,
"%s.Flatten() key %q does not match its cfg tag %q", name, k, key,
)
}
}
@@ -126,29 +132,36 @@ func checkSections(
// checkSectionList checks the items of the list of sections at key implement
// Section and that their methods agree with the list's value in all.
func checkSectionList(t *testing.T, all map[string]any, key string, list reflect.Value) {
func checkSectionList(
t *testing.T, all map[string]any, key string, list reflect.Value,
) {
t.Helper()
name := list.Type().Elem().Name()
if !reflect.PointerTo(list.Type().Elem()).Implements(sectionType) {
t.Errorf("*%s (items of %s) does not implement Section", name, key)
return
}
items, _ := all[key].([]map[string]any)
if len(items) != list.Len() {
t.Errorf("%s has %d items; Flatten gives %d", key, list.Len(), len(items))
return
}
for i := range list.Len() {
item, itemName := list.Index(i), fmt.Sprintf("%s[%d]", key, i)
want := flattenMap("", items[i])
// Items are not prefixed, and their kinds are their fields' kinds.
if got := item.Addr().Interface().(Section).Flatten(); !reflect.DeepEqual(got, want) {
got := item.Addr().Interface().(Section).Flatten()
if !reflect.DeepEqual(got, want) {
t.Errorf("%s.Flatten() =\n%v\nwant\n%v", itemName, got, want)
}
kinds := map[string]reflect.Kind{}
for k := range want {
_, f, _ := lookup(item, k)
kinds[k] = kindOf(f.typ)
kinds[k] = kindOf(f.fieldType)
}
checkSectionMethods(t, itemName, item, want, kinds)
}
@@ -164,10 +177,12 @@ func checkSectionMethods(
want map[string]any,
kinds map[string]reflect.Kind,
) {
t.Helper()
s := section.Addr().Interface().(Section)
if got := flattenMap("", s.Map()); !reflect.DeepEqual(got, want) {
t.Errorf("%s.Map() =\n%v\nwant\n%v", name, got, want)
}
before := section.Interface()
for k, v := range want {
if kind, err := s.TypeOf(k); err != nil || kind != kinds[k] {
@@ -180,6 +195,7 @@ func checkSectionMethods(
t.Errorf("%s.SetValue(%q, %#v): %v", name, k, v, err)
}
}
if !reflect.DeepEqual(section.Interface(), before) {
t.Errorf("%s changed by setting its keys to their own values", name)
}
internal/config/load.go
+17 -17
View File
@@ -1,6 +1,7 @@
package config
import (
"errors"
"fmt"
"log/slog"
"os"
@@ -41,13 +42,13 @@ func loadEnv() error {
return nil
}
// loadConfig sets up Viper with the defaults, cfgFile (or config.* found in
// loadConfig sets up Viper with the defaults, configFile (or config.* found in
// EtcConfigPath or the working directory) and JOBSEARCH_* environment
// variables. A missing config file is only a warning.
func loadConfig(cfgFile string) error {
if cfgFile != "" {
func loadConfig(configFile string) error {
if configFile != "" {
// Use config file from the flag.
viper.SetConfigFile(cfgFile)
viper.SetConfigFile(configFile)
} else {
// Search config in /etc or local directory.
viper.AddConfigPath(EtcConfigPath)
@@ -64,26 +65,25 @@ func loadConfig(cfgFile string) error {
viper.AutomaticEnv() // read in environment variables that match
err := viper.ReadInConfig()
if err != nil {
if _, ok := err.(viper.ConfigFileNotFoundError); ok {
// Indicates an explicitly set config file is not found or
// that no config file was found in any search path
slog.Warn("no config file found; using defaults")
} else {
// Config file was found but another error was produced
return err
}
} else {
// Config file found and successfully parsed
var notFound viper.ConfigFileNotFoundError
switch {
case errors.As(err, &notFound):
// No config file was found in any search path.
slog.Warn("no config file found; using defaults")
case err != nil:
// A config file was found but couldn't be read or parsed.
return err
default:
slog.Info("using config file", "file", viper.ConfigFileUsed())
}
return nil
}
// Init loads the .env file and the configuration (see loadConfig) into conf.
// A debug or error log level already set in conf, from --debug or --quiet,
// is kept over the loaded one.
func Init(cfgFile string, conf *JobSearchConfig) error {
func Init(configFile string, conf *JobSearchConfig) error {
if err := loadEnv(); err != nil {
return err
}
@@ -91,7 +91,7 @@ func Init(cfgFile string, conf *JobSearchConfig) error {
printEnv()
}
if err := loadConfig(cfgFile); err != nil {
if err := loadConfig(configFile); err != nil {
return err
}
internal/config/load_test.go
+34 -9
View File
@@ -18,6 +18,7 @@ func initDir(t *testing.T, env []string, config string) string {
resetViper(t)
dir := t.TempDir()
t.Chdir(dir)
if env != nil {
// godotenv sets variables without cleaning up after the test.
for _, kv := range env {
@@ -28,9 +29,11 @@ func initDir(t *testing.T, env []string, config string) string {
}
writeFile(t, filepath.Join(dir, ".env"), strings.Join(env, "\n")+"\n")
}
if config != "" {
writeFile(t, filepath.Join(dir, "config.yaml"), config)
}
return dir
}
@@ -63,12 +66,16 @@ func TestInitLoadsEnvAndConfigFile(t *testing.T) {
if conf.Logging.Level != "info" {
t.Errorf("Logging.Level = %q; want info", conf.Logging.Level)
}
if !conf.Sources.FTSMock.Enabled || conf.Sources.FTSMock.Interval != 30*time.Second {
if !conf.Sources.FTSMock.Enabled ||
conf.Sources.FTSMock.Interval != 30*time.Second {
t.Errorf("FTSMock = %+v; want enabled every 30s", conf.Sources.FTSMock)
}
// Keys missing from the file keep their defaults.
if conf.Sources.FindATender.Interval != 10*time.Minute {
t.Errorf("FindATender.Interval = %v; want default 10m", conf.Sources.FindATender.Interval)
t.Errorf(
"FindATender.Interval = %v; want default 10m",
conf.Sources.FindATender.Interval,
)
}
}
@@ -83,7 +90,9 @@ func TestInitExplicitConfigFile(t *testing.T) {
t.Fatal(err)
}
if conf.Logging.Level != "error" {
t.Errorf("Logging.Level = %q; want error from %s", conf.Logging.Level, other)
t.Errorf(
"Logging.Level = %q; want error from %s", conf.Logging.Level, other,
)
}
}
@@ -124,7 +133,9 @@ func TestInitKeepsCommandLineLevel(t *testing.T) {
// At debug, the environment and loaded config are logged.
gotDebug := strings.Contains(logs.String(), "loaded config value")
if gotDebug != (level == "debug") {
t.Errorf("debug output logged = %v at level %s: %s", gotDebug, level, logs)
t.Errorf(
"debug output logged = %v at level %s: %s", gotDebug, level, logs,
)
}
})
}
@@ -134,7 +145,8 @@ func TestInitErrors(t *testing.T) {
t.Run("no .env file", func(t *testing.T) {
initDir(t, nil, testConfigYAML)
var conf JobSearchConfig
if err := Init("", &conf); err == nil || !strings.Contains(err.Error(), ".env") {
if err := Init("", &conf); err == nil ||
!strings.Contains(err.Error(), ".env") {
t.Fatalf("err = %v; want an invalid .env file error", err)
}
})
@@ -169,10 +181,15 @@ func TestInitEnvOverride(t *testing.T) {
t.Fatal(err)
}
if conf.Logging.Level != "error" {
t.Errorf("Logging.Level = %q; want error from the environment", conf.Logging.Level)
t.Errorf(
"Logging.Level = %q; want error from the environment", conf.Logging.Level,
)
}
if conf.Sources.FTSMock.Interval != 45*time.Second {
t.Errorf("FTSMock.Interval = %v; want 45s from the environment", conf.Sources.FTSMock.Interval)
t.Errorf(
"FTSMock.Interval = %v; want 45s from the environment",
conf.Sources.FTSMock.Interval,
)
}
}
@@ -204,11 +221,15 @@ func loadExplicit(t *testing.T) {
t.Helper()
resetViper(t)
LoadDefaultConfig()
viper.SetEnvPrefix(EnvPrefix)
viper.SetEnvKeyReplacer(envKeyReplacer)
viper.AutomaticEnv()
viper.SetConfigType("yaml")
if err := viper.ReadConfig(strings.NewReader(explicitConfigYAML)); err != nil {
if err := viper.ReadConfig(
strings.NewReader(explicitConfigYAML),
); err != nil {
t.Fatal(err)
}
}
@@ -255,7 +276,11 @@ func TestPrintLoadedConfig(t *testing.T) {
t.Errorf("missing %q in:\n%s", want, out)
}
}
for _, unwanted := range []string{"sources.", "logging.file", "connection_limit.burst"} {
for _, unwanted := range []string{
"sources.",
"logging.file",
"connection_limit.burst",
} {
if strings.Contains(out, unwanted) {
t.Errorf("logged default key %q:\n%s", unwanted, out)
}
internal/config/modify.go
+15 -5
View File
@@ -14,14 +14,16 @@ import (
// file's own settings are written back, not defaults or environment
// variables. It returns the stored value (as ValueOf gives it) and its kind.
func StoreValue(
conf *JobSearchConfig, key string, value string,
conf *JobSearchConfig, key, value string,
) (any, reflect.Kind, error) {
return storeValue(conf, key, value)
}
// storeValue is StoreValue for any section, so key types that
// JobSearchConfig doesn't have yet can be tested.
func storeValue(section Section, key string, value string) (any, reflect.Kind, error) {
func storeValue(
section Section, key, value string,
) (any, reflect.Kind, error) {
confType, err := section.TypeOf(key)
if err != nil {
return nil, confType, err
@@ -31,7 +33,7 @@ func storeValue(section Section, key string, value string) (any, reflect.Kind, e
// SetValue gives.
if _, err := parseValue(confType, value); err != nil {
return nil, confType, fmt.Errorf(
"config value type not accepted, expected <%s>", confType,
"config: value type not accepted, expected <%s>", confType,
)
}
@@ -69,7 +71,11 @@ func parseValue(kind reflect.Kind, value string) (any, error) {
return value, nil
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return strconv.ParseInt(value, 10, 64)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
case reflect.Uint,
reflect.Uint8,
reflect.Uint16,
reflect.Uint32,
reflect.Uint64:
return strconv.ParseUint(value, 10, 64)
case reflect.Float32, reflect.Float64:
return strconv.ParseFloat(value, 64)
@@ -91,6 +97,7 @@ func PurgeValue(conf *JobSearchConfig, key string) error {
if err != nil {
return err
}
out := viper.New()
out.SetConfigFile(file.ConfigFileUsed())
for _, fileKey := range file.AllKeys() {
@@ -98,6 +105,7 @@ func PurgeValue(conf *JobSearchConfig, key string) error {
out.Set(fileKey, file.Get(fileKey))
}
}
return out.WriteConfig()
}
@@ -106,12 +114,14 @@ func PurgeValue(conf *JobSearchConfig, key string) error {
func readUsedConfigFile() (*viper.Viper, error) {
path := viper.ConfigFileUsed()
if path == "" {
return nil, errors.New("no configuration file in use")
return nil, errors.New("config: no configuration file in use")
}
file := viper.New()
file.SetConfigFile(path)
if err := file.ReadInConfig(); err != nil {
return nil, err
}
return file, nil
}
internal/config/modify_test.go
+38 -10
View File
@@ -20,10 +20,12 @@ func useConfigFile(t *testing.T, contents string) string {
if err := os.WriteFile(path, []byte(contents), 0o600); err != nil {
t.Fatal(err)
}
viper.SetConfigFile(path)
if err := viper.ReadInConfig(); err != nil {
t.Fatal(err)
}
return path
}
@@ -74,6 +76,7 @@ func TestStoreValueKeepsOtherKeys(t *testing.T) {
if _, _, err := StoreValue(&conf, "logging.level", "error"); err != nil {
t.Fatal(err)
}
v := readConfigFile(t, path)
if got := v.GetString("logging.file"); got != "/tmp/js.log" {
t.Errorf("logging.file = %q; want it kept", got)
@@ -159,7 +162,9 @@ func TestPurgeValueErrors(t *testing.T) {
func TestStoreValueUpdatesConf(t *testing.T) {
useConfigFile(t, "logging:\n level: warn\n")
conf := defaultConfig(t)
if _, _, err := StoreValue(&conf, "sources.fts_mock.interval", "30s"); err != nil {
if _, _, err := StoreValue(
&conf, "sources.fts_mock.interval", "30s",
); err != nil {
t.Fatal(err)
}
if conf.Sources.FTSMock.Interval != 30*time.Second {
@@ -185,7 +190,14 @@ func TestParseValue(t *testing.T) {
} {
got, err := parseValue(tt.kind, tt.value)
if err != nil || !reflect.DeepEqual(got, tt.want) {
t.Errorf("parseValue(%v, %q) = %#v, %v; want %#v", tt.kind, tt.value, got, err, tt.want)
t.Errorf(
"parseValue(%v, %q) = %#v, %v; want %#v",
tt.kind,
tt.value,
got,
err,
tt.want,
)
}
}
@@ -199,11 +211,14 @@ func TestParseValue(t *testing.T) {
{reflect.Float64, "x"},
} {
if got, err := parseValue(tt.kind, tt.value); err == nil {
t.Errorf("parseValue(%v, %q) = %#v; want an error", tt.kind, tt.value, got)
t.Errorf(
"parseValue(%v, %q) = %#v; want an error", tt.kind, tt.value, got,
)
}
}
if _, err := parseValue(reflect.Map, "a"); !errors.Is(err, errors.ErrUnsupported) {
_, err := parseValue(reflect.Map, "a")
if !errors.Is(err, errors.ErrUnsupported) {
t.Errorf("parseValue(map) err = %v; want ErrUnsupported", err)
}
}
@@ -217,11 +232,21 @@ type numberSection struct {
Ratio float32 `cfg:"ratio"`
}
func (c numberSection) Map() map[string]any { return mapOf(c) }
func (c numberSection) Flatten() map[string]any { return flattenOf("", c) }
func (c numberSection) TypeOf(key string) (reflect.Kind, error) { return typeOf(c, key) }
func (c numberSection) ValueOf(key string) (any, error) { return valueOf(c, key) }
func (c *numberSection) SetValue(key string, value any) error { return setValue(c, key, value) }
func (c numberSection) Map() map[string]any { return mapOf(c) }
func (c numberSection) Flatten() map[string]any { return flattenOf("", c) }
func (c numberSection) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
func (c numberSection) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
func (c *numberSection) SetValue(key string, value any) error {
return setValue(c, key, value)
}
// Numbers are stored as the field's own type, not as the int64, uint64 or
// float64 that parsing produces.
@@ -276,7 +301,9 @@ func TestStoreAndPurgeWriteOnlyFileSettings(t *testing.T) {
LoadDefaultConfig()
conf := defaultConfig(t)
if _, _, err := StoreValue(&conf, "sources.sleeper.enabled", "true"); err != nil {
if _, _, err := StoreValue(
&conf, "sources.sleeper.enabled", "true",
); err != nil {
t.Fatal(err)
}
want := map[string]any{
@@ -301,6 +328,7 @@ func checkFileSettings(t *testing.T, path string, want map[string]any) {
for _, k := range v.AllKeys() {
got[k] = v.Get(k)
}
if !reflect.DeepEqual(got, want) {
//nolint:gosec // G304: reading a file the test created in t.TempDir
b, _ := os.ReadFile(path)
internal/config/print.go
+7 -7
View File
@@ -15,13 +15,13 @@ func (c JobSearchConfig) PrintTerminal() {
fmt.Print(c.String() + "\n")
}
// PrintJson prints the configuration to stdout as JSON.
func (c JobSearchConfig) PrintJson() {
// PrintJSON prints the configuration to stdout as JSON.
func (c JobSearchConfig) PrintJSON() {
fmt.Print(c.JSON() + "\n")
}
// PrintYaml prints the configuration to stdout as YAML.
func (c JobSearchConfig) PrintYaml() {
// PrintYAML prints the configuration to stdout as YAML.
func (c JobSearchConfig) PrintYAML() {
fmt.Print(c.YAML() + "\n")
}
@@ -29,7 +29,7 @@ func (c JobSearchConfig) PrintYaml() {
// debug, warn, error, or info for any other level. On a terminal it logs one
// line and prints the values under it as "key = value" lines; elsewhere each
// value is a log entry of its own.
func (conf JobSearchConfig) PrintLog(atLevel slog.Level) {
func (c JobSearchConfig) PrintLog(atLevel slog.Level) {
switch atLevel {
case slog.LevelDebug, slog.LevelWarn, slog.LevelError:
default:
@@ -38,7 +38,7 @@ func (conf JobSearchConfig) PrintLog(atLevel slog.Level) {
ctx := context.Background()
logger := slog.Default()
flat := conf.Flatten()
flat := c.Flatten()
keys := slices.Sorted(maps.Keys(flat))
if logging.WritesToTerminal(logger.Handler()) {
@@ -55,4 +55,4 @@ func (conf JobSearchConfig) PrintLog(atLevel slog.Level) {
for _, key := range keys {
logger.Log(ctx, atLevel, "config value", "key", key, "value", flat[key])
}
}
}
internal/config/print_test.go
+35 -19
View File
@@ -20,17 +20,24 @@ func captureStdout(t *testing.T, fn func()) string {
if err != nil {
t.Fatal(err)
}
old := os.Stdout
os.Stdout = f
t.Cleanup(func() {
os.Stdout = old
})
fn()
os.Stdout = old
if err := f.Close(); err != nil {
t.Fatal(err)
}
b, err := os.ReadFile(filepath.Clean(path))
if err != nil {
t.Fatal(err)
}
return string(b)
}
@@ -50,6 +57,7 @@ func TestStringSorted(t *testing.T) {
key, _, _ := strings.Cut(line, " = ")
keys = append(keys, key)
}
if !slices.IsSorted(keys) || len(keys) != len(conf.Flatten()) {
t.Errorf("String keys = %v; want each key once, sorted", keys)
}
@@ -58,27 +66,30 @@ func TestStringSorted(t *testing.T) {
}
}
func TestPrintJson(t *testing.T) {
// PrintJSON and PrintYAML print the configuration in their format.
func TestPrintFormats(t *testing.T) {
conf := defaultConfig(t)
out := captureStdout(t, conf.PrintJson)
var m map[string]any
if err := json.Unmarshal([]byte(out), &m); err != nil {
t.Fatalf("PrintJson output is not JSON: %v\n%s", err, out)
tests := []struct {
name string
print func()
unmarshal func([]byte, any) error
}{
{"PrintJSON", conf.PrintJSON, json.Unmarshal},
{"PrintYAML", conf.PrintYAML, yaml.Unmarshal},
}
if got := m["logging"].(map[string]any)["level"]; got != "warn" {
t.Errorf("logging.level = %v; want warn", got)
}
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
out := captureStdout(t, tt.print)
var m map[string]any
if err := tt.unmarshal([]byte(out), &m); err != nil {
t.Fatalf("%s output doesn't parse: %v\n%s", tt.name, err, out)
}
func TestPrintYaml(t *testing.T) {
conf := defaultConfig(t)
out := captureStdout(t, conf.PrintYaml)
var m map[string]any
if err := yaml.Unmarshal([]byte(out), &m); err != nil {
t.Fatalf("PrintYaml output is not YAML: %v\n%s", err, out)
}
if got := m["logging"].(map[string]any)["level"]; got != "warn" {
t.Errorf("logging.level = %v; want warn", got)
level := m["logging"].(map[string]any)["level"]
if level != "warn" {
t.Errorf("logging.level = %v; want warn", level)
}
})
}
}
@@ -101,7 +112,12 @@ func TestPrintLog(t *testing.T) {
lines := strings.Split(strings.TrimSpace(logs.String()), "\n")
if len(lines) != len(conf.Flatten()) {
t.Errorf("PrintLog(%v) logged %d lines; want %d", tt.at, len(lines), len(conf.Flatten()))
t.Errorf(
"PrintLog(%v) logged %d lines; want %d",
tt.at,
len(lines),
len(conf.Flatten()),
)
}
for _, l := range lines {
if !strings.Contains(l, tt.want) {
internal/config/section.go
+86 -47
View File
@@ -43,6 +43,7 @@ import (
//
// A section can check values before they are stored by implementing
// validator on its pointer type.
// The struct tags section.go reads.
const (
// keyTag holds a field's configuration key.
@@ -54,7 +55,7 @@ const (
// The unknown key error and the field types section.go handles specially.
var (
// errUnknownKey is returned for a key that names no field.
errUnknownKey = errors.New("unknown configuration key")
errUnknownKey = errors.New("config: unknown configuration key")
// durationType is time.Duration, read and given out as a string.
durationType = reflect.TypeOf(time.Duration(0))
// byteSizeType is util.ByteSize, read and given out as a string.
@@ -67,9 +68,9 @@ var (
// Section is the method set of every configuration section, including
// JobSearchConfig itself, implemented on the section's pointer type. Keys are
// dotted paths relative to the section, e.g. "sleeper.interval" in
// JSCSources. Durations and byte sizes are given out as strings such as
// "10m0s" and "32KiB", matching TypeOf's reflect.String for them. TestSections checks every
// dotted paths relative to the section, e.g. "sleeper.interval" in JSCSources.
// Durations and byte sizes are given out as strings such as "10m0s" and
// "32KiB", matching TypeOf's reflect.String for them. TestSections checks every
// section, found by walking JobSearchConfig, implements it.
type Section interface {
// Map returns the section as a map of keys to values, with nested
@@ -102,18 +103,19 @@ type validator interface {
// field is a section struct field that is part of the configuration (see
// fieldsOf).
type field struct {
key string
index []int // path for reflect.Value.FieldByIndex
typ reflect.Type
section bool
def string
hasDefault bool
key string
index []int // path for reflect.Value.FieldByIndex
fieldType reflect.Type
section bool
defaultValue string
hasDefault bool
}
// isSectionList reports whether listType is a list of sections, []S for a
// section struct S.
func isSectionList(listType reflect.Type) bool {
return listType.Kind() == reflect.Slice && listType.Elem().Kind() == reflect.Struct &&
return listType.Kind() == reflect.Slice &&
listType.Elem().Kind() == reflect.Struct &&
listType.Elem() != timeType
}
@@ -123,6 +125,7 @@ func fieldsOf(structType reflect.Type) []field {
var fields []field
for i := 0; i < structType.NumField(); i++ {
structField := structType.Field(i)
if structField.Anonymous && structField.Type.Kind() == reflect.Struct {
for _, embeddedField := range fieldsOf(structField.Type) {
embeddedField.index = append([]int{i}, embeddedField.index...)
@@ -130,20 +133,24 @@ func fieldsOf(structType reflect.Type) []field {
}
continue
}
key, ok := structField.Tag.Lookup(keyTag)
if !ok || key == "" || key == "-" || !structField.IsExported() {
continue
}
defaultValue, hasDefault := structField.Tag.Lookup(defaultTag)
fields = append(fields, field{
key: key,
index: []int{i},
typ: structField.Type,
section: structField.Type.Kind() == reflect.Struct && structField.Type != timeType,
def: defaultValue,
hasDefault: hasDefault,
key: key,
index: []int{i},
fieldType: structField.Type,
section: structField.Type.Kind() == reflect.Struct &&
structField.Type != timeType,
defaultValue: defaultValue,
hasDefault: hasDefault,
})
}
return fields
}
@@ -249,6 +256,7 @@ func exportValue(fieldValue reflect.Value) any {
case byteSizeType:
return util.ByteSize(fieldValue.Int()).String()
}
if isSectionList(fieldValue.Type()) {
var items []map[string]any
for i := range fieldValue.Len() {
@@ -256,11 +264,13 @@ func exportValue(fieldValue reflect.Value) any {
}
return items
}
value := fieldValue.Interface()
if list, ok := value.([]string); ok {
// A copy, so changing it doesn't change the configuration.
return slices.Clone(list)
}
return value
}
@@ -292,7 +302,7 @@ func typeOf(value any, key string) (reflect.Kind, error) {
if err != nil {
return reflect.Invalid, err
}
return kindOf(configField.typ), nil
return kindOf(configField.fieldType), nil
}
// valueOf returns the value of the dotted key in the section value, or
@@ -318,22 +328,25 @@ func setValue(ptr any, key string, value any) error {
// validator, checks it, before setting the field. On error the field is left
// unchanged.
func setField(section reflect.Value, configField field, value any) error {
newValue, err := convert(value, configField.typ)
newValue, err := convert(value, configField.fieldType)
if err != nil {
return err
}
if checker, ok := section.Addr().Interface().(validator); ok {
if err := checker.validate(configField.key, newValue.Interface()); err != nil {
err := checker.validate(configField.key, newValue.Interface())
if err != nil {
return err
}
}
section.FieldByIndex(configField.index).Set(newValue)
return nil
}
// convert turns value into a value of type targetType. Values already of the right
// type are used as-is, except that an empty []string becomes nil, as an empty
// comma-separated list does (see splitList); strings are parsed. Lists of
// convert turns value into a value of type targetType. Values already of the
// right type are used as-is, except that an empty []string becomes nil, as an
// empty comma-separated list does (see splitList); strings are parsed. Lists of
// sections are converted by convertSectionList.
func convert(value any, targetType reflect.Type) (reflect.Value, error) {
if isSectionList(targetType) {
@@ -351,36 +364,38 @@ func convert(value any, targetType reflect.Type) (reflect.Value, error) {
return parseString(text, targetType)
}
}
return reflect.Value{}, fmt.Errorf("unsupported type %T", value)
return reflect.Value{}, fmt.Errorf("config: unsupported type %T", value)
}
// convertSectionList turns value into a list of sections of type listType: a
// value of type listType, a list of maps of keys to values, or a JSON array of objects.
// nil and empty lists become a nil list.
func convertSectionList(value any, listType reflect.Type) (reflect.Value, error) {
// value of type listType, a list of maps of keys to values, or a JSON array of
// objects. nil and empty lists become a nil list.
func convertSectionList(
value any, listType reflect.Type,
) (reflect.Value, error) {
if text, ok := value.(string); ok {
var items []map[string]any
if strings.TrimSpace(text) != "" {
if err := json.Unmarshal([]byte(text), &items); err != nil {
return reflect.Value{}, fmt.Errorf(
"unable to convert to a list of %s: %w",
listType.Elem(),
err,
"unable to convert to a list of %s: %w", listType.Elem(), err,
)
}
}
value = items
}
if value == nil {
return reflect.Zero(listType), nil
}
reflected := reflect.ValueOf(value)
if reflected.Kind() != reflect.Slice {
return reflect.Value{}, fmt.Errorf("unsupported type %T", value)
return reflect.Value{}, fmt.Errorf("config: unsupported type %T", value)
}
if reflected.Len() == 0 {
return reflect.Zero(listType), nil
}
if reflected.Type().AssignableTo(listType) {
for i := range reflected.Len() {
if err := validateSection(reflected.Index(i)); err != nil {
@@ -389,6 +404,7 @@ func convertSectionList(value any, listType reflect.Type) (reflect.Value, error)
}
return reflected, nil
}
list := reflect.MakeSlice(listType, 0, reflected.Len())
for i := range reflected.Len() {
item, err := sectionFromMap(reflected.Index(i).Interface(), listType.Elem())
@@ -397,6 +413,7 @@ func convertSectionList(value any, listType reflect.Type) (reflect.Value, error)
}
list = reflect.Append(list, item)
}
return list, nil
}
@@ -408,28 +425,32 @@ func convertSectionList(value any, listType reflect.Type) (reflect.Value, error)
func sectionFromMap(value any, structType reflect.Type) (reflect.Value, error) {
item, ok := value.(map[string]any)
if !ok {
return reflect.Value{}, fmt.Errorf("unsupported type %T", value)
return reflect.Value{}, fmt.Errorf("config: unsupported type %T", value)
}
section := reflect.New(structType).Elem()
if err := applyDefaultsTo("", section); err != nil {
return reflect.Value{}, err
}
for itemKey, itemValue := range item {
holder, configField, err := lookup(section, itemKey)
if err != nil {
return reflect.Value{}, fmt.Errorf("%w %q", err, itemKey)
}
if _, isString := itemValue.(string); itemValue != nil && !isString &&
!reflect.TypeOf(itemValue).AssignableTo(configField.typ) {
!reflect.TypeOf(itemValue).AssignableTo(configField.fieldType) {
itemValue = fmt.Sprint(itemValue)
}
if err := setField(holder, configField, itemValue); err != nil {
return reflect.Value{}, fmt.Errorf("%s: %w", itemKey, err)
}
}
if err := validateSection(section); err != nil {
return reflect.Value{}, err
}
return section, nil
}
@@ -441,37 +462,41 @@ func validateSection(section reflect.Value) error {
if !ok {
return nil
}
for _, configField := range fieldsOf(section.Type()) {
if configField.section {
continue
}
if err := checker.validate(
configField.key,
section.FieldByIndex(configField.index).Interface(),
); err != nil {
value := section.FieldByIndex(configField.index).Interface()
if err := checker.validate(configField.key, value); err != nil {
return fmt.Errorf("%s: %w", configField.key, err)
}
}
return nil
}
// parseString parses text as a value of type targetType. It supports strings, bools,
// signed and unsigned integers, floats, time.Duration, ByteSize and []string
// (from a comma-separated list, see splitList).
// parseString parses text as a value of type targetType. It supports strings,
// bools, signed and unsigned integers, floats, time.Duration, ByteSize and
// []string (from a comma-separated list, see splitList).
func parseString(text string, targetType reflect.Type) (reflect.Value, error) {
parsed := reflect.New(targetType).Elem()
switch targetType {
case durationType:
duration, err := time.ParseDuration(text)
if err != nil {
return reflect.Value{}, fmt.Errorf("unable to convert to duration: %w", err)
return reflect.Value{}, fmt.Errorf(
"unable to convert to duration: %w", err,
)
}
parsed.SetInt(int64(duration))
return parsed, nil
case byteSizeType:
size, err := util.ParseByteSize(text)
if err != nil {
return reflect.Value{}, fmt.Errorf("unable to convert to byte size: %w", err)
return reflect.Value{}, fmt.Errorf(
"unable to convert to byte size: %w", err,
)
}
parsed.SetInt(int64(size))
return parsed, nil
@@ -492,7 +517,11 @@ func parseString(text string, targetType reflect.Type) (reflect.Value, error) {
var number int64
number, err = strconv.ParseInt(text, 10, targetType.Bits())
parsed.SetInt(number)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
case reflect.Uint,
reflect.Uint8,
reflect.Uint16,
reflect.Uint32,
reflect.Uint64:
var number uint64
number, err = strconv.ParseUint(text, 10, targetType.Bits())
parsed.SetUint(number)
@@ -501,11 +530,16 @@ func parseString(text string, targetType reflect.Type) (reflect.Value, error) {
number, err = strconv.ParseFloat(text, targetType.Bits())
parsed.SetFloat(number)
default:
return reflect.Value{}, fmt.Errorf("unsupported type string for %s", targetType)
return reflect.Value{}, fmt.Errorf(
"config: unsupported type string for %s", targetType,
)
}
if err != nil {
return reflect.Value{}, fmt.Errorf("unable to convert %q to %s: %w", text, targetType, err)
return reflect.Value{}, fmt.Errorf(
"unable to convert %q to %s: %w", text, targetType, err,
)
}
return parsed, nil
}
@@ -521,16 +555,21 @@ func applyDefaults(ptr any) error {
func applyDefaultsTo(prefix string, section reflect.Value) error {
for _, configField := range fieldsOf(section.Type()) {
key := joinKey(prefix, configField.key)
if configField.section {
if err := applyDefaultsTo(key, section.FieldByIndex(configField.index)); err != nil {
nested := section.FieldByIndex(configField.index)
if err := applyDefaultsTo(key, nested); err != nil {
return err
}
continue
}
if !configField.hasDefault {
continue
}
if err := setField(section, configField, configField.def); err != nil {
err := setField(section, configField, configField.defaultValue)
if err != nil {
return fmt.Errorf("invalid default for %s: %w", key, err)
}
}
internal/config/section_test.go
+19 -9
View File
@@ -222,7 +222,8 @@ func TestUnknownKeys(t *testing.T) {
}
var s testSection
for _, key := range keys {
if k, err := typeOf(s, key); !errors.Is(err, errUnknownKey) || k != reflect.Invalid {
if k, err := typeOf(s, key); !errors.Is(err, errUnknownKey) ||
k != reflect.Invalid {
t.Errorf("typeOf(%q) = %v, %v; want Invalid, errUnknownKey", key, k, err)
}
if v, err := valueOf(s, key); !errors.Is(err, errUnknownKey) || v != nil {
@@ -266,17 +267,24 @@ func TestSetValue(t *testing.T) {
{"slice", []string{}, []string(nil)}, // empty lists are stored as nil
// lists of sections are given out as lists of maps; items start
// from their defaults and their values are parsed
{"items", `[{"n": "1", "wait": "2s"}, {"name": "b", "n": 2}]`, []map[string]any{
{"name": "item", "n": 1, "wait": "2s"},
{"name": "b", "n": 2, "wait": "0s"},
}},
{
"items",
`[{"n": "1", "wait": "2s"}, {"name": "b", "n": 2}]`,
[]map[string]any{
{"name": "item", "n": 1, "wait": "2s"},
{"name": "b", "n": 2, "wait": "0s"},
},
},
{"items", []any{map[string]any{"name": "a", "n": 3}}, []map[string]any{
{"name": "a", "n": 3, "wait": "0s"},
}},
{"items", []map[string]any{{"name": "a"}}, []map[string]any{
{"name": "a", "n": 0, "wait": "0s"},
}},
{"items", []testItem{{Name: "a", N: 4}}, []map[string]any{
{"items", []testItem{{
Name: "a",
N: 4,
}}, []map[string]any{
{"name": "a", "n": 4, "wait": "0s"},
}},
{"items", "", []map[string]any(nil)},
@@ -318,9 +326,9 @@ func TestSetValueErrors(t *testing.T) {
{"on", "maybe", "unable to convert"},
{"dur", "soon", "unable to convert to duration"},
{"items", "not json", "unable to convert to a list of config.testItem"},
{"items", `[{"bogus": 1}]`, `item 0: unknown configuration key "bogus"`},
{"items", `[{"bogus": 1}]`, `item 0: config: unknown configuration key "bogus"`},
{"items", `[{"n": "x"}]`, "item 0: n: unable to convert"},
{"items", []any{"a"}, "item 0: unsupported type string"},
{"items", []any{"a"}, "item 0: config: unsupported type string"},
{"items", 5, "unsupported type int"},
{"size", "lots", "unable to convert to byte size"},
{"size", "-1KB", "unable to convert to byte size"},
@@ -344,7 +352,9 @@ func TestSetValueErrors(t *testing.T) {
)
}
if !reflect.DeepEqual(s, before) {
t.Errorf("setValue(%q, %#v) modified the section on error", tt.key, tt.value)
t.Errorf(
"setValue(%q, %#v) modified the section on error", tt.key, tt.value,
)
}
}
}