Added CODE_CONVENTIONS.md and linters to enforce it, and applied it across the codebase

CODE_CONVENTIONS.md sets out the conventions for Go code in this
repository: naming, layout, comments, errors, logging, configuration and
tests. .golangci.json enables the linters that check the parts that can be
checked (each limited to the rules the conventions use); the rest is
applied by hand.

Applied across the codebase:
- Naming: full-word names in application code, initialisms in one case
  (BuyerID, PrintJSON, PrintYAML, ocdsGetBuyerID), no names shadowing
  built-ins (min/max are minGap/maxGap), mustRaw renamed rawJSON as it
  returns nil rather than panicking, and config.ConfigLevelToLogLevel
  renamed LevelToLogLevel.
- Layout: lines kept to 80 characters where possible, assignments taken
  out of if and for statements that don't fit, wrapped arguments all on
  one line or one per line, steps separated by blank lines, no else after
  a return, function and struct literals wrapped.
- Comments: doc comments on every exported name (including each section's
  Map, Flatten, TypeOf, ValueOf and SetValue), section banners, British
  spelling.
- Errors: a created error starts with its package name, a wrapping error
  adds only what was being done, sentinels in one commented var block,
  errors.Is and errors.As in place of == and type assertions, and no
  err := shadowing of a named result. That fixes the mock server losing
  its access log's close error.
- Logging: the fetcher passes its workers an untagged logger, so
  findatender's lines are tagged with their own component rather than
  with fetcher's as well; cmd/mock-fts logs "stopped" with slog.
- Configuration: util.WithTimeout (moved from the websocket package)
  applies a timeout of 0 or less as no limit, used for the websocket
  write and ping, TCP close and tcpserver shutdown timeouts; every setting
  documents what 0 or empty means, including the rate limit rule's keys
  in internal/config/README.md.
- Tests: related tests merged into table-driven ones with named cases.

Separately from CODE_CONVENTIONS.md, internal/store/README.md gains a soft
SQL style convention for the queries in internal/store/queries.go, and the
queries are reformatted to follow it.
This commit is contained in:
bwilliams committed 2026-10-15 16:26:42 +01:00
1 parent 465358609a
commit 67215a6aaa
104 files changed
+3187 -1204

No files matched your search

+100 -2
View File
@@ -9,19 +9,111 @@
"cyclop", "cyclop",
"dogsled", "dogsled",
"errcheck", "errcheck",
"errorlint",
"gocritic",
"godox", "godox",
"govet",
"gosec", "gosec",
"govet",
"ineffassign", "ineffassign",
"misspell",
"nilerr", "nilerr",
"nilnil", "nilnil",
"nolintlint", "nolintlint",
"revive",
"sloglint",
"thelper",
"unused", "unused",
"whitespace" "varnamelen",
"whitespace",
"wsl_v5"
], ],
"settings": { "settings": {
"cyclop": { "cyclop": {
"max-complexity": 15 "max-complexity": 15
},
"misspell": {
"locale": "UK",
"ignore-rules": [
"organization",
"humanize"
]
},
"sloglint": {
"no-mixed-args": true,
"key-naming-case": "snake",
"static-msg": true
},
"revive": {
"rules": [
{
"name": "context-as-argument",
"arguments": [
{
"allowTypesBefore": "*testing.T"
}
]
},
{
"name": "error-naming"
},
{
"name": "error-strings"
},
{
"name": "errorf"
},
{
"name": "exported"
},
{
"name": "indent-error-flow"
},
{
"name": "receiver-naming"
},
{
"name": "superfluous-else"
},
{
"name": "var-naming"
}
]
},
"varnamelen": {
"ignore-names": [
"err",
"ctx",
"i",
"idx",
"msg",
"w",
"r",
"req",
"res",
"db",
"tx",
"wg",
"mu",
"t",
"ok",
"id",
"ip"
],
"ignore-decls": []
},
"gocritic": {
"disable-all": true,
"enabled-checks": [
"builtinShadow",
"emptyStringTest",
"paramTypeCombine"
]
},
"wsl_v5": {
"default": "none",
"enable": [
"err"
]
} }
}, },
"exclusions": { "exclusions": {
@@ -39,6 +131,12 @@
"gosec" "gosec"
], ],
"text": "G104" "text": "G104"
},
{
"path": "_test\\.go",
"linters": [
"varnamelen"
]
} }
] ]
} }
+786
View File
@@ -0,0 +1,786 @@
# Code conventions
These are the conventions for Go code in this repository. Most formatting
is enforced by `golangci-lint` (see [Tooling](#tooling)); the rest are
applied by hand and checked in review.
Generated code is never edited to follow these conventions. It is
recognised by Go's standard marker, a comment before the package clause:
```go
// Code generated by ocdsgen from extensions.json. DO NOT EDIT.
```
## Tooling
```bash
golangci-lint run --build-tags sqlite_fts5 ./... # lint
golangci-lint fmt ./... # format (gofmt, goimports, gci, golines)
```
- Anything that touches the store needs the `sqlite_fts5` build tag (the
title search uses SQLite FTS5); `scripts/build.sh` passes it.
- `golines` limits lines to 100 characters, counting a tab as 2. See
[Line length](#line-length) for the length to aim for.
- In `_test.go` files, `.golangci.json` turns off `errcheck` and
`cyclop`, and gosec's G104 only (it duplicates errcheck). Every other
linter still runs on tests.
### Linters that check these conventions
Several of the conventions below are checked by linters enabled in
`.golangci.json`, so `golangci-lint run` reports them rather than review:
| Linter | Checks | Settings |
|---|---|---|
| `revive` | doc comments on exported names ([Doc comments](#doc-comments)), consistent receiver names, `ctx` first ([Functions and types](#functions-and-types)), [initialisms](#initialisms), lowercase error strings and `Err…` names ([Error messages](#error-messages)), [no `else` after a `return`](#no-else-after-a-return) | Only these rules are enabled. `context-as-argument` allows a `*testing.T` before `ctx`, for test helpers. |
| `errorlint` | errors are compared with `errors.Is` and `errors.As`, not `==` or a type assertion, so wrapped errors still match | |
| `misspell` | British spelling ([Spelling](#spelling)) in comments, strings and names | `locale: UK`. `organization` and `humanize` are ignored: they are external names (OCDS's `Organization`, the `go-humanize` package). |
| `sloglint` | snake_case keys, no mixing of key/value pairs with `slog.Attr`, constant messages ([Logging](#logging)); `slog.DiscardHandler` for discarded output | |
| `thelper` | test helpers call `t.Helper()` first ([Tests](#tests)) | |
| `gocritic` | neighbouring parameters and results of the same type share one type name, and strings are tested against `""` ([Functions and types](#functions-and-types)); no names shadowing built-ins ([Naming](#naming)) | Only `paramTypeCombine`, `emptyStringTest` and `builtinShadow` are enabled. |
| `wsl_v5` | no blank line between a call and the `if err != nil` that checks it ([Blank lines](#blank-lines)) | Only the `err` check is enabled; the rest of wsl's rules conflict with the Blank lines convention. |
| `varnamelen` | short names used over a wide scope ([Naming](#naming)) | Off in `_test.go` files. The short names in the Naming table are allowed. |
A finding that is a false positive is silenced with a `//nolint`
directive and a reason, as for any linter (see
[Other `//nolint` directives](#other-nolint-directives)).
## Files and packages
- File names are lowercase. Compound names run together (`findatender`,
`ftsmock`, `ratelimits`); underscores separate parts
(`conf_source_ftsmock.go`, `confirm_action.go`).
- Never end a file name in `_<GOOS>` or `_<GOARCH>` (e.g. `_js`, `_arm`):
Go treats it as a build constraint.
- Package directories are lowercase, with no underscores.
## Imports
Three groups, each separated by a blank line: the standard library, then
third-party packages, then this module's own packages.
```go
import (
"context"
"net/http"
"github.com/go-chi/chi/v5"
"git.avsdev.uk/Go/job-search/internal/config"
)
```
`golangci-lint fmt` applies this (gci sections `standard`, `default`,
`localmodule`; goimports' local prefix is the module path).
## Naming
Variables and parameters in application code have descriptive full-word
names, not single letters or abbreviations.
| Use | Not |
|---|---|
| `listen`, `newConn`, `client`, `limiter` | `ln`, `nc`, `c`, `lim` |
| `shutdownCtx`, `writeCtx` | `shCtx`, `wctx` |
| `release`, `opportunity`, `entry`, `value` | `r`, `t`, `e`, `v` |
These short names are conventional enough to keep:
| Name | For |
|---|---|
| a short method receiver | e.g. `h *Hub`, `t *tcpConn`, `s *Store` |
| `err`, `ctx` | errors and contexts |
| `i`, `idx` | a plain index loop |
| `msg` | a message |
| `w`, `r` | `http.ResponseWriter` and `*http.Request` in HTTP handlers and middleware, as in the standard library |
| `req`, `res` | a request and a response elsewhere, e.g. building an `*http.Request` or receiving an `*http.Response` in client code |
| `db`, `tx` | `*sql.DB` and a database transaction (`*sql.Tx`, `*sqlx.Tx`) |
| `wg`, `mu` | `sync.WaitGroup` and `sync.Mutex` |
| `t` | `*testing.T`, including in test helper packages such as `internal/cli/cmdtest` |
`w` and `r` are used for nothing else: a release is `release`, an
`io.Writer` is `out`, a `*rand.Rand` is `rng`.
Names never shadow Go's predeclared identifiers: not `min`, `max`,
`len`, `cap`, `new`, `copy`, `clear`, `close`, `error`, `string` and so
on. A gap's bounds are `minGap` and `maxGap`, not `min` and `max`, so the
built-in `min` and `max` still work inside the function.
`tx` means only a database transaction. It is not used for transmit (or
`rx` for receive), as in tx/rx: name those `transmit` and `receive`, or
more specifically (e.g. `sent`, `received`).
Test files are exempt from this rule: short names (`h`, `c`, `e`, `ln`,
`tc`) are fine there, and existing ones shouldn't be renamed.
### Initialisms
Initialisms and acronyms keep one case throughout a name, as in the
standard library: `ID`, `URL`, `URI`, `JSON`, `HTTP`, `API`, `UUID`,
`SQL`. So `ReleaseID`, `APIRoot`, `UUIDType` and `baseURL`, not
`ReleaseId`, `ApiRoot` or `baseUrl`. At the start of an unexported name
the whole initialism is lowercase: `uuid`, `apiRoot`, `jsonHandler`.
`revive` checks this (`var-naming`).
### `must` functions
A name starting with `must` (or `Must`) promises that the function never
returns an error to its caller, as in the standard library's
`regexp.MustCompile` and `template.Must`:
- In tests, a `must…` helper takes `t` first, calls `t.Helper()`, and
stops the test with `t.Fatal` on error: `mustSave(t, s, opportunity)`.
- In application code, a `must…` function panics on error, and is only
used where the input is known to be valid, such as constants or
package-level setup.
- A function that hides an error some other way, such as returning a zero
value or `nil`, isn't named `must…`: `rawJSON` returns `nil` for a value
it can't marshal.
## Functions and types
- A function that takes a context takes it first, named `ctx`:
`func (s *Store) Get(ctx context.Context, id string)`.
- Constructors are `New` when a package has one main type
(`tcpserver.New`), otherwise `NewX` (`NewHub`, `NewKeyedLimiter`).
- All of a type's methods use the same receiver name, usually its first
letter (`h *Hub`, `t *tcpConn`), for value and pointer receivers alike.
- A string is tested for emptiness against `""`, not with `len`:
`if file == ""`, not `if len(file) == 0`. It says what is meant, and
reads the same as any other string comparison.
- Neighbouring parameters or results of the same type share one type
name, as in the standard library: `func f(ocid, id string)`, not
`func f(ocid string, id string)`. Parameters are never reordered just
to share a type.
### No `else` after a `return`
When an `if` block ends in a `return` (or `break`, `continue`, `panic`),
there is no `else`: the code that would be in it follows the `if`,
un-indented. The usual path then reads straight down the left edge of
the function, with the early exits indented, which is how Go code is
normally read (see Effective Go and the Go Code Review Comments, "Indent
Error Flow").
```go
if blockShutdown {
return sleep(taskCtx, true, sleepDuration)
}
return sleep(ctx, false, sleepDuration)
```
Not:
```go
if blockShutdown {
return sleep(taskCtx, true, sleepDuration)
} else {
return sleep(ctx, false, sleepDuration)
}
```
Where there are several outcomes, none of which is the usual path, a
`switch` with no condition reads better than nested `if`/`else`:
```go
switch {
case errors.As(err, &notFound):
slog.Warn("no config file found; using defaults")
case err != nil:
return err
default:
slog.Info("using config file", "file", viper.ConfigFileUsed())
}
```
`revive` checks this (`indent-error-flow`, `superfluous-else`).
## Layout
These rules are applied by hand and checked in review; `golines` only
enforces the 100-character limit. They apply to tests as well as
application code, except where a rule says otherwise.
### Line length
Lines should be at most 80 characters (counting a tab as 2), so code
reads well side by side and in diffs. 100 is the absolute maximum,
enforced by `golines`.
Go past 80 only when breaking the line would make it harder to read:
- a line ending in a `//nolint` directive, which must stay on the line it
applies to;
- struct tags;
- string literals that would lose meaning or searchability if split:
URLs, error messages, regular expressions, JSON fixtures, help text;
- rows of table-driven tests.
Don't run `golines` at 80 to do this: it splits method chains, separates
slog keys from their values and moves `//nolint` directives off their
lines.
### Assignments in `if` and `for`
When an `if` statement's assignment and check don't fit within the line
length together, the assignment is taken out of the `if`, rather than the
call being wrapped inside it. That keeps the condition readable,
particularly an `if err != nil`, which would otherwise be left on the
call's closing line, after its arguments:
```go
_, err = tx.ExecContext(
ctx,
deleteOpportunityClassifications,
opportunity.UUID,
)
if err != nil {
return err
}
```
Not:
```go
if _, err := tx.ExecContext(
ctx,
deleteOpportunityClassifications,
opportunity.UUID,
); err != nil {
return err
}
```
The same applies to a `for` statement's init statement, and to a range
expression that doesn't fit on the `for` line: it is assigned to a
variable first, so the loop's header stays on one line, next to its body.
```go
marks := []struct {
filter MarkFilter
clause string
}{
{filter: filter.Hidden, clause: filterOpportunityMarkHidden},
{filter: filter.Skipped, clause: filterOpportunityMarkSkipped},
}
for _, mark := range marks {
```
Tests are exempt from this rule.
### Wrapped calls and declarations
When a call doesn't fit on one line, it breaks straight after the opening
bracket and the closing bracket goes on a line of its own. If the
arguments all fit on one line between them, they may stay together on
that line, followed by a comma:
```go
ctx, stop := signal.NotifyContext(
context.Background(), os.Interrupt, syscall.SIGTERM,
)
```
Otherwise each argument goes on its own line, followed by a comma. The
arguments are either all on one line or one per line, never grouped two
or three to a line:
```go
server := tcpserver.New(
&conf.Server,
websocket.NewHub(conf.Server.WebSocket, websocket.WithLogger(log)),
tcpserver.WithLogger(log),
)
```
Not:
```go
server := tcpserver.New(&conf.Server,
websocket.NewHub(conf.Server.WebSocket, websocket.WithLogger(log)),
tcpserver.WithLogger(log))
```
A function declaration whose parameters don't fit on one line wraps the
same way:
```go
func insertReleaseWithTags(
ctx context.Context, tx *sqlx.Tx, release types.OpportunityRelease,
) (bool, error) {
```
A `log/slog` call that fits on one line stays on one line, like any other
call. When it doesn't, its attributes are the exception to the rule above:
the message goes on its own line, then each field name stays on the same
line as its value, one field per line, even when the fields would all fit
together on one line.
```go
slog.Info(
"fetched opportunities",
"source", "find_a_tender",
"count", len(opportunities),
"since", since,
)
```
A `//nolint` directive on a wrapped call goes on its first line, after the
opening bracket (see [Ignoring an error](#ignoring-an-error)).
### Method chains
A chain of calls that fits on one line stays on one line. One that
doesn't breaks after each `.`, one call per line, indented once, with
the dot at the end of the line:
```go
findATender.httpClient = resty.New().
SetTimeout(60 * time.Second).
SetRetryCount(1)
```
A short accessor at the start of a chain, such as `h.logger()` or
`cmd.Flags()`, stays with the call after it; if that call doesn't fit,
its arguments are wrapped instead (see
[Wrapped calls and declarations](#wrapped-calls-and-declarations)):
```go
h.logger().Info(
"client connected",
"client", client.id,
"ip", ip,
)
```
Not:
```go
h.logger().
Info("client connected", "client", client.id, "ip", ip)
```
### Long conditions
A condition that doesn't fit on one line breaks after a `||` or `&&`,
which ends the line, and continues indented once. Each line is filled up
to the line length, keeping related comparisons together, rather than
one comparison per line:
```go
if len(cat.Buyers) == 0 || len(cat.Suppliers) == 0 ||
len(cat.Subjects) == 0 || len(cat.Methods) == 0 {
```
A condition is never broken inside a call's brackets; if a call in it
doesn't fit, its result is assigned to a variable first (see
[Assignments in `if` and `for`](#assignments-in-if-and-for)). A
condition that runs to more than about three lines is better given a
name, as a boolean variable or a small function.
### Blank lines
A function body is split into steps, with one blank line between them. A
step is a statement, or a few closely related ones, together with its
error handling. Go itself has no rule for this (`gofmt` only collapses
several blank lines into one, and the `whitespace` linter only rejects
blank lines at the start or end of a block), so it is applied by hand.
- A call and the `if err != nil` that checks it are never separated by a
blank line. `wsl_v5` checks this; the rest of the rule is applied
by hand.
- A `defer` that releases something goes straight after the check of the
call that acquired it, with no blank line, so acquiring and releasing
read as one step.
- A function or block of only one or two steps needs no blank lines.
- A final `return` after several (three or more) steps is set off by a
blank line. The `return` itself isn't counted as a step.
```go
statement, err := tx.PrepareNamedContext(
ctx, upsertClassificationCodeReturningID,
)
if err != nil {
return err
}
defer func() {
err = errors.Join(err, statement.Close())
}()
for i := range opportunity.Classifications {
...
}
return nil
```
### Function literals
A function literal (`func(...) { ... }` written inline) is never written on
one line, however short its body:
```go
t.Cleanup(func() {
server.Close()
})
```
Not `t.Cleanup(func() { server.Close() })`.
This is about function literals only. Named one-line functions and methods,
such as a config section's `Map() map[string]any { return mapOf(c) }`, are
unaffected.
### Struct literals
A struct literal is always wrapped, one field per line, and names its
fields:
```go
envelope := websocket.Envelope{
Type: websocket.TypePing,
ID: "1",
}
```
Not `websocket.Envelope{Type: websocket.TypePing, ID: "1"}`, and never
positional (`websocket.Envelope{"ping", "1", nil, ""}`).
- Empty struct literals (`netip.Addr{}`, `time.Time{}`, `struct{}{}`,
`&Server{}`) have nothing to wrap and stay on one line.
- Rows of a table-driven test (`{"size", "4096", "4KiB"},`) stay one row per
line, so the table stays easy to scan. A struct literal used as a value
inside a row is still wrapped.
- Rows of a short lookup table in application code (a slice or map
literal of small structs, used as data) may each stay on one line if
they fit within the line length, so the table is easy to scan. They
still name their fields:
`{filter: filter.Hidden, clause: filterOpportunityMarkHidden},`.
- This is about struct literals only; slice and map literals
(`[]string{"a", "b"}`) may stay on one line.
## Comments
### Doc comments
Every exported identifier has a doc comment, and it starts with the name
it documents.
```go
// NewHub creates a hub.
func NewHub(conf config.JSCServerWebSocket, opts ...Option) *Hub {
```
### Grouped declarations
Related package-level constants and variables go in one block, with a
comment for the group and one for each member.
```go
// Errors returned by CSRF.Validate for a token that is rejected.
var (
// ErrCSRFMissing is returned when no token is presented.
ErrCSRFMissing = errors.New("csrf token missing")
// ErrCSRFExpired is returned for a token older than the token TTL.
ErrCSRFExpired = errors.New("csrf token expired")
)
```
### Section banners
A file is divided into sections with a banner, followed by a blank line
so godoc doesn't attach the banner to the next declaration:
```go
// ---------- tcpConn ----------
// tcpConn adapts a net.Conn ...
type tcpConn struct {
```
In a file with more than one type (such as a config section and its
nested section), each type gets a `==========` sub-section, which then
has its own `----------` banners:
```go
// ========== HSTS Headers ==========
// JSCServerHeadersHSTS configures the Strict-Transport-Security header.
type JSCServerHeadersHSTS struct {
...
}
// ---------- Section methods (see Section) ----------
```
`internal/store/queries.go` uses `----------` banners for its per-table
groups of queries.
### Trailing comments
A comment after code on the same line is for a short note, and only when
the line still fits within the [line length](#line-length). Otherwise
the comment goes on its own line above the code it describes:
```go
// location is the zone for updatedFrom/updatedTo without an offset;
// nil means UTC.
location *time.Location
```
Not a trailing comment that runs the line past 80, and never a comment
after the closing bracket of a wrapped call.
A comment on a row of a table-driven test stays on the row, like the row
itself (see [Line length](#line-length)).
`//nolint` directives are exempt: a directive must be on the line the
linter reports, so it stays a trailing comment even when that takes the
line past 80 (see [Ignoring an error](#ignoring-an-error)).
### Spelling
Comments and documentation use British English ("normalised", "honoured",
"recognise", "behaviour"). Identifiers, and anything that must match an
external name, keep their own spelling (`Authorization`, `Color` in a
third-party API).
## Errors
### Error messages
- Error strings are lowercase and don't end with punctuation, as they are
usually wrapped into a longer message (`revive`'s `error-strings`
reports capitalised ones): `errors.New("store: not found")`.
- An error from another call is wrapped with `%w`, adding what was being
done, so the chain reads as a sentence:
`fmt.Errorf("listen %s: %w", addr, err)`.
### Package prefixes
An error names the package it came from once, where it is created, as
the standard library does (`sql: no rows in result set`, `http: server
closed`) and Effective Go recommends:
- An error a package creates itself, a sentinel, `errors.New`, or
`fmt.Errorf` without `%w`, starts with the package name:
`errors.New("store: not found")`,
`fmt.Errorf("util: byte size %q is too large", text)`.
- Wrapping an error adds what was being done, never a package name:
`fmt.Errorf("inserting release %s/%s: %w", uuid, id, err)`, not
`"store: inserting release …"`. The wrapped error already says where it
came from, so a chain reads `writing opportunity ocds-1: store: …`
rather than repeating `store:` at every level.
Two kinds of error are exempt and have no prefix:
- Errors from `package main` and the command packages in `internal/cli`.
Their text is the final message shown to the user, who has no use for
an internal package name.
- Error text that is part of a protocol sent to a client, such as the
mock Find a Tender API's error responses, which match the real API's.
- Errors that callers test for are sentinel values named `Err…`
(exported) or `err…` (unexported), checked with `errors.Is`. A
package's sentinels go in one commented `var` block (see
[Grouped declarations](#grouped-declarations)).
### Ignoring an error
An error is never ignored with `_ = f()`. Instead, the call says why
with a `//nolint` directive naming only the linters that actually report
the line (`nolintlint` reports any that don't):
```go
json.NewEncoder(w).Encode(problem) //nolint:errcheck,gosec // the client may have gone
```
- Most calls need `errcheck,gosec`. `hash.Write` needs only `gosec`
(errcheck excludes it); `fmt.Fprint*` needs only `errcheck` (gosec
excludes it).
- A deferred call needs only `errcheck`, since gosec doesn't check
deferred calls:
```go
defer rows.Close() //nolint:errcheck // read only
```
- For a call spanning several lines, the directive goes on the first
line, after the opening parenthesis:
```go
tcpSocket.SetKeepAliveConfig(net.KeepAliveConfig{ //nolint:errcheck,gosec // best effort
```
A linter that reports one argument rather than the call (such as
`sloglint` on a message that isn't a constant) needs the directive on
that argument's line instead. One that reports several lines of a
statement, such as a table of test cases, takes the directive on a line
of its own directly above the statement.
- A call whose error both linters already exclude (such as
`crypto/rand.Read`, which never returns one) gets a plain comment
rather than a directive, which would be reported as unused:
```go
rand.Read(key) // crypto/rand.Read never returns an error
```
- A multi-value assignment such as `value, _ := f()` can't take a
directive (errcheck doesn't check blank assignments, so the directive
would be unused) and stays as it is. These are mostly cobra flag
getters, which only fail for an undefined flag. Where ignoring the
error is ambiguous or potentially dangerous in future, an explanation
comment can be added.
### Errors that matter
When closing something can lose data (a file being written, a database
handle), its error is joined into the function's returned error with
`errors.Join`, through a named return.
### Named results
A function names its results only when it needs to:
- **A deferred function changes the error.** This is the only reason to
name `err` for its own sake (see [Errors that matter](#errors-that-matter)).
- **The names say which result is which.** When several results share a
type, naming them documents the order, as the Go Code Review Comments
suggest: `func releaseIDs(raw json.RawMessage) (ocid, id string, err error)`.
Go names all of a function's results or none, so `err` is named here too.
Otherwise results are unnamed. A function with named results:
- never uses a bare `return`: each `return` lists what it returns, so a
reader doesn't have to track the named variables to know;
- where a deferred function changes `err`, assigns to it with `=`, never
`:=`, in nested blocks. `err :=` there declares a new variable that
shadows the result, and the error assigned to it never reaches the
deferred `errors.Join`:
```go
for i := range opportunity.Classifications {
err = statement.GetContext(ctx, &classification.ID, classification)
if err != nil {
return err
}
}
```
### Other `//nolint` directives
Otherwise `//nolint` is only for false positives that the code can't
express, always with a reason. For example, a seeded `math/rand` in the
mock, or a gosec finding explained in the comment above it:
```go
// Secure is false only on plain HTTP (local development), where
// browsers refuse Secure cookies; HttpOnly and SameSite are always set.
http.SetCookie(w, &http.Cookie{ //nolint:gosec // G124: see above
```
## Logging
- Logging uses `log/slog`. A component (a type such as `Hub`, or a
package-level entry point such as `fetcher.Run`) logs to `slog.Default()`
unless it is given a logger with a `WithLogger` option, so callers that
are happy with the default pass nothing, and tests can still capture or
discard its output.
- The logger is given to the constructor (or entry point) as an option,
never set afterwards: a setter called once goroutines are running would
be a data race. Options follow the functional options pattern used by
`store.Open`.
- With no logger given, the component looks up `slog.Default()` when it
logs, not when it is created, so it follows `logging.Setup` and a test's
`slog.SetDefault` even if it was created first.
```go
// WithLogger makes the hub log to logger instead of slog.Default().
func WithLogger(logger *slog.Logger) Option {
return func(h *Hub) {
h.log = logger
}
}
// logger returns the logger the hub logs to, tagged with its component.
func (h *Hub) logger() *slog.Logger {
log := h.log
if log == nil {
log = slog.Default()
}
return log.With("component", "ws")
}
```
```go
hub := websocket.NewHub(conf.Server.WebSocket) // slog.Default()
hub := websocket.NewHub(conf.Server.WebSocket, websocket.WithLogger(log)) // given logger
```
- Field keys are snake_case and name one thing each: `client`, `ip`,
`request_id`, `retry_in`. Durations, sizes and counts are logged with
the matching `slog` type, not as pre-formatted strings.
- A component tags its logger with `"component"` in one place (its
`logger()` method, as above), whether the logger was given or is the
default, rather than on every line.
- Messages are short lowercase phrases ("client connected", "tcp accept
failed"); the details go in fields.
## Configuration
### Keys
Configuration keys (the `cfg` struct tags) are snake_case:
`max_message_size`, `connection_limit`. Viper lowercases every key it
reads, so a camelCase key such as `minVersion` is never loaded from a
config file or environment variable.
### Zero values
Every setting's doc comment (and its row in `internal/config/README.md`)
says what 0 or empty means, and the code treats it consistently:
- A timeout of 0 or less means **no limit**, and an interval of 0 or less
means **off** (e.g. `ping_interval`), never "already expired". Where off
makes no sense the setting rejects 0 instead (a source's `interval`: a
source is turned off with `enabled`). Code applies a timeout with a helper
rather than `context.WithTimeout(ctx, 0)`:
```go
writeCtx, cancel := util.WithTimeout(ctx, conf.WriteTimeout) // no deadline if 0 or less
```
- A size of 0 means **the default** (`max_message_size` of `0B` reads
32KiB), and -1 means **no limit** (`util.ByteSizeNoLimit`); no other size
may be negative. A setting for which no limit is unsafe rejects -1 in its
validation and says why (`max_message_size` does: a client could exhaust
the server's memory).
- An empty list means **none** (no trusted origins, no exempt paths), and
is stored as `nil`.
## Tests
- Tests are table-driven where several cases share a check, with a name
or input field identifying each case in failure messages.
- Helpers call `t.Helper()` first, and register their own teardown with
`t.Cleanup` rather than returning a cleanup function.
- Tests that touch global state (Viper, `slog.Default`, the working
directory, environment variables) don't call `t.Parallel()`, and reset
what they change: `resetViper(t)`, `t.Chdir`, `t.Setenv`.
- Network tests listen on `127.0.0.1:0` and use the address they were
given, never a fixed port.
- Short names are fine in tests (see [Naming](#naming)), and setup calls
may ignore errors with a bare call (errcheck is off for tests).
+5 -1
View File
@@ -12,15 +12,19 @@ import (
func TestCommandTree(t *testing.T) { func TestCommandTree(t *testing.T) {
root := newRootCmd() root := newRootCmd()
var names []string var names []string
for _, c := range root.Commands() { for _, c := range root.Commands() {
names = append(names, c.Name()) names = append(names, c.Name())
} }
for _, want := range []string{"config", "fetch", "migrate", "version", "wsserve"} {
commands := []string{"config", "fetch", "migrate", "version", "wsserve"}
for _, want := range commands {
if !slices.Contains(names, want) { if !slices.Contains(names, want) {
t.Errorf("job-search has no %q command; has %v", want, names) t.Errorf("job-search has no %q command; has %v", want, names)
} }
} }
for _, flag := range []string{"config", "debug", "quiet"} { for _, flag := range []string{"config", "debug", "quiet"} {
if root.PersistentFlags().Lookup(flag) == nil { if root.PersistentFlags().Lookup(flag) == nil {
t.Errorf("job-search has no --%s flag", flag) t.Errorf("job-search has no --%s flag", flag)
+52 -30
View File
@@ -52,7 +52,7 @@ type options struct {
rateWindow time.Duration rateWindow time.Duration
accessLogPath string accessLogPath string
logFormat string logFormat string
tz string timeZone string
} }
const longHelp = `Mock of the Find a Tender (FTS) OCDS release package endpoint: const longHelp = `Mock of the Find a Tender (FTS) OCDS release package endpoint:
@@ -170,10 +170,19 @@ func newRootCmd() *cobra.Command {
"maximum time between generated releases", "maximum time between generated releases",
) )
flags.IntVar(&opts.rateLimit, "rate-limit", 12, "requests allowed per window per client") flags.IntVar(
flags.DurationVar(&opts.rateWindow, "rate-window", 5*time.Minute, "rate limit window length") &opts.rateLimit, "rate-limit", 12, "requests allowed per window per client",
)
flags.DurationVar(
&opts.rateWindow, "rate-window", 5*time.Minute, "rate limit window length",
)
flags.StringVar(&opts.accessLogPath, "access-log", "-", `access log file, or "-" for stdout`) flags.StringVar(
&opts.accessLogPath,
"access-log",
"-",
`access log file, or "-" for stdout`,
)
flags.StringVar( flags.StringVar(
&opts.logFormat, &opts.logFormat,
"log-format", "log-format",
@@ -181,7 +190,7 @@ func newRootCmd() *cobra.Command {
`access log format: "combined" or "common"`, `access log format: "combined" or "common"`,
) )
flags.StringVar( flags.StringVar(
&opts.tz, &opts.timeZone,
"tz", "tz",
"Europe/London", "Europe/London",
`time zone for updatedFrom/updatedTo without an offset ("Local" for this machine's)`, `time zone for updatedFrom/updatedTo without an offset ("Local" for this machine's)`,
@@ -193,34 +202,40 @@ func newRootCmd() *cobra.Command {
// validate checks flag values before anything is started. // validate checks flag values before anything is started.
func (o *options) validate() error { func (o *options) validate() error {
if o.logFormat != "combined" && o.logFormat != "common" { if o.logFormat != "combined" && o.logFormat != "common" {
return fmt.Errorf(`--log-format must be "combined" or "common", got %q`, o.logFormat) return fmt.Errorf(
`--log-format must be "combined" or "common", got %q`, o.logFormat,
)
} }
if o.minGap <= 0 || o.maxGap < o.minGap { if o.minGap <= 0 || o.maxGap < o.minGap {
return errors.New("--min-gap must be positive and no larger than --max-gap") return errors.New("--min-gap must be positive and no larger than --max-gap")
} }
if o.rateLimit < 1 || o.rateWindow <= 0 { if o.rateLimit < 1 || o.rateWindow <= 0 {
return errors.New("--rate-limit must be at least 1 and --rate-window positive") return errors.New(
"--rate-limit must be at least 1 and --rate-window positive",
)
} }
if _, err := time.LoadLocation(o.tz); err != nil { if _, err := time.LoadLocation(o.timeZone); err != nil {
return fmt.Errorf("--tz: %w", err) return fmt.Errorf("--tz: %w", err)
} }
return nil return nil
} }
// run starts the mock and blocks until ctx is cancelled or the server fails. // run starts the mock and blocks until ctx is cancelled or the server fails.
func run(ctx context.Context, opts *options) (err error) { func run(ctx context.Context, opts *options) (err error) {
location, err := time.LoadLocation(opts.tz) // already validated location, err := time.LoadLocation(opts.timeZone) // already validated
if err != nil { if err != nil {
return fmt.Errorf("--tz: %w", err) return fmt.Errorf("--tz: %w", err)
} }
if err := os.MkdirAll(opts.dataDir, 0o750); err != nil {
return err if err = os.MkdirAll(opts.dataDir, 0o750); err != nil {
return fmt.Errorf("creating data directory: %w", err)
} }
// ---------- catalog ---------- // Load the catalog.
catalog, err := mock.LoadCatalog() catalog, err := mock.LoadCatalog()
if err != nil { if err != nil {
return err return fmt.Errorf("loading catalog: %w", err)
} }
slog.Info( slog.Info(
"catalog loaded", "catalog loaded",
@@ -231,27 +246,30 @@ func run(ctx context.Context, opts *options) (err error) {
"classifications", len(catalog.Classifications), "classifications", len(catalog.Classifications),
) )
// ---------- store ---------- // Open (or reset) the store.
storePath := filepath.Join(opts.dataDir, mock.ReleasesFile) storePath := filepath.Join(opts.dataDir, mock.ReleasesFile)
if opts.reset { if opts.reset {
if err := os.Remove(storePath); err != nil && !errors.Is(err, os.ErrNotExist) { if err = os.Remove(storePath); err != nil &&
return err !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("resetting store: %w", err)
} }
slog.Info("store reset", "path", storePath) slog.Info("store reset", "path", storePath)
} }
store, err := mock.OpenStore(storePath) store, err := mock.OpenStore(storePath)
if err != nil { if err != nil {
return err return fmt.Errorf("opening store: %w", err)
} }
defer func() { defer func() {
err = errors.Join(err, store.Close()) err = errors.Join(err, store.Close())
}() }()
// ---------- generator ---------- // Backfill releases, then keep generating them.
generator := mock.NewGenerator(catalog, store, time.Now().UnixNano()) generator := mock.NewGenerator(catalog, store, time.Now().UnixNano())
generated, err := generator.CatchUp(time.Now(), opts.backfill, opts.minGap, opts.maxGap) generated, err := generator.CatchUp(
time.Now(), opts.backfill, opts.minGap, opts.maxGap,
)
if err != nil { if err != nil {
return err return fmt.Errorf("backfilling releases: %w", err)
} }
slog.Info("store ready", "releases", store.Len(), "generated", generated) slog.Info("store ready", "releases", store.Len(), "generated", generated)
@@ -267,10 +285,11 @@ func run(ctx context.Context, opts *options) (err error) {
generator.Run(ctx, opts.minGap, opts.maxGap) generator.Run(ctx, opts.minGap, opts.maxGap)
}() }()
// ---------- HTTP ---------- // Serve the API.
out := os.Stdout out := os.Stdout
if opts.accessLogPath != "-" { if opts.accessLogPath != "-" {
logFile, err := os.OpenFile( var logFile *os.File
logFile, err = os.OpenFile(
opts.accessLogPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600, opts.accessLogPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600,
) )
if err != nil { if err != nil {
@@ -288,12 +307,15 @@ func run(ctx context.Context, opts *options) (err error) {
rateLimiter := mock.NewRateLimiter(opts.rateLimit, opts.rateWindow) rateLimiter := mock.NewRateLimiter(opts.rateLimit, opts.rateWindow)
mux := http.NewServeMux() mux := http.NewServeMux()
mux.HandleFunc( mux.HandleFunc(
mock.BasePath, rateLimiter.Middleware(mock.NewHandler(store, opts.baseURL, location)), mock.BasePath,
rateLimiter.Middleware(mock.NewHandler(store, opts.baseURL, location)),
) )
httpSrv := &http.Server{ httpServer := &http.Server{
Addr: opts.addr, Addr: opts.addr,
Handler: logging.AccessLog(accessLogger, opts.logFormat == "combined", mux), Handler: logging.AccessLog(
accessLogger, opts.logFormat == "combined", mux,
),
ReadHeaderTimeout: 10 * time.Second, ReadHeaderTimeout: 10 * time.Second,
} }
go func() { go func() {
@@ -303,7 +325,7 @@ func run(ctx context.Context, opts *options) (err error) {
context.WithoutCancel(ctx), 5*time.Second, context.WithoutCancel(ctx), 5*time.Second,
) )
defer cancelShutdown() defer cancelShutdown()
if err := httpSrv.Shutdown(shutdownCtx); err != nil { if err := httpServer.Shutdown(shutdownCtx); err != nil {
slog.Error("shutdown failed", "error", err) slog.Error("shutdown failed", "error", err)
} }
}() }()
@@ -314,15 +336,15 @@ func run(ctx context.Context, opts *options) (err error) {
"path", mock.BasePath, "path", mock.BasePath,
"time_zone", location.String(), "time_zone", location.String(),
) )
serveErr := httpSrv.ListenAndServe() serveErr := httpServer.ListenAndServe()
cancel() // stop the worker (and the shutdown goroutine) if the server failed cancel() // stop the worker (and the shutdown goroutine) if the server failed
<-workerDone <-workerDone
if serveErr != nil && !errors.Is(serveErr, http.ErrServerClosed) { if serveErr != nil && !errors.Is(serveErr, http.ErrServerClosed) {
return serveErr return fmt.Errorf("serving: %w", serveErr)
} }
log.Print("stopped") slog.Info("stopped")
return nil return nil
} }
+11 -9
View File
@@ -52,9 +52,11 @@ func newCommand(t *testing.T, opts *Options, args ...string) *cobra.Command {
Use: "test", Use: "test",
} }
opts.Bind(cmd) opts.Bind(cmd)
if err := cmd.ParseFlags(args); err != nil { if err := cmd.ParseFlags(args); err != nil {
t.Fatal(err) t.Fatal(err)
} }
return cmd return cmd
} }
@@ -144,12 +146,16 @@ func TestRunSetupWithoutLoadingConfig(t *testing.T) {
"-d -q": "debug", // --debug wins "-d -q": "debug", // --debug wins
} { } {
opts := &Options{} opts := &Options{}
conf, err := opts.RunSetup(newCommand(t, opts, strings.Fields(args)...), false, true) conf, err := opts.RunSetup(
newCommand(t, opts, strings.Fields(args)...), false, true,
)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if conf.Logging.Level != want { if conf.Logging.Level != want {
t.Errorf("%q: Logging.Level = %q, want %q", args, conf.Logging.Level, want) t.Errorf(
"%q: Logging.Level = %q, want %q", args, conf.Logging.Level, want,
)
} }
} }
} }
@@ -192,22 +198,18 @@ func confirmWithInput(t *testing.T, input string) bool {
dir := t.TempDir() dir := t.TempDir()
in := filepath.Join(dir, "stdin") in := filepath.Join(dir, "stdin")
writeFile(t, in, input) writeFile(t, in, input)
stdin, err := os.Open(filepath.Clean(in)) stdin, err := os.Open(filepath.Clean(in))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
defer stdin.Close() defer stdin.Close()
stdout, err := os.Create(filepath.Clean(filepath.Join(dir, "stdout"))) stdout, err := os.Create(filepath.Clean(filepath.Join(dir, "stdout")))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
defer stdout.Close() defer stdout.Close()
oldIn, oldOut := os.Stdin, os.Stdout return Confirm(stdout, stdin, "Proceed?")
os.Stdin, os.Stdout = stdin, stdout
defer func() {
os.Stdin, os.Stdout = oldIn, oldOut
}()
return Confirm("Proceed?")
} }
+14 -7
View File
@@ -28,10 +28,12 @@ func Setup(t *testing.T, config string) string {
t.Helper() t.Helper()
dir := t.TempDir() dir := t.TempDir()
t.Chdir(dir) t.Chdir(dir)
WriteFile(t, ".env", "") WriteFile(t, ".env", "")
if config != "" { if config != "" {
WriteFile(t, "config.yaml", config) WriteFile(t, "config.yaml", config)
} }
return dir return dir
} }
@@ -78,6 +80,7 @@ func NewRoot(opts *cli.Options, cmds ...*cobra.Command) *cobra.Command {
cobra.EnableTraverseRunHooks = true cobra.EnableTraverseRunHooks = true
opts.Bind(root) opts.Bind(root)
root.AddCommand(cmds...) root.AddCommand(cmds...)
return root return root
} }
@@ -95,18 +98,19 @@ func Run(
dir := t.TempDir() dir := t.TempDir()
inPath, outPath := filepath.Join(dir, "stdin"), filepath.Join(dir, "stdout") inPath, outPath := filepath.Join(dir, "stdin"), filepath.Join(dir, "stdout")
WriteFile(t, inPath, stdin) WriteFile(t, inPath, stdin)
in, err := os.Open(filepath.Clean(inPath)) input, err := os.Open(filepath.Clean(inPath))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
defer in.Close() //nolint:errcheck // read only defer input.Close() //nolint:errcheck // read only
out, err := os.Create(filepath.Clean(outPath)) out, err := os.Create(filepath.Clean(outPath))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
oldIn, oldOut, oldLog := os.Stdin, os.Stdout, slog.Default() oldIn, oldOut, oldLog := os.Stdin, os.Stdout, slog.Default()
os.Stdin, os.Stdout = in, out os.Stdin, os.Stdout = input, out
defer func() { defer func() {
os.Stdin, os.Stdout = oldIn, oldOut os.Stdin, os.Stdout = oldIn, oldOut
slog.SetDefault(oldLog) slog.SetDefault(oldLog)
@@ -120,11 +124,14 @@ func Run(
if err := out.Close(); err != nil { if err := out.Close(); err != nil {
t.Fatal(err) t.Fatal(err)
} }
return ReadFile(t, outPath), runErr return ReadFile(t, outPath), runErr
} }
// MustRun is Run with no input, for commands that must succeed. // MustRun is Run with no input, for commands that must succeed.
func MustRun(t *testing.T, newRoot func() *cobra.Command, args ...string) string { func MustRun(
t *testing.T, newRoot func() *cobra.Command, args ...string,
) string {
t.Helper() t.Helper()
out, err := Run(t, newRoot, "", args...) out, err := Run(t, newRoot, "", args...)
if err != nil { if err != nil {
@@ -136,9 +143,9 @@ func MustRun(t *testing.T, newRoot func() *cobra.Command, args ...string) string
// Contains fails the test unless out contains every string in want. // Contains fails the test unless out contains every string in want.
func Contains(t *testing.T, out string, want ...string) { func Contains(t *testing.T, out string, want ...string) {
t.Helper() t.Helper()
for _, w := range want { for _, wanted := range want {
if !strings.Contains(out, w) { if !strings.Contains(out, wanted) {
t.Errorf("output does not contain %q:\n%s", w, out) t.Errorf("output does not contain %q:\n%s", wanted, out)
} }
} }
} }
+3 -1
View File
@@ -8,7 +8,8 @@ import (
"git.avsdev.uk/Go/job-search/internal/cli" "git.avsdev.uk/Go/job-search/internal/cli"
) )
// NewCommand returns the "config" command, which shows and edits the configuration. // NewCommand returns the "config" command, which shows and edits the
// configuration.
func NewCommand(opts *cli.Options) *cobra.Command { func NewCommand(opts *cli.Options) *cobra.Command {
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "config", Use: "config",
@@ -24,5 +25,6 @@ func NewCommand(opts *cli.Options) *cobra.Command {
cmd.AddCommand(commandGet(opts)) cmd.AddCommand(commandGet(opts))
cmd.AddCommand(commandSet(opts)) cmd.AddCommand(commandSet(opts))
cmd.AddCommand(commandInit()) cmd.AddCommand(commandInit())
return cmd return cmd
} }
+36 -10
View File
@@ -41,9 +41,12 @@ func mustRun(t *testing.T, args ...string) string {
func fileSettings(t *testing.T, path string) map[string]any { func fileSettings(t *testing.T, path string) map[string]any {
t.Helper() t.Helper()
var nested map[string]any var nested map[string]any
if err := yaml.Unmarshal([]byte(cmdtest.ReadFile(t, path)), &nested); err != nil { if err := yaml.Unmarshal(
[]byte(cmdtest.ReadFile(t, path)), &nested,
); err != nil {
t.Fatal(err) t.Fatal(err)
} }
flat := map[string]any{} flat := map[string]any{}
var walk func(prefix string, m map[string]any) var walk func(prefix string, m map[string]any)
walk = func(prefix string, m map[string]any) { walk = func(prefix string, m map[string]any) {
@@ -56,6 +59,7 @@ func fileSettings(t *testing.T, path string) map[string]any {
} }
} }
walk("", nested) walk("", nested)
return flat return flat
} }
@@ -69,11 +73,17 @@ func TestConfigShow(t *testing.T) {
"logging.level = info <string>", "logging.level = info <string>",
"sources.sleeper.interval = 2m0s <string>", "sources.sleeper.interval = 2m0s <string>",
) )
// Keys are listed in order, the same every time. // Keys are listed in order, the same every time.
if again := mustRun(t, "config", "show"); again != out { if again := mustRun(t, "config", "show"); again != out {
t.Errorf("config show output changed between runs:\n%s\n---\n%s", out, again) t.Errorf(
"config show output changed between runs:\n%s\n---\n%s", out, again,
)
} }
if i, j := strings.Index(out, "database.file"), strings.Index(out, "logging.level"); i > j {
database := strings.Index(out, "database.file")
logging := strings.Index(out, "logging.level")
if database > logging {
t.Errorf("keys not sorted:\n%s", out) t.Errorf("keys not sorted:\n%s", out)
} }
@@ -87,9 +97,17 @@ func TestConfigShow(t *testing.T) {
func TestConfigGet(t *testing.T) { func TestConfigGet(t *testing.T) {
cmdtest.Setup(t, appConfig) cmdtest.Setup(t, appConfig)
cmdtest.Contains(t, mustRun(t, "config", "get", "logging.level"), "info <string>") for _, tt := range []struct {
cmdtest.Contains(t, mustRun(t, "config", "get", "sources.sleeper.interval"), "2m0s <string>") key, want string
cmdtest.Contains(t, mustRun(t, "config", "get", "sources.sleeper.enabled"), "false <bool>") }{
{"logging.level", "info <string>"},
{"sources.sleeper.interval", "2m0s <string>"},
{"sources.sleeper.enabled", "false <bool>"},
} {
t.Run(tt.key, func(t *testing.T) {
cmdtest.Contains(t, mustRun(t, "config", "get", tt.key), tt.want)
})
}
if _, err := run(t, "config", "get", "no.such.key"); err == nil { if _, err := run(t, "config", "get", "no.such.key"); err == nil {
t.Error("get of an unknown key succeeded") t.Error("get of an unknown key succeeded")
@@ -104,7 +122,9 @@ func TestConfigSet(t *testing.T) {
out := mustRun(t, "config", "set", "logging.level", "debug") out := mustRun(t, "config", "set", "logging.level", "debug")
cmdtest.Contains(t, out, "logging.level = debug <string>") cmdtest.Contains(t, out, "logging.level = debug <string>")
cmdtest.Contains(t, mustRun(t, "config", "get", "logging.level"), "debug <string>") cmdtest.Contains(
t, mustRun(t, "config", "get", "logging.level"), "debug <string>",
)
out = mustRun(t, "config", "set", "sources.sleeper.enabled", "true") out = mustRun(t, "config", "set", "sources.sleeper.enabled", "true")
cmdtest.Contains(t, out, "sources.sleeper.enabled = true <bool>") cmdtest.Contains(t, out, "sources.sleeper.enabled = true <bool>")
@@ -156,11 +176,13 @@ func TestConfigSetUnset(t *testing.T) {
cmdtest.Setup(t, appConfig) cmdtest.Setup(t, appConfig)
mustRun(t, "config", "set", "--unset", "sources.sleeper.interval") mustRun(t, "config", "set", "--unset", "sources.sleeper.interval")
// Only the unset key is gone: no defaults are written in its place. // Only the unset key is gone: no defaults are written in its place.
want := map[string]any{"logging.level": "info"} want := map[string]any{"logging.level": "info"}
if got := fileSettings(t, "config.yaml"); !equalSettings(got, want) { if got := fileSettings(t, "config.yaml"); !equalSettings(got, want) {
t.Errorf("config.yaml settings = %v; want %v", got, want) t.Errorf("config.yaml settings = %v; want %v", got, want)
} }
// Back to the default. // Back to the default.
cmdtest.Contains( cmdtest.Contains(
t, t,
@@ -180,7 +202,9 @@ func TestConfigInit(t *testing.T) {
out := mustRun(t, "config", "init", "./new.yaml") out := mustRun(t, "config", "init", "./new.yaml")
cmdtest.Contains(t, out, "writing default configuration to file: ./new.yaml") cmdtest.Contains(t, out, "writing default configuration to file: ./new.yaml")
cmdtest.Contains(t, cmdtest.ReadFile(t, "new.yaml"), "level: warn", "interval: 10m0s") cmdtest.Contains(
t, cmdtest.ReadFile(t, "new.yaml"), "level: warn", "interval: 10m0s",
)
_, err := run(t, "config", "init", "./new.yaml") _, err := run(t, "config", "init", "./new.yaml")
cmdtest.ErrContains(t, err, "already exists") cmdtest.ErrContains(t, err, "already exists")
@@ -199,8 +223,10 @@ func TestConfigInitDefaultLocations(t *testing.T) {
{"config", "init"}, {"config", "init"},
{"config", "init", " "}, {"config", "init", " "},
} { } {
_, err := run(t, args...) t.Run(strings.Join(args, " "), func(t *testing.T) {
cmdtest.ErrContains(t, err, "already exists") _, err := run(t, args...)
cmdtest.ErrContains(t, err, "already exists")
})
} }
} }
+1 -1
View File
@@ -29,7 +29,7 @@ func commandGet(opts *cli.Options) *cobra.Command {
conf, err := opts.RunSetup(cmd, true, true) conf, err := opts.RunSetup(cmd, true, true)
if err != nil { if err != nil {
return fmt.Errorf("no configuration loaded (%w)", err) return fmt.Errorf("load configuration: %w", err)
} }
value, err := conf.ValueOf(args[0]) value, err := conf.ValueOf(args[0])
+10 -8
View File
@@ -39,13 +39,13 @@ func commandInit() *cobra.Command {
cmd.SilenceUsage = true cmd.SilenceUsage = true
// With no path, the default locations are used. // With no path, the default locations are used.
cfgFile := "" configFile := ""
if len(args) > 0 { if len(args) > 0 {
cfgFile = strings.TrimSpace(args[0]) configFile = strings.TrimSpace(args[0])
} }
if cfgFile != "" { if configFile != "" {
// Use the config file given. // Use the config file given.
viper.SetConfigFile(cfgFile) viper.SetConfigFile(configFile)
} else { } else {
// Search config in /etc or local directory. // Search config in /etc or local directory.
viper.AddConfigPath(config.EtcConfigPath) viper.AddConfigPath(config.EtcConfigPath)
@@ -58,8 +58,8 @@ func commandInit() *cobra.Command {
return errors.New("configuration file already exists") return errors.New("configuration file already exists")
} }
_, ok := err.(viper.ConfigFileNotFoundError) var notFound viper.ConfigFileNotFoundError
if !errors.Is(err, os.ErrNotExist) && !ok { if !errors.Is(err, os.ErrNotExist) && !errors.As(err, &notFound) {
// Config file was found but another error was produced // Config file was found but another error was produced
return err return err
} }
@@ -80,13 +80,15 @@ func commandInit() *cobra.Command {
if err := viper.WriteConfig(); err != nil { if err := viper.WriteConfig(); err != nil {
return err return err
} }
return nil return nil
}, },
} }
// Config flag should be hidden as the config path should be provided // Config flag should be hidden as the config path should be provided
cmd.SetHelpFunc(func(command *cobra.Command, strings []string) { cmd.SetHelpFunc(func(command *cobra.Command, args []string) {
command.Flags().MarkHidden("config") //nolint:errcheck,gosec // a root flag, so it always exists command.Flags().MarkHidden("config") //nolint:errcheck,gosec // a root flag, so it always exists
command.Parent().HelpFunc()(command, strings) command.Parent().HelpFunc()(command, args)
}) })
return cmd return cmd
} }
+2 -1
View File
@@ -43,7 +43,7 @@ func commandSet(opts *cli.Options) *cobra.Command {
conf, err := opts.RunSetup(cmd, true, true) conf, err := opts.RunSetup(cmd, true, true)
if err != nil { if err != nil {
return fmt.Errorf("no configuration loaded (%w)", err) return fmt.Errorf("load configuration: %w", err)
} }
if viper.ConfigFileUsed() == "" { if viper.ConfigFileUsed() == "" {
@@ -65,5 +65,6 @@ func commandSet(opts *cli.Options) *cobra.Command {
}, },
} }
cmd.Flags().BoolP("unset", "x", false, "clear configuration setting") cmd.Flags().BoolP("unset", "x", false, "clear configuration setting")
return cmd return cmd
} }
+4 -3
View File
@@ -24,13 +24,13 @@ func commandShow(opts *cli.Options) *cobra.Command {
conf, err := opts.RunSetup(cmd, true, true) conf, err := opts.RunSetup(cmd, true, true)
if err != nil { if err != nil {
return fmt.Errorf("no configuration loaded (%w)", err) return fmt.Errorf("load configuration: %w", err)
} }
if yaml, _ := cmd.Flags().GetBool("yaml"); yaml { if yaml, _ := cmd.Flags().GetBool("yaml"); yaml {
conf.PrintYaml() conf.PrintYAML()
} else if json, _ := cmd.Flags().GetBool("json"); json { } else if json, _ := cmd.Flags().GetBool("json"); json {
conf.PrintJson() conf.PrintJSON()
} else { } else {
conf.PrintTerminal() conf.PrintTerminal()
} }
@@ -40,5 +40,6 @@ func commandShow(opts *cli.Options) *cobra.Command {
} }
cmd.Flags().Bool("json", false, "format configuration as json") cmd.Flags().Bool("json", false, "format configuration as json")
cmd.Flags().Bool("yaml", false, "format configuration as yaml") cmd.Flags().Bool("yaml", false, "format configuration as yaml")
return cmd return cmd
} }
+9 -5
View File
@@ -3,7 +3,7 @@ package cli
import ( import (
"bufio" "bufio"
"fmt" "fmt"
"log/slog" "io"
"os" "os"
"strings" "strings"
) )
@@ -11,15 +11,19 @@ import (
// Confirm asks the user the yes/no question prompt on stdin and reports whether // Confirm asks the user the yes/no question prompt on stdin and reports whether
// they answered yes. "y" and "yes" confirm, "n" and "no" refuse, in any case; // they answered yes. "y" and "yes" confirm, "n" and "no" refuse, in any case;
// anything else asks again. It exits the program if stdin can't be read. // anything else asks again. It exits the program if stdin can't be read.
func Confirm(prompt string) bool { func Confirm(out io.Writer, in io.Reader, prompt string) bool {
reader := bufio.NewReader(os.Stdin) reader := bufio.NewReader(in)
for { for {
fmt.Printf("%s [y/n]: ", prompt) _, err := fmt.Fprintf(out, "%s [y/n]: ", prompt)
if err != nil {
fmt.Fprintf(os.Stderr, "error asking question: %s\n", err)
os.Exit(1)
}
response, err := reader.ReadString('\n') response, err := reader.ReadString('\n')
if err != nil { if err != nil {
slog.Error("reading the answer", "error", err) fmt.Fprintf(os.Stderr, "error reading answer: %s\n", err)
os.Exit(1) os.Exit(1)
} }
+3 -1
View File
@@ -8,7 +8,8 @@ import (
"git.avsdev.uk/Go/job-search/internal/cli" "git.avsdev.uk/Go/job-search/internal/cli"
) )
// NewCommand returns the "fetch" command, which fetches opportunities once or as a service. // NewCommand returns the "fetch" command, which fetches opportunities once or
// as a service.
func NewCommand(opts *cli.Options) *cobra.Command { func NewCommand(opts *cli.Options) *cobra.Command {
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "fetch", Use: "fetch",
@@ -22,5 +23,6 @@ func NewCommand(opts *cli.Options) *cobra.Command {
} }
cmd.AddCommand(commandOnce(opts)) cmd.AddCommand(commandOnce(opts))
cmd.AddCommand(commandService(opts)) cmd.AddCommand(commandService(opts))
return cmd return cmd
} }
+17 -8
View File
@@ -29,7 +29,9 @@ var stageTags = map[string][]string{
// filter keeps the opportunities whose latest release is at stage. Releases // filter keeps the opportunities whose latest release is at stage. Releases
// are oldest first, so the last one is the procedure's current stage. // are oldest first, so the last one is the procedure's current stage.
func filter(opportunities []*types.Opportunity, stage string) []*types.Opportunity { func filter(
opportunities []*types.Opportunity, stage string,
) []*types.Opportunity {
var matching []*types.Opportunity var matching []*types.Opportunity
for _, opportunity := range opportunities { for _, opportunity := range opportunities {
@@ -63,7 +65,7 @@ func commandOnce(opts *cli.Options) *cobra.Command {
conf, err := opts.RunSetup(cmd, true, true) conf, err := opts.RunSetup(cmd, true, true)
if err != nil { if err != nil {
return fmt.Errorf("no configuration loaded (%w)", err) return fmt.Errorf("load configuration: %w", err)
} }
since, sinceErr := cmd.Flags().GetTime("since") since, sinceErr := cmd.Flags().GetTime("since")
@@ -96,15 +98,15 @@ func commandOnce(opts *cli.Options) *cobra.Command {
filterTag = "award" filterTag = "award"
} }
if len(filterTag) > 0 { if filterTag != "" {
results = filter(results, filterTag) results = filter(results, filterTag)
} }
jbytes, err := json.MarshalIndent(results, "", " ") encoded, err := json.MarshalIndent(results, "", " ")
if err != nil { if err != nil {
return err return err
} }
fmt.Println(string(jbytes)) fmt.Println(string(encoded))
return nil return nil
}, },
@@ -118,17 +120,24 @@ func commandOnce(opts *cli.Options) *cobra.Command {
"contract updates published since YYYY-MM-DD[ hh:mm:ss]", "contract updates published since YYYY-MM-DD[ hh:mm:ss]",
) )
cmd.Flags().Bool( cmd.Flags().Bool(
"planning", false, "only return planning (i.e. exclude opportunities and awards)", "planning",
false,
"only return planning (i.e. exclude opportunities and awards)",
) )
cmd.Flags().Bool( cmd.Flags().Bool(
"opportunities", false, "only return opportunities (i.e. exclude awards and planning)", "opportunities",
false,
"only return opportunities (i.e. exclude awards and planning)",
) )
cmd.Flags().Bool( cmd.Flags().Bool(
"awards", false, "only return awards (i.e. exclude planning and opportunities)", "awards",
false,
"only return awards (i.e. exclude planning and opportunities)",
) )
cmd.Flags().Bool( cmd.Flags().Bool(
"ignore-rate-limit", false, "ignore the rate limit and fail if exceeded", "ignore-rate-limit", false, "ignore the rate limit and fail if exceeded",
) )
cmd.MarkFlagsMutuallyExclusive("planning", "opportunities", "awards") cmd.MarkFlagsMutuallyExclusive("planning", "opportunities", "awards")
return cmd return cmd
} }
+4 -1
View File
@@ -17,7 +17,9 @@ func opp(uuid string, tags ...string) *types.Opportunity {
} }
for _, tag := range tags { for _, tag := range tags {
o.Releases = append(o.Releases, types.OpportunityRelease{ o.Releases = append(o.Releases, types.OpportunityRelease{
Tags: []types.OpportunityReleaseTag{{Tag: tag}}, Tags: []types.OpportunityReleaseTag{{
Tag: tag,
}},
}) })
} }
return o return o
@@ -56,6 +58,7 @@ func TestOnceFiltersExclusive(t *testing.T) {
cmd.SetArgs([]string{"--planning", "--awards"}) cmd.SetArgs([]string{"--planning", "--awards"})
cmd.SetOut(io.Discard) cmd.SetOut(io.Discard)
cmd.SetErr(io.Discard) cmd.SetErr(io.Discard)
err := cmd.Execute() err := cmd.Execute()
if err == nil || !strings.Contains(err.Error(), "none of the others can be") { if err == nil || !strings.Contains(err.Error(), "none of the others can be") {
t.Fatalf("err = %v; want a mutually exclusive flags error", err) t.Fatalf("err = %v; want a mutually exclusive flags error", err)
+2 -1
View File
@@ -32,7 +32,7 @@ func commandService(opts *cli.Options) *cobra.Command {
conf, err := opts.RunSetup(cmd, true, foreground) conf, err := opts.RunSetup(cmd, true, foreground)
if err != nil { if err != nil {
return fmt.Errorf("no configuration loaded (%w)", err) return fmt.Errorf("load configuration: %w", err)
} }
since, sinceErr := cmd.Flags().GetTime("since") since, sinceErr := cmd.Flags().GetTime("since")
@@ -65,5 +65,6 @@ func commandService(opts *cli.Options) *cobra.Command {
"contract updates published since YYYY-MM-DD[ hh:mm:ss]", "contract updates published since YYYY-MM-DD[ hh:mm:ss]",
) )
cmd.Flags().Bool("foreground", false, "run in the foreground") cmd.Flags().Bool("foreground", false, "run in the foreground")
return cmd return cmd
} }
+5 -2
View File
@@ -3,6 +3,7 @@ package cli_test
import ( import (
"fmt" "fmt"
"os" "os"
"strings"
"testing" "testing"
"github.com/spf13/cobra" "github.com/spf13/cobra"
@@ -49,8 +50,10 @@ func TestLogLevelFlags(t *testing.T) {
{[]string{"-q"}, "level: error"}, {[]string{"-q"}, "level: error"},
{[]string{"-q", "-d"}, "level: debug"}, {[]string{"-q", "-d"}, "level: debug"},
} { } {
out := cmdtest.MustRun(t, newRoot, append([]string{"level"}, tt.args...)...) args := append([]string{"level"}, tt.args...)
cmdtest.Contains(t, out, tt.want) t.Run(strings.Join(args, " "), func(t *testing.T) {
cmdtest.Contains(t, cmdtest.MustRun(t, newRoot, args...), tt.want)
})
} }
} }
+11 -9
View File
@@ -24,15 +24,15 @@ import (
// If the database file doesn't exist, create decides between asking whether // If the database file doesn't exist, create decides between asking whether
// to create it (up) and failing (down, status: nothing to work on). // to create it (up) and failing (down, status: nothing to work on).
func openMigrator( func openMigrator(
cmd *cobra.Command, opts *cli.Options, create bool, createConfirmed bool, cmd *cobra.Command, opts *cli.Options, create, createConfirmed bool,
) (*goose.Provider, *sql.DB, error) { ) (*goose.Provider, *sql.DB, error) {
conf, err := opts.RunSetup(cmd, true, true) conf, err := opts.RunSetup(cmd, true, true)
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("no configuration loaded (%w)", err) return nil, nil, fmt.Errorf("load configuration: %w", err)
} }
file := conf.Database.File file := conf.Database.File
if len(file) == 0 { if file == "" {
return nil, nil, errors.New( return nil, nil, errors.New(
"no database file specified. Update your configuration", "no database file specified. Update your configuration",
) )
@@ -46,8 +46,8 @@ func openMigrator(
question := fmt.Sprintf( question := fmt.Sprintf(
"Database file '%s' does not exist. Would you like to create it?", file, "Database file '%s' does not exist. Would you like to create it?", file,
) )
if !cli.Confirm(question) { if !cli.Confirm(cmd.OutOrStdout(), cmd.InOrStdin(), question) {
return nil, nil, errors.New("migrate: no database file") return nil, nil, errors.New("no database file")
} }
} }
} else if err != nil { } else if err != nil {
@@ -75,14 +75,14 @@ func closeDB(db *sql.DB, err *error) {
// errWriter writes until a write fails, then keeps the error, so a run of // errWriter writes until a write fails, then keeps the error, so a run of
// prints needs checking only once. // prints needs checking only once.
type errWriter struct { type errWriter struct {
w io.Writer out io.Writer
err error err error
} }
// printf writes like fmt.Fprintf, unless an earlier write failed. // printf writes like fmt.Fprintf, unless an earlier write failed.
func (e *errWriter) printf(format string, args ...any) { func (e *errWriter) printf(format string, args ...any) {
if e.err == nil { if e.err == nil {
_, e.err = fmt.Fprintf(e.w, format, args...) _, e.err = fmt.Fprintf(e.out, format, args...)
} }
} }
@@ -91,7 +91,7 @@ func (e *errWriter) printf(format string, args ...any) {
// applied 00003_create_release_tags_table.sql (2ms) // applied 00003_create_release_tags_table.sql (2ms)
func printResults(out io.Writer, results []*goose.MigrationResult) error { func printResults(out io.Writer, results []*goose.MigrationResult) error {
output := &errWriter{ output := &errWriter{
w: out, out: out,
} }
for _, result := range results { for _, result := range results {
verb := "applied" verb := "applied"
@@ -136,7 +136,9 @@ func failure(out io.Writer, err error) error {
} }
// printVersion reports where the database ended up. // printVersion reports where the database ended up.
func printVersion(ctx context.Context, out io.Writer, migrator *goose.Provider) error { func printVersion(
ctx context.Context, out io.Writer, migrator *goose.Provider,
) error {
dbVersion, err := migrator.GetDBVersion(ctx) dbVersion, err := migrator.GetDBVersion(ctx)
if err != nil { if err != nil {
return err return err
+14 -4
View File
@@ -39,7 +39,11 @@ func TestFailurePartial(t *testing.T) {
if !errors.Is(err, cause) { if !errors.Is(err, cause) {
t.Errorf("err = %v, want it to wrap the migration error", err) t.Errorf("err = %v, want it to wrap the migration error", err)
} }
for _, want := range []string{"00002_second.sql", "rolled back", "last version that succeeded"} { for _, want := range []string{
"00002_second.sql",
"rolled back",
"last version that succeeded",
} {
if err == nil || !strings.Contains(err.Error(), want) { if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("err = %v, want it to mention %q", err, want) t.Errorf("err = %v, want it to mention %q", err, want)
} }
@@ -58,7 +62,10 @@ func TestFailureWithoutFailedSource(t *testing.T) {
func TestFailureOtherError(t *testing.T) { func TestFailureOtherError(t *testing.T) {
cause := errors.New("database is locked") cause := errors.New("database is locked")
var out bytes.Buffer var out bytes.Buffer
if err := failure(&out, cause); err != cause { if err := failure(
&out,
cause,
); err != cause { //nolint:errorlint // checks the error is returned unchanged
t.Errorf("err = %v, want the error unchanged", err) t.Errorf("err = %v, want the error unchanged", err)
} }
if out.Len() != 0 { if out.Len() != 0 {
@@ -75,7 +82,8 @@ func TestPrintResults(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
want := " applied 00001_first.sql (2ms)\n reverted 00002_second.sql (2ms)\n" want := " applied 00001_first.sql (2ms)\n" +
" reverted 00002_second.sql (2ms)\n"
if out.String() != want { if out.String() != want {
t.Errorf("output = %q, want %q", out.String(), want) t.Errorf("output = %q, want %q", out.String(), want)
} }
@@ -97,7 +105,9 @@ func TestWriteErrors(t *testing.T) {
cause := errors.New("boom") cause := errors.New("boom")
err := failure(failingWriter{}, &goose.PartialError{ err := failure(failingWriter{}, &goose.PartialError{
Applied: results, Failed: result("00002_second.sql", "up"), Err: cause, Applied: results,
Failed: result("00002_second.sql", "up"),
Err: cause,
}) })
if !errors.Is(err, cause) || !errors.Is(err, errWrite) { if !errors.Is(err, cause) || !errors.Is(err, errWrite) {
t.Errorf("failure: err = %v, want both the migration and write errors", err) t.Errorf("failure: err = %v, want both the migration and write errors", err)
+4 -8
View File
@@ -39,12 +39,11 @@ confirmed by providing -yes.`,
defer closeDB(db, &err) defer closeDB(db, &err)
results, err := store.StepDown(cmd.Context(), migrator, int(steps), yes) results, err := store.StepDown(cmd.Context(), migrator, int(steps), yes)
if len(results) == 0 && err == nil { if len(results) == 0 && err == nil {
fmt.Println("No changes made to database (empty)") fmt.Println("No changes made to database (empty)")
return nil return nil
} }
if err == store.ErrWouldEmptyDatabase { if errors.Is(err, store.ErrWouldEmptyDatabase) {
return fmt.Errorf( return fmt.Errorf(
"%w\n\nTo confirm you wish to proceed, pass the --yes flag", err, "%w\n\nTo confirm you wish to proceed, pass the --yes flag", err,
) )
@@ -62,17 +61,14 @@ confirmed by providing -yes.`,
}, },
} }
cmd.Flags().Uint( cmd.Flags().Uint(
"steps", "steps", 0, "number of migrations to roll back. 0 = all (deletes all data)",
0,
"number of migrations to roll back. 0 = all (deletes all data)",
) )
if err := cmd.MarkFlagRequired("steps"); err != nil { if err := cmd.MarkFlagRequired("steps"); err != nil {
panic(err) // only fails if the flag isn't defined panic(err) // only fails if the flag isn't defined
} }
cmd.Flags().BoolP( cmd.Flags().BoolP(
"yes", "y", "yes", "y", false, "confirm you are intending to roll back every migration",
false,
"confirm you are intending to roll back every migration",
) )
return cmd return cmd
} }
+3 -2
View File
@@ -14,8 +14,8 @@ import (
"git.avsdev.uk/Go/job-search/internal/cli" "git.avsdev.uk/Go/job-search/internal/cli"
) )
// NewCommand returns the "migrate" command, which migrates the database schema (see // NewCommand returns the "migrate" command, which migrates the database schema
// internal/store). // (see internal/store).
func NewCommand(opts *cli.Options) *cobra.Command { func NewCommand(opts *cli.Options) *cobra.Command {
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "migrate", Use: "migrate",
@@ -39,5 +39,6 @@ migration is applied.`,
cmd.AddCommand(commandUp(opts)) cmd.AddCommand(commandUp(opts))
cmd.AddCommand(commandDown(opts)) cmd.AddCommand(commandDown(opts))
cmd.AddCommand(commandStatus(opts)) cmd.AddCommand(commandStatus(opts))
return cmd return cmd
} }
+22 -6
View File
@@ -3,6 +3,7 @@ package migrate
import ( import (
"fmt" "fmt"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"github.com/spf13/cobra" "github.com/spf13/cobra"
@@ -43,10 +44,12 @@ func latestVersion(t *testing.T) int64 {
t.Fatal(err) t.Fatal(err)
} }
defer db.Close() defer db.Close()
m, err := store.NewMigrator(db) m, err := store.NewMigrator(db)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
return store.LatestVersion(m) return store.LatestVersion(m)
} }
@@ -83,7 +86,9 @@ func TestMigrateWithoutSubcommand(t *testing.T) {
cmdtest.WriteFile(t, "test.db", "") // exists, so no question is asked cmdtest.WriteFile(t, "test.db", "") // exists, so no question is asked
out := mustRun(t, "migrate") out := mustRun(t, "migrate")
cmdtest.Contains(t, out, fmt.Sprintf("Database migrated to version %d", latestVersion(t))) cmdtest.Contains(
t, out, fmt.Sprintf("Database migrated to version %d", latestVersion(t)),
)
} }
func TestMigrateUpAsksBeforeCreating(t *testing.T) { func TestMigrateUpAsksBeforeCreating(t *testing.T) {
@@ -130,16 +135,24 @@ func TestMigrateDown(t *testing.T) {
mustRun(t, "migrate", "up", "--create") mustRun(t, "migrate", "up", "--create")
out := mustRun(t, "migrate", "down", "--steps", "1") out := mustRun(t, "migrate", "down", "--steps", "1")
cmdtest.Contains(t, out, "reverted", fmt.Sprintf("Database migrated to version %d", latest-1)) cmdtest.Contains(
t,
out,
"reverted",
fmt.Sprintf("Database migrated to version %d", latest-1),
)
// Rolling back everything deletes all data, so it must be confirmed. // Rolling back everything deletes all data, so it must be confirmed.
for _, args := range [][]string{ for _, args := range [][]string{
{"migrate", "down", "--steps", "0"}, {"migrate", "down", "--steps", "0"},
{"migrate", "down", "--steps", "99"}, {"migrate", "down", "--steps", "99"},
} { } {
_, err := run(t, "", args...) t.Run(strings.Join(args, " "), func(t *testing.T) {
cmdtest.ErrContains(t, err, "pass the --yes flag") _, err := run(t, "", args...)
cmdtest.ErrContains(t, err, "pass the --yes flag")
})
} }
cmdtest.Contains( cmdtest.Contains(
t, t,
mustRun(t, "migrate", "status"), mustRun(t, "migrate", "status"),
@@ -166,9 +179,12 @@ func TestMigrateMissingDatabase(t *testing.T) {
{"migrate", "status"}, {"migrate", "status"},
{"migrate", "down", "--steps", "1"}, {"migrate", "down", "--steps", "1"},
} { } {
_, err := run(t, "", args...) t.Run(strings.Join(args, " "), func(t *testing.T) {
cmdtest.ErrContains(t, err, "does not exist") _, err := run(t, "", args...)
cmdtest.ErrContains(t, err, "does not exist")
})
} }
if cmdtest.Exists("test.db") { if cmdtest.Exists("test.db") {
t.Error("status or down created the database") t.Error("status or down created the database")
} }
+9 -4
View File
@@ -22,7 +22,9 @@ import (
// 1 applied 2026-10-07 10:12:01 00001_create_opportunities_table.sql // 1 applied 2026-10-07 10:12:01 00001_create_opportunities_table.sql
// ... // ...
// 5 pending 00005_create_opportunity_classifications_table.sql // 5 pending 00005_create_opportunity_classifications_table.sql
func printStatus(ctx context.Context, out io.Writer, migrator *goose.Provider) error { func printStatus(
ctx context.Context, out io.Writer, migrator *goose.Provider,
) error {
statuses, err := migrator.Status(ctx) statuses, err := migrator.Status(ctx)
if err != nil { if err != nil {
return err return err
@@ -43,7 +45,7 @@ func printStatus(ctx context.Context, out io.Writer, migrator *goose.Provider) e
} }
output := &errWriter{ output := &errWriter{
w: out, out: out,
} }
switch { switch {
case current > latest: case current > latest:
@@ -55,7 +57,9 @@ func printStatus(ctx context.Context, out io.Writer, migrator *goose.Provider) e
case pending == 0: case pending == 0:
output.printf("Database version %d of %d (up-to-date)\n", current, latest) output.printf("Database version %d of %d (up-to-date)\n", current, latest)
default: default:
output.printf("Database version %d of %d (%d pending)\n", current, latest, pending) output.printf(
"Database version %d of %d (%d pending)\n", current, latest, pending,
)
} }
if output.err != nil { if output.err != nil {
return output.err return output.err
@@ -63,7 +67,7 @@ func printStatus(ctx context.Context, out io.Writer, migrator *goose.Provider) e
table := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0) table := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
output = &errWriter{ output = &errWriter{
w: table, out: table,
} }
for _, status := range statuses { for _, status := range statuses {
applied := "" applied := ""
@@ -78,6 +82,7 @@ func printStatus(ctx context.Context, out io.Writer, migrator *goose.Provider) e
filepath.Base(status.Source.Path), filepath.Base(status.Source.Path),
) )
} }
return errors.Join(output.err, table.Flush()) return errors.Join(output.err, table.Flush())
} }
+9 -12
View File
@@ -1,6 +1,7 @@
package migrate package migrate
import ( import (
"errors"
"fmt" "fmt"
"log/slog" "log/slog"
@@ -23,14 +24,15 @@ func runStepUp(
defer closeDB(db, &err) defer closeDB(db, &err)
results, err := store.StepUp(cmd.Context(), migrator, int(steps)) results, err := store.StepUp(cmd.Context(), migrator, int(steps))
if len(results) == 0 && err == nil { if len(results) == 0 && err == nil {
_, err := fmt.Fprintln(cmd.OutOrStdout(), "No changes made to database (up-to-date)") _, err = fmt.Fprintln(
cmd.OutOrStdout(), "No changes made to database (up-to-date)",
)
return err return err
} }
if err := printResults(cmd.OutOrStdout(), results); err != nil { if printErr := printResults(cmd.OutOrStdout(), results); printErr != nil {
return err return errors.Join(err, printErr)
} }
if err != nil { if err != nil {
@@ -60,15 +62,10 @@ func commandUp(opts *cli.Options) *cobra.Command {
return runStepUp(cmd, opts, int(steps), create) return runStepUp(cmd, opts, int(steps), create)
}, },
} }
cmd.Flags().Uint( cmd.Flags().Uint("steps", 0, "number of migrations to apply. 0 = all pending")
"steps",
0,
"number of migrations to apply. 0 = all pending",
)
cmd.Flags().Bool( cmd.Flags().Bool(
"create", "create", false, "create the database file if it does not already exist",
false,
"create the database file if it does not already exist",
) )
return cmd return cmd
} }
+3 -3
View File
@@ -34,13 +34,13 @@ func (o *Options) Bind(cmd *cobra.Command) {
// terminalLog is true, switches logging to the configured log file (staying // terminalLog is true, switches logging to the configured log file (staying
// on the terminal, with a warning, if none is configured). // on the terminal, with a warning, if none is configured).
func (o *Options) RunSetup( func (o *Options) RunSetup(
cmd *cobra.Command, loadConfig bool, terminalLog bool, cmd *cobra.Command, loadConfig, terminalLog bool,
) (*config.JobSearchConfig, error) { ) (*config.JobSearchConfig, error) {
// Flags and arguments are valid from here on, so errors are runtime // Flags and arguments are valid from here on, so errors are runtime
// failures: report them without cobra's usage text. // failures: report them without cobra's usage text.
cmd.SilenceUsage = true cmd.SilenceUsage = true
// Check for debug level or quiet supression via args // Check for debug level or quiet suppression via args
conf := &config.JobSearchConfig{ conf := &config.JobSearchConfig{
Logging: config.JSCLogging{}, Logging: config.JSCLogging{},
} }
@@ -64,7 +64,7 @@ func (o *Options) RunSetup(
// Check if the logging should be switched to file // Check if the logging should be switched to file
if !terminalLog { if !terminalLog {
if len(conf.Logging.File) == 0 { if conf.Logging.File == "" {
slog.Warn("no log file configured; logging to the terminal") slog.Warn("no log file configured; logging to the terminal")
} else if err := logging.LogToFile(conf.Logging.File); err != nil { } else if err := logging.LogToFile(conf.Logging.File); err != nil {
return nil, fmt.Errorf("opening log file: %w", err) return nil, fmt.Errorf("opening log file: %w", err)
+12 -6
View File
@@ -2,6 +2,7 @@ package wsserve
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"log/slog" "log/slog"
"os" "os"
@@ -39,21 +40,21 @@ func NewCommand(opts *cli.Options) *cobra.Command {
if addrOverride != "" { if addrOverride != "" {
listenParts := strings.Split(addrOverride, ":") listenParts := strings.Split(addrOverride, ":")
if len(listenParts) != 2 { if len(listenParts) != 2 {
return fmt.Errorf( return errors.New(
"--listen should be in the format of 127.0.0.1:9393", "--listen should be in the format of 127.0.0.1:9393",
) )
} }
i, err := strconv.Atoi(listenParts[1]) port, err := strconv.Atoi(listenParts[1])
if err != nil { if err != nil {
return fmt.Errorf("failed to parse --listen port: %w", err) return fmt.Errorf("failed to parse --listen port: %w", err)
} }
hostOverride = listenParts[0] hostOverride = listenParts[0]
portOverride = i portOverride = port
} }
conf, err := opts.RunSetup(cmd, true, true) conf, err := opts.RunSetup(cmd, true, true)
if err != nil { if err != nil {
return fmt.Errorf("no configuration loaded (%w)", err) return fmt.Errorf("load configuration: %w", err)
} }
if portOverride != 0 { if portOverride != 0 {
@@ -82,11 +83,16 @@ func NewCommand(opts *cli.Options) *cobra.Command {
return server.Run(ctx) return server.Run(ctx)
}, },
} }
cmd.Flags().String("listen", "", "listen address (default from config, 127.0.0.1:9393)") cmd.Flags().String(
cmd.Flags().String("host", "", "listen address (default from config, 127.0.0.1)") "listen", "", "listen address (default from config, 127.0.0.1:9393)",
)
cmd.Flags().String(
"host", "", "listen address (default from config, 127.0.0.1)",
)
cmd.Flags().Int("port", 0, "listen address (default from config, 9393)") cmd.Flags().Int("port", 0, "listen address (default from config, 9393)")
cmd.Flags().Bool("foreground", false, "run in the foreground") cmd.Flags().Bool("foreground", false, "run in the foreground")
cmd.MarkFlagsMutuallyExclusive("listen", "host") cmd.MarkFlagsMutuallyExclusive("listen", "host")
cmd.MarkFlagsMutuallyExclusive("listen", "port") cmd.MarkFlagsMutuallyExclusive("listen", "port")
return cmd return cmd
} }
+5 -2
View File
@@ -48,6 +48,9 @@ is kept.
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory | | `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB. `-1` (no limit) is rejected, as a client could then exhaust the server's memory |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` | | `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection | | `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.*.enabled` | bool | `false` | In a rate limit rule: whether it applies |
| `server.websocket.*.per_second` | int | `50` | In a rate limit rule: sustained rate. `0` or less: the rule is off |
| `server.websocket.*.burst` | int | `100` | In a rate limit rule: largest burst. Below `1`: 1 |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none | | `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings | | `server.websocket.ping_interval` | duration | `30s` | How long a connection may be idle before it is pinged; `0s`: no pings |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none | | `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
@@ -91,7 +94,7 @@ per key:
passed to `SetValue` (e.g. `"10m"`, `"true"`). passed to `SetValue` (e.g. `"10m"`, `"true"`).
- A tagged field whose type is a struct is a nested section, addressed as - A tagged field whose type is a struct is a nested section, addressed as
`section.key`. An embedded struct has its fields promoted, as Go does; `section.key`. An embedded struct has its fields promoted, as Go does;
every source embeds `JobSearchConfigSource` for `enabled` and `interval`. every source embeds `JSCSourcesSource` for `enabled` and `interval`.
- `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a - `time.Duration` fields report `reflect.String` from `TypeOf`, accept either a
`time.Duration` or a duration string in `SetValue`, and are given out as `time.Duration` or a duration string in `SetValue`, and are given out as
strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`, strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
@@ -125,7 +128,7 @@ To add a new section:
its key, and an implementation of `Section` (copy an existing section's its key, and an implementation of `Section` (copy an existing section's
methods). methods).
2. Add the struct as a tagged field of its parent: `JobSearchConfig` in 2. Add the struct as a tagged field of its parent: `JobSearchConfig` in
`config.go`, or `JobSearchConfigSources` in `conf_sources.go` for a source, `config.go`, or `JSCSources` in `conf_sources.go` for a source,
etc. The field's `cfg` tag must match the `JSCK*` constant. etc. The field's `cfg` tag must match the `JSCK*` constant.
Nothing in `section.go` needs changing. Nothing in `section.go` needs changing.
+5
View File
@@ -17,20 +17,25 @@ type JSCDatabase struct {
// Keys returns the keys directly in the section. // Keys returns the keys directly in the section.
func (c *JSCDatabase) Keys() []string { return keysOf(c) } func (c *JSCDatabase) Keys() []string { return keysOf(c) }
// Map implements Section.
func (c JSCDatabase) Map() map[string]any { return mapOf(c) } func (c JSCDatabase) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCDatabase) Flatten() map[string]any { func (c JSCDatabase) Flatten() map[string]any {
return flattenOf(JSCKDatabase, c) return flattenOf(JSCKDatabase, c)
} }
// TypeOf implements Section.
func (c JSCDatabase) TypeOf(key string) (reflect.Kind, error) { func (c JSCDatabase) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCDatabase) ValueOf(key string) (any, error) { func (c JSCDatabase) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCDatabase) SetValue(key string, value any) error { func (c *JSCDatabase) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
+9 -4
View File
@@ -19,20 +19,25 @@ type JSCLogging struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCLogging) Map() map[string]any { return mapOf(c) } func (c JSCLogging) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCLogging) Flatten() map[string]any { func (c JSCLogging) Flatten() map[string]any {
return flattenOf(JSCKLogging, c) return flattenOf(JSCKLogging, c)
} }
// TypeOf implements Section.
func (c JSCLogging) TypeOf(key string) (reflect.Kind, error) { func (c JSCLogging) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCLogging) ValueOf(key string) (any, error) { func (c JSCLogging) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCLogging) SetValue(key string, value any) error { func (c *JSCLogging) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
@@ -50,16 +55,16 @@ func (c *JSCLogging) validate(key string, value any) error {
case "debug", "info", "warn", "error": case "debug", "info", "warn", "error":
default: default:
return errors.New( return errors.New(
"invalid log level. Must be one of: debug, info, warn, error", "config: invalid log level: must be one of debug, info, warn, error",
) )
} }
} }
return nil return nil
} }
// ConfigLevelToLogLevel converts a configured level name to an slog level. An // LevelToLogLevel converts a configured level name to an slog level. An
// unknown name gives 99, above every real level, so nothing is logged. // unknown name gives 99, above every real level, so nothing is logged.
func ConfigLevelToLogLevel(level string) slog.Level { func LevelToLogLevel(level string) slog.Level {
switch level { switch level {
case "debug": case "debug":
return slog.LevelDebug return slog.LevelDebug
@@ -77,5 +82,5 @@ func ConfigLevelToLogLevel(level string) slog.Level {
// LogLevel returns the configured level as an slog level. An unknown level // LogLevel returns the configured level as an slog level. An unknown level
// gives 99, above every real level, so nothing is logged. // gives 99, above every real level, so nothing is logged.
func (c *JSCLogging) LogLevel() slog.Level { func (c *JSCLogging) LogLevel() slog.Level {
return ConfigLevelToLogLevel(c.Level) return LevelToLogLevel(c.Level)
} }
+5
View File
@@ -14,18 +14,23 @@ type JSCRateLimitRule struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCRateLimitRule) Map() map[string]any { return mapOf(c) } func (c JSCRateLimitRule) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCRateLimitRule) Flatten() map[string]any { return flattenOf("", c) } func (c JSCRateLimitRule) Flatten() map[string]any { return flattenOf("", c) }
// TypeOf implements Section.
func (c JSCRateLimitRule) TypeOf(key string) (reflect.Kind, error) { func (c JSCRateLimitRule) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCRateLimitRule) ValueOf(key string) (any, error) { func (c JSCRateLimitRule) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCRateLimitRule) SetValue(key string, value any) error { func (c *JSCRateLimitRule) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
+89 -56
View File
@@ -25,9 +25,10 @@ func captureLogs(t *testing.T, level slog.Level) *bytes.Buffer {
t.Helper() t.Helper()
var buf bytes.Buffer var buf bytes.Buffer
def := slog.Default() def := slog.Default()
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{ handler := slog.NewTextHandler(&buf, &slog.HandlerOptions{
Level: level, Level: level,
}))) })
slog.SetDefault(slog.New(handler))
t.Cleanup(func() { t.Cleanup(func() {
slog.SetDefault(def) slog.SetDefault(def)
}) })
@@ -48,7 +49,8 @@ func TestDefaults(t *testing.T) {
if conf.Logging.Level != "warn" { if conf.Logging.Level != "warn" {
t.Errorf("Logging.Level = %q; want warn", conf.Logging.Level) t.Errorf("Logging.Level = %q; want warn", conf.Logging.Level)
} }
if conf.Sources.FindATender.APIRoot != "https://www.find-tender.service.gov.uk/api/1.0" { const apiRoot = "https://www.find-tender.service.gov.uk/api/1.0"
if conf.Sources.FindATender.APIRoot != apiRoot {
t.Errorf("FindATender.APIRoot = %q", conf.Sources.FindATender.APIRoot) t.Errorf("FindATender.APIRoot = %q", conf.Sources.FindATender.APIRoot)
} }
if conf.Sources.FTSMock.APIRoot != "" { if conf.Sources.FTSMock.APIRoot != "" {
@@ -71,25 +73,22 @@ func TestLoggingSection(t *testing.T) {
File: "/tmp/x.log", File: "/tmp/x.log",
} }
if got, want := c.Keys(), []string{"level", "file"}; !reflect.DeepEqual(got, want) { if got, want := c.Keys(), []string{
"level",
"file",
}; !reflect.DeepEqual(got, want) {
t.Errorf("Keys = %v; want %v", got, want) t.Errorf("Keys = %v; want %v", got, want)
} }
if got, want := c.Map(), map[string]any{ if got, want := c.Map(), map[string]any{
"level": "info", "level": "info",
"file": "/tmp/x.log", "file": "/tmp/x.log",
}; !reflect.DeepEqual( }; !reflect.DeepEqual(got, want) {
got,
want,
) {
t.Errorf("Map = %v; want %v", got, want) t.Errorf("Map = %v; want %v", got, want)
} }
if got, want := c.Flatten(), map[string]any{ if got, want := c.Flatten(), map[string]any{
"logging.level": "info", "logging.level": "info",
"logging.file": "/tmp/x.log", "logging.file": "/tmp/x.log",
}; !reflect.DeepEqual( }; !reflect.DeepEqual(got, want) {
got,
want,
) {
t.Errorf("Flatten = %v; want %v", got, want) t.Errorf("Flatten = %v; want %v", got, want)
} }
if k, err := c.TypeOf("file"); err != nil || k != reflect.String { if k, err := c.TypeOf("file"); err != nil || k != reflect.String {
@@ -101,7 +100,8 @@ func TestLoggingSection(t *testing.T) {
if _, err := c.TypeOf("nope"); err == nil { if _, err := c.TypeOf("nope"); err == nil {
t.Error("TypeOf(nope) succeeded") t.Error("TypeOf(nope) succeeded")
} }
if err := c.SetValue("file", "/var/log/js.log"); err != nil || c.File != "/var/log/js.log" { if err := c.SetValue("file", "/var/log/js.log"); err != nil ||
c.File != "/var/log/js.log" {
t.Errorf("SetValue(file) = %v, File = %q", err, c.File) t.Errorf("SetValue(file) = %v, File = %q", err, c.File)
} }
} }
@@ -114,10 +114,7 @@ func TestLoggingLevelValidation(t *testing.T) {
} }
} }
for _, lvl := range []string{"", "trace", "WARN", "fatal"} { for _, lvl := range []string{"", "trace", "WARN", "fatal"} {
if err := c.SetValue( if err := c.SetValue("level", lvl); err == nil ||
"level",
lvl,
); err == nil ||
!strings.Contains(err.Error(), "invalid log level") { !strings.Contains(err.Error(), "invalid log level") {
t.Errorf("SetValue(level, %q) = %v; want invalid log level", lvl, err) t.Errorf("SetValue(level, %q) = %v; want invalid log level", lvl, err)
} }
@@ -136,7 +133,7 @@ func TestLoggingLevelValidation(t *testing.T) {
} }
} }
func TestConfigLevelToLogLevel(t *testing.T) { func TestLevelToLogLevel(t *testing.T) {
tests := map[string]slog.Level{ tests := map[string]slog.Level{
"debug": slog.LevelDebug, "debug": slog.LevelDebug,
"info": slog.LevelInfo, "info": slog.LevelInfo,
@@ -146,8 +143,8 @@ func TestConfigLevelToLogLevel(t *testing.T) {
"bogus": 99, "bogus": 99,
} }
for in, want := range tests { for in, want := range tests {
if got := ConfigLevelToLogLevel(in); got != want { if got := LevelToLogLevel(in); got != want {
t.Errorf("ConfigLevelToLogLevel(%q) = %v; want %v", in, got, want) t.Errorf("LevelToLogLevel(%q) = %v; want %v", in, got, want)
} }
c := JSCLogging{ c := JSCLogging{
Level: in, Level: in,
@@ -164,7 +161,10 @@ func TestSourceSection(t *testing.T) {
Interval: time.Minute, Interval: time.Minute,
} }
if got, want := c.Keys(), []string{"enabled", "interval"}; !reflect.DeepEqual(got, want) { if got, want := c.Keys(), []string{
"enabled",
"interval",
}; !reflect.DeepEqual(got, want) {
t.Errorf("Keys = %v; want %v", got, want) t.Errorf("Keys = %v; want %v", got, want)
} }
want := map[string]any{"enabled": true, "interval": "1m0s"} want := map[string]any{"enabled": true, "interval": "1m0s"}
@@ -184,7 +184,8 @@ func TestSourceSection(t *testing.T) {
if v, err := c.ValueOf("interval"); err != nil || v != "1m0s" { if v, err := c.ValueOf("interval"); err != nil || v != "1m0s" {
t.Errorf("ValueOf(interval) = %v, %v", v, err) t.Errorf("ValueOf(interval) = %v, %v", v, err)
} }
if err := c.SetValue("interval", "5s"); err != nil || c.Interval != 5*time.Second { if err := c.SetValue("interval", "5s"); err != nil ||
c.Interval != 5*time.Second {
t.Errorf("SetValue(interval, 5s) = %v, Interval = %v", err, c.Interval) t.Errorf("SetValue(interval, 5s) = %v, Interval = %v", err, c.Interval)
} }
if err := c.SetValue("interval", "never"); err == nil { if err := c.SetValue("interval", "never"); err == nil {
@@ -229,7 +230,8 @@ func TestConcreteSourceSections(t *testing.T) {
if err := s.SetValue("api_root", "http://localhost:8080"); err != nil { if err := s.SetValue("api_root", "http://localhost:8080"); err != nil {
t.Fatalf("SetValue(api_root): %v", err) t.Fatalf("SetValue(api_root): %v", err)
} }
if v, err := s.ValueOf("api_root"); err != nil || v != "http://localhost:8080" { if v, err := s.ValueOf("api_root"); err != nil ||
v != "http://localhost:8080" {
t.Errorf("ValueOf(api_root) = %v, %v", v, err) t.Errorf("ValueOf(api_root) = %v, %v", v, err)
} }
wantMap["api_root"] = "http://localhost:8080" wantMap["api_root"] = "http://localhost:8080"
@@ -241,7 +243,8 @@ func TestConcreteSourceSections(t *testing.T) {
if err := s.SetValue("block_shutdown", "true"); err != nil { if err := s.SetValue("block_shutdown", "true"); err != nil {
t.Fatalf("SetValue(block_shutdown): %v", err) t.Fatalf("SetValue(block_shutdown): %v", err)
} }
if k, err := s.TypeOf("block_shutdown"); err != nil || k != reflect.Bool { if k, err := s.TypeOf("block_shutdown"); err != nil ||
k != reflect.Bool {
t.Errorf("TypeOf(block_shutdown) = %v, %v", k, err) t.Errorf("TypeOf(block_shutdown) = %v, %v", k, err)
} }
wantMap["block_shutdown"] = true wantMap["block_shutdown"] = true
@@ -271,7 +274,8 @@ func TestSourcesSection(t *testing.T) {
if c.FTSMock.APIRoot != "http://mock" { if c.FTSMock.APIRoot != "http://mock" {
t.Errorf("FTSMock.APIRoot = %q", c.FTSMock.APIRoot) t.Errorf("FTSMock.APIRoot = %q", c.FTSMock.APIRoot)
} }
if v, err := c.ValueOf("fts_mock.api_root"); err != nil || v != "http://mock" { if v, err := c.ValueOf("fts_mock.api_root"); err != nil ||
v != "http://mock" {
t.Errorf("ValueOf = %v, %v", v, err) t.Errorf("ValueOf = %v, %v", v, err)
} }
if k, err := c.TypeOf("sleeper.enabled"); err != nil || k != reflect.Bool { if k, err := c.TypeOf("sleeper.enabled"); err != nil || k != reflect.Bool {
@@ -299,6 +303,7 @@ func TestSourcesSection(t *testing.T) {
func TestJobSearchConfigMap(t *testing.T) { func TestJobSearchConfigMap(t *testing.T) {
conf := defaultConfig(t) conf := defaultConfig(t)
m := conf.Map() m := conf.Map()
logging, ok := m["logging"].(map[string]any) logging, ok := m["logging"].(map[string]any)
if !ok || logging["level"] != "warn" { if !ok || logging["level"] != "warn" {
t.Errorf("Map()[logging] = %#v", m["logging"]) t.Errorf("Map()[logging] = %#v", m["logging"])
@@ -330,31 +335,33 @@ func TestString(t *testing.T) {
} }
} }
func TestJSON(t *testing.T) { // JSON and YAML give the configuration as nested sections.
func TestSerialised(t *testing.T) {
conf := defaultConfig(t) conf := defaultConfig(t)
var m map[string]map[string]any tests := []struct {
if err := json.Unmarshal([]byte(conf.JSON()), &m); err != nil { name string
t.Fatalf("JSON() is not valid JSON: %v\n%s", err, conf.JSON()) output string
unmarshal func([]byte, any) error
}{
{"JSON", conf.JSON(), json.Unmarshal},
{"YAML", conf.YAML(), yaml.Unmarshal},
} }
if m["logging"]["level"] != "warn" { for _, tt := range tests {
t.Errorf("JSON logging.level = %#v", m["logging"]["level"]) t.Run(tt.name, func(t *testing.T) {
} var m map[string]map[string]any
if _, ok := m["sources"]["fts_mock"]; !ok { if err := tt.unmarshal([]byte(tt.output), &m); err != nil {
t.Errorf("JSON missing sources.fts_mock: %v", m["sources"]) t.Fatalf("%s() doesn't parse: %v\n%s", tt.name, err, tt.output)
} }
}
func TestYAML(t *testing.T) { if m["logging"]["level"] != "warn" {
conf := defaultConfig(t) t.Errorf("logging.level = %#v", m["logging"]["level"])
var m map[string]map[string]any }
if err := yaml.Unmarshal([]byte(conf.YAML()), &m); err != nil { for _, source := range []string{"fts_mock", "sleeper"} {
t.Fatalf("YAML() is not valid YAML: %v\n%s", err, conf.YAML()) if _, ok := m["sources"][source]; !ok {
} t.Errorf("missing sources.%s: %v", source, m["sources"])
if m["logging"]["level"] != "warn" { }
t.Errorf("YAML logging.level = %#v", m["logging"]["level"]) }
} })
if _, ok := m["sources"]["sleeper"]; !ok {
t.Errorf("YAML missing sources.sleeper: %v", m["sources"])
} }
} }
@@ -365,6 +372,7 @@ func TestSerialisedRoundTrip(t *testing.T) {
orig.Sources.FTSMock.Enabled = true orig.Sources.FTSMock.Enabled = true
orig.Sources.FTSMock.APIRoot = "http://mock" orig.Sources.FTSMock.APIRoot = "http://mock"
orig.Server.WebSocket.MaxMessageSize = 64 << 10 orig.Server.WebSocket.MaxMessageSize = 64 << 10
for _, format := range []string{"json", "yaml"} { for _, format := range []string{"json", "yaml"} {
t.Run(format, func(t *testing.T) { t.Run(format, func(t *testing.T) {
resetViper(t) resetViper(t)
@@ -372,10 +380,12 @@ func TestSerialisedRoundTrip(t *testing.T) {
if format == "yaml" { if format == "yaml" {
out = orig.YAML() out = orig.YAML()
} }
viper.SetConfigType(format) viper.SetConfigType(format)
if err := viper.ReadConfig(strings.NewReader(out)); err != nil { if err := viper.ReadConfig(strings.NewReader(out)); err != nil {
t.Fatalf("ReadConfig: %v", err) t.Fatalf("ReadConfig: %v", err)
} }
var got JobSearchConfig var got JobSearchConfig
LoadFromViper(&got) LoadFromViper(&got)
if !reflect.DeepEqual(got, orig) { if !reflect.DeepEqual(got, orig) {
@@ -402,7 +412,9 @@ func TestLoadDefaultConfig(t *testing.T) {
if got := viper.GetString("logging.level"); got != "warn" { if got := viper.GetString("logging.level"); got != "warn" {
t.Errorf("logging.level = %q; want warn", got) t.Errorf("logging.level = %q; want warn", got)
} }
if got := viper.GetDuration("sources.sleeper.interval"); got != 10*time.Minute { if got := viper.GetDuration(
"sources.sleeper.interval",
); got != 10*time.Minute {
t.Errorf("sources.sleeper.interval = %v; want 10m", got) t.Errorf("sources.sleeper.interval = %v; want 10m", got)
} }
} }
@@ -460,7 +472,9 @@ func TestLoadFromViperInvalidValues(t *testing.T) {
t.Errorf("Logging.Level = %q; want warn kept", conf.Logging.Level) t.Errorf("Logging.Level = %q; want warn kept", conf.Logging.Level)
} }
if conf.Sources.Sleeper.Interval != 10*time.Minute { if conf.Sources.Sleeper.Interval != 10*time.Minute {
t.Errorf("Sleeper.Interval = %v; want 10m kept", conf.Sources.Sleeper.Interval) t.Errorf(
"Sleeper.Interval = %v; want 10m kept", conf.Sources.Sleeper.Interval,
)
} }
for _, key := range []string{"logging.level", "sources.sleeper.interval"} { for _, key := range []string{"logging.level", "sources.sleeper.interval"} {
if !strings.Contains(logs.String(), "key="+key) { if !strings.Contains(logs.String(), "key="+key) {
@@ -477,7 +491,9 @@ func TestDatabaseSection(t *testing.T) {
if got, want := c.Keys(), []string{"file"}; !reflect.DeepEqual(got, want) { if got, want := c.Keys(), []string{"file"}; !reflect.DeepEqual(got, want) {
t.Errorf("Keys = %v; want %v", got, want) t.Errorf("Keys = %v; want %v", got, want)
} }
if got, want := c.Map(), map[string]any{"file": "./jobsearch.db"}; !reflect.DeepEqual(got, want) { if got, want := c.Map(), map[string]any{
"file": "./jobsearch.db",
}; !reflect.DeepEqual(got, want) {
t.Errorf("Map = %v; want %v", got, want) t.Errorf("Map = %v; want %v", got, want)
} }
want := map[string]any{"database.file": "./jobsearch.db"} want := map[string]any{"database.file": "./jobsearch.db"}
@@ -496,7 +512,8 @@ func TestDatabaseSection(t *testing.T) {
if _, err := c.ValueOf("nope"); err == nil { if _, err := c.ValueOf("nope"); err == nil {
t.Error("ValueOf(nope) succeeded") t.Error("ValueOf(nope) succeeded")
} }
if err := c.SetValue("file", "/var/lib/js.db"); err != nil || c.File != "/var/lib/js.db" { if err := c.SetValue("file", "/var/lib/js.db"); err != nil ||
c.File != "/var/lib/js.db" {
t.Errorf("SetValue(file) = %v, File = %q", err, c.File) t.Errorf("SetValue(file) = %v, File = %q", err, c.File)
} }
if err := c.SetValue("file", 42); err == nil { if err := c.SetValue("file", 42); err == nil {
@@ -532,10 +549,18 @@ func TestViperValue(t *testing.T) {
} { } {
got, ok := viperValue(tt.key, tt.kind) got, ok := viperValue(tt.key, tt.kind)
if !ok || !reflect.DeepEqual(got, tt.want) { if !ok || !reflect.DeepEqual(got, tt.want) {
t.Errorf("viperValue(%q, %v) = %#v, %v; want %#v", tt.key, tt.kind, got, ok, tt.want) t.Errorf(
"viperValue(%q, %v) = %#v, %v; want %#v",
tt.key,
tt.kind,
got,
ok,
tt.want,
)
} }
} }
for _, kind := range []reflect.Kind{reflect.Float64, reflect.Int8, reflect.Map} { unhandled := []reflect.Kind{reflect.Float64, reflect.Int8, reflect.Map}
for _, kind := range unhandled {
if got, ok := viperValue("i", kind); ok { if got, ok := viperValue("i", kind); ok {
t.Errorf("viperValue(i, %v) = %#v, true; want unhandled", kind, got) t.Errorf("viperValue(i, %v) = %#v, true; want unhandled", kind, got)
} }
@@ -553,14 +578,20 @@ func TestWebSocketMaxMessageSizeNoLimit(t *testing.T) {
if c.MaxMessageSize != 0 { if c.MaxMessageSize != 0 {
t.Errorf("rejected size was stored: %v", c.MaxMessageSize) t.Errorf("rejected size was stored: %v", c.MaxMessageSize)
} }
for _, text := range []string{"0B", "64KiB"} { for _, text := range []string{"0B", "64KiB"} {
if err := c.SetValue("max_message_size", text); err != nil { if err := c.SetValue("max_message_size", text); err != nil {
t.Errorf("SetValue(max_message_size, %s): %v", text, err) t.Errorf("SetValue(max_message_size, %s): %v", text, err)
} }
} }
conf := defaultConfig(t) conf := defaultConfig(t)
if err := conf.SetValue("server.websocket.max_message_size", "-1"); err == nil { if err := conf.SetValue(
t.Error("JobSearchConfig.SetValue(server.websocket.max_message_size, -1) succeeded") "server.websocket.max_message_size", "-1",
); err == nil {
t.Error(
"JobSearchConfig.SetValue(server.websocket.max_message_size, -1) succeeded",
)
} }
} }
@@ -584,6 +615,8 @@ func TestSourceIntervalMustBePositive(t *testing.T) {
} }
} }
if conf.Sources.FindATender.Interval.Minutes() != 1 { if conf.Sources.FindATender.Interval.Minutes() != 1 {
t.Errorf("FindATender.Interval = %v; want 1m", conf.Sources.FindATender.Interval) t.Errorf(
"FindATender.Interval = %v; want 1m", conf.Sources.FindATender.Interval,
)
} }
} }
+5
View File
@@ -20,20 +20,25 @@ type JSCServer struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCServer) Map() map[string]any { return mapOf(c) } func (c JSCServer) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCServer) Flatten() map[string]any { func (c JSCServer) Flatten() map[string]any {
return flattenOf(JSCKServer, c) return flattenOf(JSCKServer, c)
} }
// TypeOf implements Section.
func (c JSCServer) TypeOf(key string) (reflect.Kind, error) { func (c JSCServer) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCServer) ValueOf(key string) (any, error) { func (c JSCServer) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCServer) SetValue(key string, value any) error { func (c *JSCServer) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
@@ -17,20 +17,25 @@ type JSCServerRateLimits struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCServerRateLimits) Map() map[string]any { return mapOf(c) } func (c JSCServerRateLimits) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCServerRateLimits) Flatten() map[string]any { func (c JSCServerRateLimits) Flatten() map[string]any {
return flattenOf(JSCKServerRateLimits, c) return flattenOf(JSCKServerRateLimits, c)
} }
// TypeOf implements Section.
func (c JSCServerRateLimits) TypeOf(key string) (reflect.Kind, error) { func (c JSCServerRateLimits) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCServerRateLimits) ValueOf(key string) (any, error) { func (c JSCServerRateLimits) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCServerRateLimits) SetValue(key string, value any) error { func (c *JSCServerRateLimits) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
+5
View File
@@ -17,20 +17,25 @@ type JSCServerTimeouts struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCServerTimeouts) Map() map[string]any { return mapOf(c) } func (c JSCServerTimeouts) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCServerTimeouts) Flatten() map[string]any { func (c JSCServerTimeouts) Flatten() map[string]any {
return flattenOf(JSCKServerTimeouts, c) return flattenOf(JSCKServerTimeouts, c)
} }
// TypeOf implements Section.
func (c JSCServerTimeouts) TypeOf(key string) (reflect.Kind, error) { func (c JSCServerTimeouts) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCServerTimeouts) ValueOf(key string) (any, error) { func (c JSCServerTimeouts) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCServerTimeouts) SetValue(key string, value any) error { func (c *JSCServerTimeouts) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
+8 -2
View File
@@ -28,20 +28,25 @@ type JSCServerWebSocket struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCServerWebSocket) Map() map[string]any { return mapOf(c) } func (c JSCServerWebSocket) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCServerWebSocket) Flatten() map[string]any { func (c JSCServerWebSocket) Flatten() map[string]any {
return flattenOf(JSCKServerWebSocket, c) return flattenOf(JSCKServerWebSocket, c)
} }
// TypeOf implements Section.
func (c JSCServerWebSocket) TypeOf(key string) (reflect.Kind, error) { func (c JSCServerWebSocket) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCServerWebSocket) ValueOf(key string) (any, error) { func (c JSCServerWebSocket) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCServerWebSocket) SetValue(key string, value any) error { func (c *JSCServerWebSocket) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
@@ -52,9 +57,10 @@ func (c *JSCServerWebSocket) SetValue(key string, value any) error {
// max_message_size can't be -1 (no limit): a client could then send a // max_message_size can't be -1 (no limit): a client could then send a
// message of any size, and the server holds each message in memory. // message of any size, and the server holds each message in memory.
func (c *JSCServerWebSocket) validate(key string, value any) error { func (c *JSCServerWebSocket) validate(key string, value any) error {
if key == "max_message_size" && value.(util.ByteSize) == util.ByteSizeNoLimit { if key == "max_message_size" &&
value.(util.ByteSize) == util.ByteSizeNoLimit {
return errors.New( return errors.New(
"max_message_size can't be -1 (no limit): set an explicit size, or 0 for the default", "config: max_message_size can't be -1 (no limit): set an explicit size, or 0 for the default",
) )
} }
return nil return nil
@@ -14,20 +14,25 @@ type JSCSourceFindATender struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSourceFindATender) Map() map[string]any { return mapOf(c) } func (c JSCSourceFindATender) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSourceFindATender) Flatten() map[string]any { func (c JSCSourceFindATender) Flatten() map[string]any {
return flattenOf(JSCKSourceFindATender, c) return flattenOf(JSCKSourceFindATender, c)
} }
// TypeOf implements Section.
func (c JSCSourceFindATender) TypeOf(key string) (reflect.Kind, error) { func (c JSCSourceFindATender) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCSourceFindATender) ValueOf(key string) (any, error) { func (c JSCSourceFindATender) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCSourceFindATender) SetValue(key string, value any) error { func (c *JSCSourceFindATender) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
+5
View File
@@ -15,20 +15,25 @@ type JSCSourceFTSMock struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSourceFTSMock) Map() map[string]any { return mapOf(c) } func (c JSCSourceFTSMock) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSourceFTSMock) Flatten() map[string]any { func (c JSCSourceFTSMock) Flatten() map[string]any {
return flattenOf(JSCKSourceFTSMock, c) return flattenOf(JSCKSourceFTSMock, c)
} }
// TypeOf implements Section.
func (c JSCSourceFTSMock) TypeOf(key string) (reflect.Kind, error) { func (c JSCSourceFTSMock) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCSourceFTSMock) ValueOf(key string) (any, error) { func (c JSCSourceFTSMock) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCSourceFTSMock) SetValue(key string, value any) error { func (c *JSCSourceFTSMock) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
+5
View File
@@ -15,20 +15,25 @@ type JSCSourceSleeper struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSourceSleeper) Map() map[string]any { return mapOf(c) } func (c JSCSourceSleeper) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSourceSleeper) Flatten() map[string]any { func (c JSCSourceSleeper) Flatten() map[string]any {
return flattenOf(JSCKSourceSleeper, c) return flattenOf(JSCKSourceSleeper, c)
} }
// TypeOf implements Section.
func (c JSCSourceSleeper) TypeOf(key string) (reflect.Kind, error) { func (c JSCSourceSleeper) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCSourceSleeper) ValueOf(key string) (any, error) { func (c JSCSourceSleeper) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCSourceSleeper) SetValue(key string, value any) error { func (c *JSCSourceSleeper) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
+13 -1
View File
@@ -6,6 +6,8 @@ import (
"time" "time"
) )
// ========== Sources ==========
// JSCKSources is the sources section's configuration key. // JSCKSources is the sources section's configuration key.
const JSCKSources string = "sources" const JSCKSources string = "sources"
@@ -19,20 +21,25 @@ type JSCSources struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSources) Map() map[string]any { return mapOf(c) } func (c JSCSources) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSources) Flatten() map[string]any { func (c JSCSources) Flatten() map[string]any {
return flattenOf(JSCKSources, c) return flattenOf(JSCKSources, c)
} }
// TypeOf implements Section.
func (c JSCSources) TypeOf(key string) (reflect.Kind, error) { func (c JSCSources) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCSources) ValueOf(key string) (any, error) { func (c JSCSources) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCSources) SetValue(key string, value any) error { func (c *JSCSources) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
@@ -50,20 +57,25 @@ type JSCSourcesSource struct {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JSCSourcesSource) Map() map[string]any { return mapOf(c) } func (c JSCSourcesSource) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JSCSourcesSource) Flatten() map[string]any { func (c JSCSourcesSource) Flatten() map[string]any {
return flattenOf("", c) return flattenOf("", c)
} }
// TypeOf implements Section.
func (c JSCSourcesSource) TypeOf(key string) (reflect.Kind, error) { func (c JSCSourcesSource) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JSCSourcesSource) ValueOf(key string) (any, error) { func (c JSCSourcesSource) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JSCSourcesSource) SetValue(key string, value any) error { func (c *JSCSourcesSource) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
@@ -76,7 +88,7 @@ func (c *JSCSourcesSource) Keys() []string { return keysOf(c) }
func (c *JSCSourcesSource) validate(key string, value any) error { func (c *JSCSourcesSource) validate(key string, value any) error {
if key == "interval" && value.(time.Duration) <= 0 { if key == "interval" && value.(time.Duration) <= 0 {
return errors.New( return errors.New(
"interval must be greater than 0; set enabled to false to turn the source off", "config: interval must be greater than 0; set enabled to false to turn the source off",
) )
} }
return nil return nil
+17 -8
View File
@@ -42,18 +42,23 @@ func LoadDefaultConfig() {
// ---------- Section methods (see Section) ---------- // ---------- Section methods (see Section) ----------
// Map implements Section.
func (c JobSearchConfig) Map() map[string]any { return mapOf(c) } func (c JobSearchConfig) Map() map[string]any { return mapOf(c) }
// Flatten implements Section.
func (c JobSearchConfig) Flatten() map[string]any { return flattenOf("", c) } func (c JobSearchConfig) Flatten() map[string]any { return flattenOf("", c) }
// TypeOf implements Section.
func (c JobSearchConfig) TypeOf(key string) (reflect.Kind, error) { func (c JobSearchConfig) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key) return typeOf(c, key)
} }
// ValueOf implements Section.
func (c JobSearchConfig) ValueOf(key string) (any, error) { func (c JobSearchConfig) ValueOf(key string) (any, error) {
return valueOf(c, key) return valueOf(c, key)
} }
// SetValue implements Section.
func (c *JobSearchConfig) SetValue(key string, value any) error { func (c *JobSearchConfig) SetValue(key string, value any) error {
return setValue(c, key, value) return setValue(c, key, value)
} }
@@ -70,7 +75,11 @@ func LoadFromViper(conf *JobSearchConfig) {
} }
value, ok := viperValue(key, keyType) value, ok := viperValue(key, keyType)
if !ok { if !ok {
slog.Error("unknown config key type", "key", key, "kind", keyType.String()) slog.Error(
"unknown config key type",
"key", key,
"kind", keyType.String(),
)
continue continue
} }
if err := conf.SetValue(key, value); err != nil { if err := conf.SetValue(key, value); err != nil {
@@ -79,9 +88,9 @@ func LoadFromViper(conf *JobSearchConfig) {
} }
} }
// viperValue reads key from Viper as a value of kind. ok is false for a kind // viperValue reads key from Viper as a value of kind. Its bool is false for a
// LoadFromViper doesn't handle. // kind LoadFromViper doesn't handle.
func viperValue(key string, kind reflect.Kind) (value any, ok bool) { func viperValue(key string, kind reflect.Kind) (any, bool) {
switch kind { switch kind {
case reflect.String: case reflect.String:
return viper.GetString(key), true return viper.GetString(key), true
@@ -134,13 +143,13 @@ func (c JobSearchConfig) String() string {
// JSON formats the configuration as indented JSON. // JSON formats the configuration as indented JSON.
func (c JobSearchConfig) JSON() string { func (c JobSearchConfig) JSON() string {
confMap := c.Map() confMap := c.Map()
jbytes, _ := json.MarshalIndent(confMap, "", " ") encoded, _ := json.MarshalIndent(confMap, "", " ")
return string(jbytes) return string(encoded)
} }
// YAML formats the configuration as YAML. // YAML formats the configuration as YAML.
func (c JobSearchConfig) YAML() string { func (c JobSearchConfig) YAML() string {
confMap := c.Map() confMap := c.Map()
ybytes, _ := yaml.Marshal(confMap) encoded, _ := yaml.Marshal(confMap)
return string(ybytes) return string(encoded)
} }
+29 -13
View File
@@ -10,7 +10,10 @@ import (
// Kinds that LoadFromViper knows how to read. // Kinds that LoadFromViper knows how to read.
var viperKinds = map[reflect.Kind]bool{ var viperKinds = map[reflect.Kind]bool{
reflect.String: true, reflect.Int: true, reflect.Bool: true, reflect.Slice: true, reflect.String: true,
reflect.Int: true,
reflect.Bool: true,
reflect.Slice: true,
} }
// Every key round-trips through TypeOf/ValueOf/SetValue and has a type that // Every key round-trips through TypeOf/ValueOf/SetValue and has a type that
@@ -25,7 +28,9 @@ func TestKeysRoundTrip(t *testing.T) {
if err != nil { if err != nil {
t.Errorf("TypeOf(%q): %v", key, err) t.Errorf("TypeOf(%q): %v", key, err)
} else if !viperKinds[kind] { } else if !viperKinds[kind] {
t.Errorf("TypeOf(%q) = %v, which LoadFromViper does not handle", key, kind) t.Errorf(
"TypeOf(%q) = %v, which LoadFromViper does not handle", key, kind,
)
} }
got, err := conf.ValueOf(key) got, err := conf.ValueOf(key)
if err != nil || !reflect.DeepEqual(got, want) { if err != nil || !reflect.DeepEqual(got, want) {
@@ -53,6 +58,7 @@ func TestSections(t *testing.T) {
if !reflect.PointerTo(reflect.TypeOf(conf)).Implements(sectionType) { if !reflect.PointerTo(reflect.TypeOf(conf)).Implements(sectionType) {
t.Error("*JobSearchConfig does not implement Section") t.Error("*JobSearchConfig does not implement Section")
} }
all := conf.Flatten() all := conf.Flatten()
uses := map[reflect.Type]int{} uses := map[reflect.Type]int{}
countSections(uses, reflect.TypeOf(conf)) countSections(uses, reflect.TypeOf(conf))
@@ -64,8 +70,8 @@ func TestSections(t *testing.T) {
func countSections(uses map[reflect.Type]int, section reflect.Type) { func countSections(uses map[reflect.Type]int, section reflect.Type) {
for _, f := range fieldsOf(section) { for _, f := range fieldsOf(section) {
if f.section { if f.section {
uses[f.typ]++ uses[f.fieldType]++
countSections(uses, f.typ) countSections(uses, f.fieldType)
} }
} }
} }
@@ -78,14 +84,16 @@ func checkSections(
prefix string, prefix string,
section reflect.Value, section reflect.Value,
) { ) {
t.Helper()
for _, f := range fieldsOf(section.Type()) { for _, f := range fieldsOf(section.Type()) {
key := joinKey(prefix, f.key) key := joinKey(prefix, f.key)
if isSectionList(f.typ) { if isSectionList(f.fieldType) {
checkSectionList(t, all, key, section.FieldByIndex(f.index)) checkSectionList(t, all, key, section.FieldByIndex(f.index))
} }
if !f.section { if !f.section {
continue continue
} }
fv := section.FieldByIndex(f.index) fv := section.FieldByIndex(f.index)
name := fv.Type().Name() name := fv.Type().Name()
if !fv.Addr().Type().Implements(sectionType) { if !fv.Addr().Type().Implements(sectionType) {
@@ -93,18 +101,16 @@ func checkSections(
continue continue
} }
checkEmbedded(t, fv.Type()) checkEmbedded(t, fv.Type())
keys := fv.Addr().Interface().(Section).Flatten() keys := fv.Addr().Interface().(Section).Flatten()
keyPrefix := prefix keyPrefix := prefix
if uses[f.typ] > 1 { if uses[f.fieldType] > 1 {
keyPrefix = key keyPrefix = key
} }
for k := range keys { for k := range keys {
if _, ok := all[joinKey(keyPrefix, k)]; !ok { if _, ok := all[joinKey(keyPrefix, k)]; !ok {
t.Errorf( t.Errorf(
"%s.Flatten() key %q does not match its cfg tag %q", "%s.Flatten() key %q does not match its cfg tag %q", name, k, key,
name,
k,
key,
) )
} }
} }
@@ -126,29 +132,36 @@ func checkSections(
// checkSectionList checks the items of the list of sections at key implement // checkSectionList checks the items of the list of sections at key implement
// Section and that their methods agree with the list's value in all. // Section and that their methods agree with the list's value in all.
func checkSectionList(t *testing.T, all map[string]any, key string, list reflect.Value) { func checkSectionList(
t *testing.T, all map[string]any, key string, list reflect.Value,
) {
t.Helper() t.Helper()
name := list.Type().Elem().Name() name := list.Type().Elem().Name()
if !reflect.PointerTo(list.Type().Elem()).Implements(sectionType) { if !reflect.PointerTo(list.Type().Elem()).Implements(sectionType) {
t.Errorf("*%s (items of %s) does not implement Section", name, key) t.Errorf("*%s (items of %s) does not implement Section", name, key)
return return
} }
items, _ := all[key].([]map[string]any) items, _ := all[key].([]map[string]any)
if len(items) != list.Len() { if len(items) != list.Len() {
t.Errorf("%s has %d items; Flatten gives %d", key, list.Len(), len(items)) t.Errorf("%s has %d items; Flatten gives %d", key, list.Len(), len(items))
return return
} }
for i := range list.Len() { for i := range list.Len() {
item, itemName := list.Index(i), fmt.Sprintf("%s[%d]", key, i) item, itemName := list.Index(i), fmt.Sprintf("%s[%d]", key, i)
want := flattenMap("", items[i]) want := flattenMap("", items[i])
// Items are not prefixed, and their kinds are their fields' kinds. // Items are not prefixed, and their kinds are their fields' kinds.
if got := item.Addr().Interface().(Section).Flatten(); !reflect.DeepEqual(got, want) { got := item.Addr().Interface().(Section).Flatten()
if !reflect.DeepEqual(got, want) {
t.Errorf("%s.Flatten() =\n%v\nwant\n%v", itemName, got, want) t.Errorf("%s.Flatten() =\n%v\nwant\n%v", itemName, got, want)
} }
kinds := map[string]reflect.Kind{} kinds := map[string]reflect.Kind{}
for k := range want { for k := range want {
_, f, _ := lookup(item, k) _, f, _ := lookup(item, k)
kinds[k] = kindOf(f.typ) kinds[k] = kindOf(f.fieldType)
} }
checkSectionMethods(t, itemName, item, want, kinds) checkSectionMethods(t, itemName, item, want, kinds)
} }
@@ -164,10 +177,12 @@ func checkSectionMethods(
want map[string]any, want map[string]any,
kinds map[string]reflect.Kind, kinds map[string]reflect.Kind,
) { ) {
t.Helper()
s := section.Addr().Interface().(Section) s := section.Addr().Interface().(Section)
if got := flattenMap("", s.Map()); !reflect.DeepEqual(got, want) { if got := flattenMap("", s.Map()); !reflect.DeepEqual(got, want) {
t.Errorf("%s.Map() =\n%v\nwant\n%v", name, got, want) t.Errorf("%s.Map() =\n%v\nwant\n%v", name, got, want)
} }
before := section.Interface() before := section.Interface()
for k, v := range want { for k, v := range want {
if kind, err := s.TypeOf(k); err != nil || kind != kinds[k] { if kind, err := s.TypeOf(k); err != nil || kind != kinds[k] {
@@ -180,6 +195,7 @@ func checkSectionMethods(
t.Errorf("%s.SetValue(%q, %#v): %v", name, k, v, err) t.Errorf("%s.SetValue(%q, %#v): %v", name, k, v, err)
} }
} }
if !reflect.DeepEqual(section.Interface(), before) { if !reflect.DeepEqual(section.Interface(), before) {
t.Errorf("%s changed by setting its keys to their own values", name) t.Errorf("%s changed by setting its keys to their own values", name)
} }
+17 -17
View File
@@ -1,6 +1,7 @@
package config package config
import ( import (
"errors"
"fmt" "fmt"
"log/slog" "log/slog"
"os" "os"
@@ -41,13 +42,13 @@ func loadEnv() error {
return nil return nil
} }
// loadConfig sets up Viper with the defaults, cfgFile (or config.* found in // loadConfig sets up Viper with the defaults, configFile (or config.* found in
// EtcConfigPath or the working directory) and JOBSEARCH_* environment // EtcConfigPath or the working directory) and JOBSEARCH_* environment
// variables. A missing config file is only a warning. // variables. A missing config file is only a warning.
func loadConfig(cfgFile string) error { func loadConfig(configFile string) error {
if cfgFile != "" { if configFile != "" {
// Use config file from the flag. // Use config file from the flag.
viper.SetConfigFile(cfgFile) viper.SetConfigFile(configFile)
} else { } else {
// Search config in /etc or local directory. // Search config in /etc or local directory.
viper.AddConfigPath(EtcConfigPath) viper.AddConfigPath(EtcConfigPath)
@@ -64,26 +65,25 @@ func loadConfig(cfgFile string) error {
viper.AutomaticEnv() // read in environment variables that match viper.AutomaticEnv() // read in environment variables that match
err := viper.ReadInConfig() err := viper.ReadInConfig()
if err != nil { var notFound viper.ConfigFileNotFoundError
if _, ok := err.(viper.ConfigFileNotFoundError); ok { switch {
// Indicates an explicitly set config file is not found or case errors.As(err, &notFound):
// that no config file was found in any search path // No config file was found in any search path.
slog.Warn("no config file found; using defaults") slog.Warn("no config file found; using defaults")
} else { case err != nil:
// Config file was found but another error was produced // A config file was found but couldn't be read or parsed.
return err return err
} default:
} else {
// Config file found and successfully parsed
slog.Info("using config file", "file", viper.ConfigFileUsed()) slog.Info("using config file", "file", viper.ConfigFileUsed())
} }
return nil return nil
} }
// Init loads the .env file and the configuration (see loadConfig) into conf. // Init loads the .env file and the configuration (see loadConfig) into conf.
// A debug or error log level already set in conf, from --debug or --quiet, // A debug or error log level already set in conf, from --debug or --quiet,
// is kept over the loaded one. // is kept over the loaded one.
func Init(cfgFile string, conf *JobSearchConfig) error { func Init(configFile string, conf *JobSearchConfig) error {
if err := loadEnv(); err != nil { if err := loadEnv(); err != nil {
return err return err
} }
@@ -91,7 +91,7 @@ func Init(cfgFile string, conf *JobSearchConfig) error {
printEnv() printEnv()
} }
if err := loadConfig(cfgFile); err != nil { if err := loadConfig(configFile); err != nil {
return err return err
} }
+34 -9
View File
@@ -18,6 +18,7 @@ func initDir(t *testing.T, env []string, config string) string {
resetViper(t) resetViper(t)
dir := t.TempDir() dir := t.TempDir()
t.Chdir(dir) t.Chdir(dir)
if env != nil { if env != nil {
// godotenv sets variables without cleaning up after the test. // godotenv sets variables without cleaning up after the test.
for _, kv := range env { for _, kv := range env {
@@ -28,9 +29,11 @@ func initDir(t *testing.T, env []string, config string) string {
} }
writeFile(t, filepath.Join(dir, ".env"), strings.Join(env, "\n")+"\n") writeFile(t, filepath.Join(dir, ".env"), strings.Join(env, "\n")+"\n")
} }
if config != "" { if config != "" {
writeFile(t, filepath.Join(dir, "config.yaml"), config) writeFile(t, filepath.Join(dir, "config.yaml"), config)
} }
return dir return dir
} }
@@ -63,12 +66,16 @@ func TestInitLoadsEnvAndConfigFile(t *testing.T) {
if conf.Logging.Level != "info" { if conf.Logging.Level != "info" {
t.Errorf("Logging.Level = %q; want info", conf.Logging.Level) t.Errorf("Logging.Level = %q; want info", conf.Logging.Level)
} }
if !conf.Sources.FTSMock.Enabled || conf.Sources.FTSMock.Interval != 30*time.Second { if !conf.Sources.FTSMock.Enabled ||
conf.Sources.FTSMock.Interval != 30*time.Second {
t.Errorf("FTSMock = %+v; want enabled every 30s", conf.Sources.FTSMock) t.Errorf("FTSMock = %+v; want enabled every 30s", conf.Sources.FTSMock)
} }
// Keys missing from the file keep their defaults. // Keys missing from the file keep their defaults.
if conf.Sources.FindATender.Interval != 10*time.Minute { if conf.Sources.FindATender.Interval != 10*time.Minute {
t.Errorf("FindATender.Interval = %v; want default 10m", conf.Sources.FindATender.Interval) t.Errorf(
"FindATender.Interval = %v; want default 10m",
conf.Sources.FindATender.Interval,
)
} }
} }
@@ -83,7 +90,9 @@ func TestInitExplicitConfigFile(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
if conf.Logging.Level != "error" { if conf.Logging.Level != "error" {
t.Errorf("Logging.Level = %q; want error from %s", conf.Logging.Level, other) t.Errorf(
"Logging.Level = %q; want error from %s", conf.Logging.Level, other,
)
} }
} }
@@ -124,7 +133,9 @@ func TestInitKeepsCommandLineLevel(t *testing.T) {
// At debug, the environment and loaded config are logged. // At debug, the environment and loaded config are logged.
gotDebug := strings.Contains(logs.String(), "loaded config value") gotDebug := strings.Contains(logs.String(), "loaded config value")
if gotDebug != (level == "debug") { if gotDebug != (level == "debug") {
t.Errorf("debug output logged = %v at level %s: %s", gotDebug, level, logs) t.Errorf(
"debug output logged = %v at level %s: %s", gotDebug, level, logs,
)
} }
}) })
} }
@@ -134,7 +145,8 @@ func TestInitErrors(t *testing.T) {
t.Run("no .env file", func(t *testing.T) { t.Run("no .env file", func(t *testing.T) {
initDir(t, nil, testConfigYAML) initDir(t, nil, testConfigYAML)
var conf JobSearchConfig var conf JobSearchConfig
if err := Init("", &conf); err == nil || !strings.Contains(err.Error(), ".env") { if err := Init("", &conf); err == nil ||
!strings.Contains(err.Error(), ".env") {
t.Fatalf("err = %v; want an invalid .env file error", err) t.Fatalf("err = %v; want an invalid .env file error", err)
} }
}) })
@@ -169,10 +181,15 @@ func TestInitEnvOverride(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
if conf.Logging.Level != "error" { if conf.Logging.Level != "error" {
t.Errorf("Logging.Level = %q; want error from the environment", conf.Logging.Level) t.Errorf(
"Logging.Level = %q; want error from the environment", conf.Logging.Level,
)
} }
if conf.Sources.FTSMock.Interval != 45*time.Second { if conf.Sources.FTSMock.Interval != 45*time.Second {
t.Errorf("FTSMock.Interval = %v; want 45s from the environment", conf.Sources.FTSMock.Interval) t.Errorf(
"FTSMock.Interval = %v; want 45s from the environment",
conf.Sources.FTSMock.Interval,
)
} }
} }
@@ -204,11 +221,15 @@ func loadExplicit(t *testing.T) {
t.Helper() t.Helper()
resetViper(t) resetViper(t)
LoadDefaultConfig() LoadDefaultConfig()
viper.SetEnvPrefix(EnvPrefix) viper.SetEnvPrefix(EnvPrefix)
viper.SetEnvKeyReplacer(envKeyReplacer) viper.SetEnvKeyReplacer(envKeyReplacer)
viper.AutomaticEnv() viper.AutomaticEnv()
viper.SetConfigType("yaml") viper.SetConfigType("yaml")
if err := viper.ReadConfig(strings.NewReader(explicitConfigYAML)); err != nil { if err := viper.ReadConfig(
strings.NewReader(explicitConfigYAML),
); err != nil {
t.Fatal(err) t.Fatal(err)
} }
} }
@@ -255,7 +276,11 @@ func TestPrintLoadedConfig(t *testing.T) {
t.Errorf("missing %q in:\n%s", want, out) t.Errorf("missing %q in:\n%s", want, out)
} }
} }
for _, unwanted := range []string{"sources.", "logging.file", "connection_limit.burst"} { for _, unwanted := range []string{
"sources.",
"logging.file",
"connection_limit.burst",
} {
if strings.Contains(out, unwanted) { if strings.Contains(out, unwanted) {
t.Errorf("logged default key %q:\n%s", unwanted, out) t.Errorf("logged default key %q:\n%s", unwanted, out)
} }
+15 -5
View File
@@ -14,14 +14,16 @@ import (
// file's own settings are written back, not defaults or environment // file's own settings are written back, not defaults or environment
// variables. It returns the stored value (as ValueOf gives it) and its kind. // variables. It returns the stored value (as ValueOf gives it) and its kind.
func StoreValue( func StoreValue(
conf *JobSearchConfig, key string, value string, conf *JobSearchConfig, key, value string,
) (any, reflect.Kind, error) { ) (any, reflect.Kind, error) {
return storeValue(conf, key, value) return storeValue(conf, key, value)
} }
// storeValue is StoreValue for any section, so key types that // storeValue is StoreValue for any section, so key types that
// JobSearchConfig doesn't have yet can be tested. // JobSearchConfig doesn't have yet can be tested.
func storeValue(section Section, key string, value string) (any, reflect.Kind, error) { func storeValue(
section Section, key, value string,
) (any, reflect.Kind, error) {
confType, err := section.TypeOf(key) confType, err := section.TypeOf(key)
if err != nil { if err != nil {
return nil, confType, err return nil, confType, err
@@ -31,7 +33,7 @@ func storeValue(section Section, key string, value string) (any, reflect.Kind, e
// SetValue gives. // SetValue gives.
if _, err := parseValue(confType, value); err != nil { if _, err := parseValue(confType, value); err != nil {
return nil, confType, fmt.Errorf( return nil, confType, fmt.Errorf(
"config value type not accepted, expected <%s>", confType, "config: value type not accepted, expected <%s>", confType,
) )
} }
@@ -69,7 +71,11 @@ func parseValue(kind reflect.Kind, value string) (any, error) {
return value, nil return value, nil
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return strconv.ParseInt(value, 10, 64) return strconv.ParseInt(value, 10, 64)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: case reflect.Uint,
reflect.Uint8,
reflect.Uint16,
reflect.Uint32,
reflect.Uint64:
return strconv.ParseUint(value, 10, 64) return strconv.ParseUint(value, 10, 64)
case reflect.Float32, reflect.Float64: case reflect.Float32, reflect.Float64:
return strconv.ParseFloat(value, 64) return strconv.ParseFloat(value, 64)
@@ -91,6 +97,7 @@ func PurgeValue(conf *JobSearchConfig, key string) error {
if err != nil { if err != nil {
return err return err
} }
out := viper.New() out := viper.New()
out.SetConfigFile(file.ConfigFileUsed()) out.SetConfigFile(file.ConfigFileUsed())
for _, fileKey := range file.AllKeys() { for _, fileKey := range file.AllKeys() {
@@ -98,6 +105,7 @@ func PurgeValue(conf *JobSearchConfig, key string) error {
out.Set(fileKey, file.Get(fileKey)) out.Set(fileKey, file.Get(fileKey))
} }
} }
return out.WriteConfig() return out.WriteConfig()
} }
@@ -106,12 +114,14 @@ func PurgeValue(conf *JobSearchConfig, key string) error {
func readUsedConfigFile() (*viper.Viper, error) { func readUsedConfigFile() (*viper.Viper, error) {
path := viper.ConfigFileUsed() path := viper.ConfigFileUsed()
if path == "" { if path == "" {
return nil, errors.New("no configuration file in use") return nil, errors.New("config: no configuration file in use")
} }
file := viper.New() file := viper.New()
file.SetConfigFile(path) file.SetConfigFile(path)
if err := file.ReadInConfig(); err != nil { if err := file.ReadInConfig(); err != nil {
return nil, err return nil, err
} }
return file, nil return file, nil
} }
+38 -10
View File
@@ -20,10 +20,12 @@ func useConfigFile(t *testing.T, contents string) string {
if err := os.WriteFile(path, []byte(contents), 0o600); err != nil { if err := os.WriteFile(path, []byte(contents), 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
viper.SetConfigFile(path) viper.SetConfigFile(path)
if err := viper.ReadInConfig(); err != nil { if err := viper.ReadInConfig(); err != nil {
t.Fatal(err) t.Fatal(err)
} }
return path return path
} }
@@ -74,6 +76,7 @@ func TestStoreValueKeepsOtherKeys(t *testing.T) {
if _, _, err := StoreValue(&conf, "logging.level", "error"); err != nil { if _, _, err := StoreValue(&conf, "logging.level", "error"); err != nil {
t.Fatal(err) t.Fatal(err)
} }
v := readConfigFile(t, path) v := readConfigFile(t, path)
if got := v.GetString("logging.file"); got != "/tmp/js.log" { if got := v.GetString("logging.file"); got != "/tmp/js.log" {
t.Errorf("logging.file = %q; want it kept", got) t.Errorf("logging.file = %q; want it kept", got)
@@ -159,7 +162,9 @@ func TestPurgeValueErrors(t *testing.T) {
func TestStoreValueUpdatesConf(t *testing.T) { func TestStoreValueUpdatesConf(t *testing.T) {
useConfigFile(t, "logging:\n level: warn\n") useConfigFile(t, "logging:\n level: warn\n")
conf := defaultConfig(t) conf := defaultConfig(t)
if _, _, err := StoreValue(&conf, "sources.fts_mock.interval", "30s"); err != nil { if _, _, err := StoreValue(
&conf, "sources.fts_mock.interval", "30s",
); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if conf.Sources.FTSMock.Interval != 30*time.Second { if conf.Sources.FTSMock.Interval != 30*time.Second {
@@ -185,7 +190,14 @@ func TestParseValue(t *testing.T) {
} { } {
got, err := parseValue(tt.kind, tt.value) got, err := parseValue(tt.kind, tt.value)
if err != nil || !reflect.DeepEqual(got, tt.want) { if err != nil || !reflect.DeepEqual(got, tt.want) {
t.Errorf("parseValue(%v, %q) = %#v, %v; want %#v", tt.kind, tt.value, got, err, tt.want) t.Errorf(
"parseValue(%v, %q) = %#v, %v; want %#v",
tt.kind,
tt.value,
got,
err,
tt.want,
)
} }
} }
@@ -199,11 +211,14 @@ func TestParseValue(t *testing.T) {
{reflect.Float64, "x"}, {reflect.Float64, "x"},
} { } {
if got, err := parseValue(tt.kind, tt.value); err == nil { if got, err := parseValue(tt.kind, tt.value); err == nil {
t.Errorf("parseValue(%v, %q) = %#v; want an error", tt.kind, tt.value, got) t.Errorf(
"parseValue(%v, %q) = %#v; want an error", tt.kind, tt.value, got,
)
} }
} }
if _, err := parseValue(reflect.Map, "a"); !errors.Is(err, errors.ErrUnsupported) { _, err := parseValue(reflect.Map, "a")
if !errors.Is(err, errors.ErrUnsupported) {
t.Errorf("parseValue(map) err = %v; want ErrUnsupported", err) t.Errorf("parseValue(map) err = %v; want ErrUnsupported", err)
} }
} }
@@ -217,11 +232,21 @@ type numberSection struct {
Ratio float32 `cfg:"ratio"` Ratio float32 `cfg:"ratio"`
} }
func (c numberSection) Map() map[string]any { return mapOf(c) } func (c numberSection) Map() map[string]any { return mapOf(c) }
func (c numberSection) Flatten() map[string]any { return flattenOf("", c) }
func (c numberSection) TypeOf(key string) (reflect.Kind, error) { return typeOf(c, key) } func (c numberSection) Flatten() map[string]any { return flattenOf("", c) }
func (c numberSection) ValueOf(key string) (any, error) { return valueOf(c, key) }
func (c *numberSection) SetValue(key string, value any) error { return setValue(c, key, value) } func (c numberSection) TypeOf(key string) (reflect.Kind, error) {
return typeOf(c, key)
}
func (c numberSection) ValueOf(key string) (any, error) {
return valueOf(c, key)
}
func (c *numberSection) SetValue(key string, value any) error {
return setValue(c, key, value)
}
// Numbers are stored as the field's own type, not as the int64, uint64 or // Numbers are stored as the field's own type, not as the int64, uint64 or
// float64 that parsing produces. // float64 that parsing produces.
@@ -276,7 +301,9 @@ func TestStoreAndPurgeWriteOnlyFileSettings(t *testing.T) {
LoadDefaultConfig() LoadDefaultConfig()
conf := defaultConfig(t) conf := defaultConfig(t)
if _, _, err := StoreValue(&conf, "sources.sleeper.enabled", "true"); err != nil { if _, _, err := StoreValue(
&conf, "sources.sleeper.enabled", "true",
); err != nil {
t.Fatal(err) t.Fatal(err)
} }
want := map[string]any{ want := map[string]any{
@@ -301,6 +328,7 @@ func checkFileSettings(t *testing.T, path string, want map[string]any) {
for _, k := range v.AllKeys() { for _, k := range v.AllKeys() {
got[k] = v.Get(k) got[k] = v.Get(k)
} }
if !reflect.DeepEqual(got, want) { if !reflect.DeepEqual(got, want) {
//nolint:gosec // G304: reading a file the test created in t.TempDir //nolint:gosec // G304: reading a file the test created in t.TempDir
b, _ := os.ReadFile(path) b, _ := os.ReadFile(path)
+6 -6
View File
@@ -15,13 +15,13 @@ func (c JobSearchConfig) PrintTerminal() {
fmt.Print(c.String() + "\n") fmt.Print(c.String() + "\n")
} }
// PrintJson prints the configuration to stdout as JSON. // PrintJSON prints the configuration to stdout as JSON.
func (c JobSearchConfig) PrintJson() { func (c JobSearchConfig) PrintJSON() {
fmt.Print(c.JSON() + "\n") fmt.Print(c.JSON() + "\n")
} }
// PrintYaml prints the configuration to stdout as YAML. // PrintYAML prints the configuration to stdout as YAML.
func (c JobSearchConfig) PrintYaml() { func (c JobSearchConfig) PrintYAML() {
fmt.Print(c.YAML() + "\n") fmt.Print(c.YAML() + "\n")
} }
@@ -29,7 +29,7 @@ func (c JobSearchConfig) PrintYaml() {
// debug, warn, error, or info for any other level. On a terminal it logs one // debug, warn, error, or info for any other level. On a terminal it logs one
// line and prints the values under it as "key = value" lines; elsewhere each // line and prints the values under it as "key = value" lines; elsewhere each
// value is a log entry of its own. // value is a log entry of its own.
func (conf JobSearchConfig) PrintLog(atLevel slog.Level) { func (c JobSearchConfig) PrintLog(atLevel slog.Level) {
switch atLevel { switch atLevel {
case slog.LevelDebug, slog.LevelWarn, slog.LevelError: case slog.LevelDebug, slog.LevelWarn, slog.LevelError:
default: default:
@@ -38,7 +38,7 @@ func (conf JobSearchConfig) PrintLog(atLevel slog.Level) {
ctx := context.Background() ctx := context.Background()
logger := slog.Default() logger := slog.Default()
flat := conf.Flatten() flat := c.Flatten()
keys := slices.Sorted(maps.Keys(flat)) keys := slices.Sorted(maps.Keys(flat))
if logging.WritesToTerminal(logger.Handler()) { if logging.WritesToTerminal(logger.Handler()) {
+35 -19
View File
@@ -20,17 +20,24 @@ func captureStdout(t *testing.T, fn func()) string {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
old := os.Stdout old := os.Stdout
os.Stdout = f os.Stdout = f
t.Cleanup(func() {
os.Stdout = old
})
fn() fn()
os.Stdout = old os.Stdout = old
if err := f.Close(); err != nil { if err := f.Close(); err != nil {
t.Fatal(err) t.Fatal(err)
} }
b, err := os.ReadFile(filepath.Clean(path)) b, err := os.ReadFile(filepath.Clean(path))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
return string(b) return string(b)
} }
@@ -50,6 +57,7 @@ func TestStringSorted(t *testing.T) {
key, _, _ := strings.Cut(line, " = ") key, _, _ := strings.Cut(line, " = ")
keys = append(keys, key) keys = append(keys, key)
} }
if !slices.IsSorted(keys) || len(keys) != len(conf.Flatten()) { if !slices.IsSorted(keys) || len(keys) != len(conf.Flatten()) {
t.Errorf("String keys = %v; want each key once, sorted", keys) t.Errorf("String keys = %v; want each key once, sorted", keys)
} }
@@ -58,27 +66,30 @@ func TestStringSorted(t *testing.T) {
} }
} }
func TestPrintJson(t *testing.T) { // PrintJSON and PrintYAML print the configuration in their format.
func TestPrintFormats(t *testing.T) {
conf := defaultConfig(t) conf := defaultConfig(t)
out := captureStdout(t, conf.PrintJson) tests := []struct {
var m map[string]any name string
if err := json.Unmarshal([]byte(out), &m); err != nil { print func()
t.Fatalf("PrintJson output is not JSON: %v\n%s", err, out) unmarshal func([]byte, any) error
}{
{"PrintJSON", conf.PrintJSON, json.Unmarshal},
{"PrintYAML", conf.PrintYAML, yaml.Unmarshal},
} }
if got := m["logging"].(map[string]any)["level"]; got != "warn" { for _, tt := range tests {
t.Errorf("logging.level = %v; want warn", got) t.Run(tt.name, func(t *testing.T) {
} out := captureStdout(t, tt.print)
} var m map[string]any
if err := tt.unmarshal([]byte(out), &m); err != nil {
t.Fatalf("%s output doesn't parse: %v\n%s", tt.name, err, out)
}
func TestPrintYaml(t *testing.T) { level := m["logging"].(map[string]any)["level"]
conf := defaultConfig(t) if level != "warn" {
out := captureStdout(t, conf.PrintYaml) t.Errorf("logging.level = %v; want warn", level)
var m map[string]any }
if err := yaml.Unmarshal([]byte(out), &m); err != nil { })
t.Fatalf("PrintYaml output is not YAML: %v\n%s", err, out)
}
if got := m["logging"].(map[string]any)["level"]; got != "warn" {
t.Errorf("logging.level = %v; want warn", got)
} }
} }
@@ -101,7 +112,12 @@ func TestPrintLog(t *testing.T) {
lines := strings.Split(strings.TrimSpace(logs.String()), "\n") lines := strings.Split(strings.TrimSpace(logs.String()), "\n")
if len(lines) != len(conf.Flatten()) { if len(lines) != len(conf.Flatten()) {
t.Errorf("PrintLog(%v) logged %d lines; want %d", tt.at, len(lines), len(conf.Flatten())) t.Errorf(
"PrintLog(%v) logged %d lines; want %d",
tt.at,
len(lines),
len(conf.Flatten()),
)
} }
for _, l := range lines { for _, l := range lines {
if !strings.Contains(l, tt.want) { if !strings.Contains(l, tt.want) {
+86 -47
View File
@@ -43,6 +43,7 @@ import (
// //
// A section can check values before they are stored by implementing // A section can check values before they are stored by implementing
// validator on its pointer type. // validator on its pointer type.
// The struct tags section.go reads. // The struct tags section.go reads.
const ( const (
// keyTag holds a field's configuration key. // keyTag holds a field's configuration key.
@@ -54,7 +55,7 @@ const (
// The unknown key error and the field types section.go handles specially. // The unknown key error and the field types section.go handles specially.
var ( var (
// errUnknownKey is returned for a key that names no field. // errUnknownKey is returned for a key that names no field.
errUnknownKey = errors.New("unknown configuration key") errUnknownKey = errors.New("config: unknown configuration key")
// durationType is time.Duration, read and given out as a string. // durationType is time.Duration, read and given out as a string.
durationType = reflect.TypeOf(time.Duration(0)) durationType = reflect.TypeOf(time.Duration(0))
// byteSizeType is util.ByteSize, read and given out as a string. // byteSizeType is util.ByteSize, read and given out as a string.
@@ -67,9 +68,9 @@ var (
// Section is the method set of every configuration section, including // Section is the method set of every configuration section, including
// JobSearchConfig itself, implemented on the section's pointer type. Keys are // JobSearchConfig itself, implemented on the section's pointer type. Keys are
// dotted paths relative to the section, e.g. "sleeper.interval" in // dotted paths relative to the section, e.g. "sleeper.interval" in JSCSources.
// JSCSources. Durations and byte sizes are given out as strings such as // Durations and byte sizes are given out as strings such as "10m0s" and
// "10m0s" and "32KiB", matching TypeOf's reflect.String for them. TestSections checks every // "32KiB", matching TypeOf's reflect.String for them. TestSections checks every
// section, found by walking JobSearchConfig, implements it. // section, found by walking JobSearchConfig, implements it.
type Section interface { type Section interface {
// Map returns the section as a map of keys to values, with nested // Map returns the section as a map of keys to values, with nested
@@ -102,18 +103,19 @@ type validator interface {
// field is a section struct field that is part of the configuration (see // field is a section struct field that is part of the configuration (see
// fieldsOf). // fieldsOf).
type field struct { type field struct {
key string key string
index []int // path for reflect.Value.FieldByIndex index []int // path for reflect.Value.FieldByIndex
typ reflect.Type fieldType reflect.Type
section bool section bool
def string defaultValue string
hasDefault bool hasDefault bool
} }
// isSectionList reports whether listType is a list of sections, []S for a // isSectionList reports whether listType is a list of sections, []S for a
// section struct S. // section struct S.
func isSectionList(listType reflect.Type) bool { func isSectionList(listType reflect.Type) bool {
return listType.Kind() == reflect.Slice && listType.Elem().Kind() == reflect.Struct && return listType.Kind() == reflect.Slice &&
listType.Elem().Kind() == reflect.Struct &&
listType.Elem() != timeType listType.Elem() != timeType
} }
@@ -123,6 +125,7 @@ func fieldsOf(structType reflect.Type) []field {
var fields []field var fields []field
for i := 0; i < structType.NumField(); i++ { for i := 0; i < structType.NumField(); i++ {
structField := structType.Field(i) structField := structType.Field(i)
if structField.Anonymous && structField.Type.Kind() == reflect.Struct { if structField.Anonymous && structField.Type.Kind() == reflect.Struct {
for _, embeddedField := range fieldsOf(structField.Type) { for _, embeddedField := range fieldsOf(structField.Type) {
embeddedField.index = append([]int{i}, embeddedField.index...) embeddedField.index = append([]int{i}, embeddedField.index...)
@@ -130,20 +133,24 @@ func fieldsOf(structType reflect.Type) []field {
} }
continue continue
} }
key, ok := structField.Tag.Lookup(keyTag) key, ok := structField.Tag.Lookup(keyTag)
if !ok || key == "" || key == "-" || !structField.IsExported() { if !ok || key == "" || key == "-" || !structField.IsExported() {
continue continue
} }
defaultValue, hasDefault := structField.Tag.Lookup(defaultTag) defaultValue, hasDefault := structField.Tag.Lookup(defaultTag)
fields = append(fields, field{ fields = append(fields, field{
key: key, key: key,
index: []int{i}, index: []int{i},
typ: structField.Type, fieldType: structField.Type,
section: structField.Type.Kind() == reflect.Struct && structField.Type != timeType, section: structField.Type.Kind() == reflect.Struct &&
def: defaultValue, structField.Type != timeType,
hasDefault: hasDefault, defaultValue: defaultValue,
hasDefault: hasDefault,
}) })
} }
return fields return fields
} }
@@ -249,6 +256,7 @@ func exportValue(fieldValue reflect.Value) any {
case byteSizeType: case byteSizeType:
return util.ByteSize(fieldValue.Int()).String() return util.ByteSize(fieldValue.Int()).String()
} }
if isSectionList(fieldValue.Type()) { if isSectionList(fieldValue.Type()) {
var items []map[string]any var items []map[string]any
for i := range fieldValue.Len() { for i := range fieldValue.Len() {
@@ -256,11 +264,13 @@ func exportValue(fieldValue reflect.Value) any {
} }
return items return items
} }
value := fieldValue.Interface() value := fieldValue.Interface()
if list, ok := value.([]string); ok { if list, ok := value.([]string); ok {
// A copy, so changing it doesn't change the configuration. // A copy, so changing it doesn't change the configuration.
return slices.Clone(list) return slices.Clone(list)
} }
return value return value
} }
@@ -292,7 +302,7 @@ func typeOf(value any, key string) (reflect.Kind, error) {
if err != nil { if err != nil {
return reflect.Invalid, err return reflect.Invalid, err
} }
return kindOf(configField.typ), nil return kindOf(configField.fieldType), nil
} }
// valueOf returns the value of the dotted key in the section value, or // valueOf returns the value of the dotted key in the section value, or
@@ -318,22 +328,25 @@ func setValue(ptr any, key string, value any) error {
// validator, checks it, before setting the field. On error the field is left // validator, checks it, before setting the field. On error the field is left
// unchanged. // unchanged.
func setField(section reflect.Value, configField field, value any) error { func setField(section reflect.Value, configField field, value any) error {
newValue, err := convert(value, configField.typ) newValue, err := convert(value, configField.fieldType)
if err != nil { if err != nil {
return err return err
} }
if checker, ok := section.Addr().Interface().(validator); ok { if checker, ok := section.Addr().Interface().(validator); ok {
if err := checker.validate(configField.key, newValue.Interface()); err != nil { err := checker.validate(configField.key, newValue.Interface())
if err != nil {
return err return err
} }
} }
section.FieldByIndex(configField.index).Set(newValue) section.FieldByIndex(configField.index).Set(newValue)
return nil return nil
} }
// convert turns value into a value of type targetType. Values already of the right // convert turns value into a value of type targetType. Values already of the
// type are used as-is, except that an empty []string becomes nil, as an empty // right type are used as-is, except that an empty []string becomes nil, as an
// comma-separated list does (see splitList); strings are parsed. Lists of // empty comma-separated list does (see splitList); strings are parsed. Lists of
// sections are converted by convertSectionList. // sections are converted by convertSectionList.
func convert(value any, targetType reflect.Type) (reflect.Value, error) { func convert(value any, targetType reflect.Type) (reflect.Value, error) {
if isSectionList(targetType) { if isSectionList(targetType) {
@@ -351,36 +364,38 @@ func convert(value any, targetType reflect.Type) (reflect.Value, error) {
return parseString(text, targetType) return parseString(text, targetType)
} }
} }
return reflect.Value{}, fmt.Errorf("unsupported type %T", value) return reflect.Value{}, fmt.Errorf("config: unsupported type %T", value)
} }
// convertSectionList turns value into a list of sections of type listType: a // convertSectionList turns value into a list of sections of type listType: a
// value of type listType, a list of maps of keys to values, or a JSON array of objects. // value of type listType, a list of maps of keys to values, or a JSON array of
// nil and empty lists become a nil list. // objects. nil and empty lists become a nil list.
func convertSectionList(value any, listType reflect.Type) (reflect.Value, error) { func convertSectionList(
value any, listType reflect.Type,
) (reflect.Value, error) {
if text, ok := value.(string); ok { if text, ok := value.(string); ok {
var items []map[string]any var items []map[string]any
if strings.TrimSpace(text) != "" { if strings.TrimSpace(text) != "" {
if err := json.Unmarshal([]byte(text), &items); err != nil { if err := json.Unmarshal([]byte(text), &items); err != nil {
return reflect.Value{}, fmt.Errorf( return reflect.Value{}, fmt.Errorf(
"unable to convert to a list of %s: %w", "unable to convert to a list of %s: %w", listType.Elem(), err,
listType.Elem(),
err,
) )
} }
} }
value = items value = items
} }
if value == nil { if value == nil {
return reflect.Zero(listType), nil return reflect.Zero(listType), nil
} }
reflected := reflect.ValueOf(value) reflected := reflect.ValueOf(value)
if reflected.Kind() != reflect.Slice { if reflected.Kind() != reflect.Slice {
return reflect.Value{}, fmt.Errorf("unsupported type %T", value) return reflect.Value{}, fmt.Errorf("config: unsupported type %T", value)
} }
if reflected.Len() == 0 { if reflected.Len() == 0 {
return reflect.Zero(listType), nil return reflect.Zero(listType), nil
} }
if reflected.Type().AssignableTo(listType) { if reflected.Type().AssignableTo(listType) {
for i := range reflected.Len() { for i := range reflected.Len() {
if err := validateSection(reflected.Index(i)); err != nil { if err := validateSection(reflected.Index(i)); err != nil {
@@ -389,6 +404,7 @@ func convertSectionList(value any, listType reflect.Type) (reflect.Value, error)
} }
return reflected, nil return reflected, nil
} }
list := reflect.MakeSlice(listType, 0, reflected.Len()) list := reflect.MakeSlice(listType, 0, reflected.Len())
for i := range reflected.Len() { for i := range reflected.Len() {
item, err := sectionFromMap(reflected.Index(i).Interface(), listType.Elem()) item, err := sectionFromMap(reflected.Index(i).Interface(), listType.Elem())
@@ -397,6 +413,7 @@ func convertSectionList(value any, listType reflect.Type) (reflect.Value, error)
} }
list = reflect.Append(list, item) list = reflect.Append(list, item)
} }
return list, nil return list, nil
} }
@@ -408,28 +425,32 @@ func convertSectionList(value any, listType reflect.Type) (reflect.Value, error)
func sectionFromMap(value any, structType reflect.Type) (reflect.Value, error) { func sectionFromMap(value any, structType reflect.Type) (reflect.Value, error) {
item, ok := value.(map[string]any) item, ok := value.(map[string]any)
if !ok { if !ok {
return reflect.Value{}, fmt.Errorf("unsupported type %T", value) return reflect.Value{}, fmt.Errorf("config: unsupported type %T", value)
} }
section := reflect.New(structType).Elem() section := reflect.New(structType).Elem()
if err := applyDefaultsTo("", section); err != nil { if err := applyDefaultsTo("", section); err != nil {
return reflect.Value{}, err return reflect.Value{}, err
} }
for itemKey, itemValue := range item { for itemKey, itemValue := range item {
holder, configField, err := lookup(section, itemKey) holder, configField, err := lookup(section, itemKey)
if err != nil { if err != nil {
return reflect.Value{}, fmt.Errorf("%w %q", err, itemKey) return reflect.Value{}, fmt.Errorf("%w %q", err, itemKey)
} }
if _, isString := itemValue.(string); itemValue != nil && !isString && if _, isString := itemValue.(string); itemValue != nil && !isString &&
!reflect.TypeOf(itemValue).AssignableTo(configField.typ) { !reflect.TypeOf(itemValue).AssignableTo(configField.fieldType) {
itemValue = fmt.Sprint(itemValue) itemValue = fmt.Sprint(itemValue)
} }
if err := setField(holder, configField, itemValue); err != nil { if err := setField(holder, configField, itemValue); err != nil {
return reflect.Value{}, fmt.Errorf("%s: %w", itemKey, err) return reflect.Value{}, fmt.Errorf("%s: %w", itemKey, err)
} }
} }
if err := validateSection(section); err != nil { if err := validateSection(section); err != nil {
return reflect.Value{}, err return reflect.Value{}, err
} }
return section, nil return section, nil
} }
@@ -441,37 +462,41 @@ func validateSection(section reflect.Value) error {
if !ok { if !ok {
return nil return nil
} }
for _, configField := range fieldsOf(section.Type()) { for _, configField := range fieldsOf(section.Type()) {
if configField.section { if configField.section {
continue continue
} }
if err := checker.validate( value := section.FieldByIndex(configField.index).Interface()
configField.key, if err := checker.validate(configField.key, value); err != nil {
section.FieldByIndex(configField.index).Interface(),
); err != nil {
return fmt.Errorf("%s: %w", configField.key, err) return fmt.Errorf("%s: %w", configField.key, err)
} }
} }
return nil return nil
} }
// parseString parses text as a value of type targetType. It supports strings, bools, // parseString parses text as a value of type targetType. It supports strings,
// signed and unsigned integers, floats, time.Duration, ByteSize and []string // bools, signed and unsigned integers, floats, time.Duration, ByteSize and
// (from a comma-separated list, see splitList). // []string (from a comma-separated list, see splitList).
func parseString(text string, targetType reflect.Type) (reflect.Value, error) { func parseString(text string, targetType reflect.Type) (reflect.Value, error) {
parsed := reflect.New(targetType).Elem() parsed := reflect.New(targetType).Elem()
switch targetType { switch targetType {
case durationType: case durationType:
duration, err := time.ParseDuration(text) duration, err := time.ParseDuration(text)
if err != nil { if err != nil {
return reflect.Value{}, fmt.Errorf("unable to convert to duration: %w", err) return reflect.Value{}, fmt.Errorf(
"unable to convert to duration: %w", err,
)
} }
parsed.SetInt(int64(duration)) parsed.SetInt(int64(duration))
return parsed, nil return parsed, nil
case byteSizeType: case byteSizeType:
size, err := util.ParseByteSize(text) size, err := util.ParseByteSize(text)
if err != nil { if err != nil {
return reflect.Value{}, fmt.Errorf("unable to convert to byte size: %w", err) return reflect.Value{}, fmt.Errorf(
"unable to convert to byte size: %w", err,
)
} }
parsed.SetInt(int64(size)) parsed.SetInt(int64(size))
return parsed, nil return parsed, nil
@@ -492,7 +517,11 @@ func parseString(text string, targetType reflect.Type) (reflect.Value, error) {
var number int64 var number int64
number, err = strconv.ParseInt(text, 10, targetType.Bits()) number, err = strconv.ParseInt(text, 10, targetType.Bits())
parsed.SetInt(number) parsed.SetInt(number)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: case reflect.Uint,
reflect.Uint8,
reflect.Uint16,
reflect.Uint32,
reflect.Uint64:
var number uint64 var number uint64
number, err = strconv.ParseUint(text, 10, targetType.Bits()) number, err = strconv.ParseUint(text, 10, targetType.Bits())
parsed.SetUint(number) parsed.SetUint(number)
@@ -501,11 +530,16 @@ func parseString(text string, targetType reflect.Type) (reflect.Value, error) {
number, err = strconv.ParseFloat(text, targetType.Bits()) number, err = strconv.ParseFloat(text, targetType.Bits())
parsed.SetFloat(number) parsed.SetFloat(number)
default: default:
return reflect.Value{}, fmt.Errorf("unsupported type string for %s", targetType) return reflect.Value{}, fmt.Errorf(
"config: unsupported type string for %s", targetType,
)
} }
if err != nil { if err != nil {
return reflect.Value{}, fmt.Errorf("unable to convert %q to %s: %w", text, targetType, err) return reflect.Value{}, fmt.Errorf(
"unable to convert %q to %s: %w", text, targetType, err,
)
} }
return parsed, nil return parsed, nil
} }
@@ -521,16 +555,21 @@ func applyDefaults(ptr any) error {
func applyDefaultsTo(prefix string, section reflect.Value) error { func applyDefaultsTo(prefix string, section reflect.Value) error {
for _, configField := range fieldsOf(section.Type()) { for _, configField := range fieldsOf(section.Type()) {
key := joinKey(prefix, configField.key) key := joinKey(prefix, configField.key)
if configField.section { if configField.section {
if err := applyDefaultsTo(key, section.FieldByIndex(configField.index)); err != nil { nested := section.FieldByIndex(configField.index)
if err := applyDefaultsTo(key, nested); err != nil {
return err return err
} }
continue continue
} }
if !configField.hasDefault { if !configField.hasDefault {
continue continue
} }
if err := setField(section, configField, configField.def); err != nil {
err := setField(section, configField, configField.defaultValue)
if err != nil {
return fmt.Errorf("invalid default for %s: %w", key, err) return fmt.Errorf("invalid default for %s: %w", key, err)
} }
} }
+19 -9
View File
@@ -222,7 +222,8 @@ func TestUnknownKeys(t *testing.T) {
} }
var s testSection var s testSection
for _, key := range keys { for _, key := range keys {
if k, err := typeOf(s, key); !errors.Is(err, errUnknownKey) || k != reflect.Invalid { if k, err := typeOf(s, key); !errors.Is(err, errUnknownKey) ||
k != reflect.Invalid {
t.Errorf("typeOf(%q) = %v, %v; want Invalid, errUnknownKey", key, k, err) t.Errorf("typeOf(%q) = %v, %v; want Invalid, errUnknownKey", key, k, err)
} }
if v, err := valueOf(s, key); !errors.Is(err, errUnknownKey) || v != nil { if v, err := valueOf(s, key); !errors.Is(err, errUnknownKey) || v != nil {
@@ -266,17 +267,24 @@ func TestSetValue(t *testing.T) {
{"slice", []string{}, []string(nil)}, // empty lists are stored as nil {"slice", []string{}, []string(nil)}, // empty lists are stored as nil
// lists of sections are given out as lists of maps; items start // lists of sections are given out as lists of maps; items start
// from their defaults and their values are parsed // from their defaults and their values are parsed
{"items", `[{"n": "1", "wait": "2s"}, {"name": "b", "n": 2}]`, []map[string]any{ {
{"name": "item", "n": 1, "wait": "2s"}, "items",
{"name": "b", "n": 2, "wait": "0s"}, `[{"n": "1", "wait": "2s"}, {"name": "b", "n": 2}]`,
}}, []map[string]any{
{"name": "item", "n": 1, "wait": "2s"},
{"name": "b", "n": 2, "wait": "0s"},
},
},
{"items", []any{map[string]any{"name": "a", "n": 3}}, []map[string]any{ {"items", []any{map[string]any{"name": "a", "n": 3}}, []map[string]any{
{"name": "a", "n": 3, "wait": "0s"}, {"name": "a", "n": 3, "wait": "0s"},
}}, }},
{"items", []map[string]any{{"name": "a"}}, []map[string]any{ {"items", []map[string]any{{"name": "a"}}, []map[string]any{
{"name": "a", "n": 0, "wait": "0s"}, {"name": "a", "n": 0, "wait": "0s"},
}}, }},
{"items", []testItem{{Name: "a", N: 4}}, []map[string]any{ {"items", []testItem{{
Name: "a",
N: 4,
}}, []map[string]any{
{"name": "a", "n": 4, "wait": "0s"}, {"name": "a", "n": 4, "wait": "0s"},
}}, }},
{"items", "", []map[string]any(nil)}, {"items", "", []map[string]any(nil)},
@@ -318,9 +326,9 @@ func TestSetValueErrors(t *testing.T) {
{"on", "maybe", "unable to convert"}, {"on", "maybe", "unable to convert"},
{"dur", "soon", "unable to convert to duration"}, {"dur", "soon", "unable to convert to duration"},
{"items", "not json", "unable to convert to a list of config.testItem"}, {"items", "not json", "unable to convert to a list of config.testItem"},
{"items", `[{"bogus": 1}]`, `item 0: unknown configuration key "bogus"`}, {"items", `[{"bogus": 1}]`, `item 0: config: unknown configuration key "bogus"`},
{"items", `[{"n": "x"}]`, "item 0: n: unable to convert"}, {"items", `[{"n": "x"}]`, "item 0: n: unable to convert"},
{"items", []any{"a"}, "item 0: unsupported type string"}, {"items", []any{"a"}, "item 0: config: unsupported type string"},
{"items", 5, "unsupported type int"}, {"items", 5, "unsupported type int"},
{"size", "lots", "unable to convert to byte size"}, {"size", "lots", "unable to convert to byte size"},
{"size", "-1KB", "unable to convert to byte size"}, {"size", "-1KB", "unable to convert to byte size"},
@@ -344,7 +352,9 @@ func TestSetValueErrors(t *testing.T) {
) )
} }
if !reflect.DeepEqual(s, before) { if !reflect.DeepEqual(s, before) {
t.Errorf("setValue(%q, %#v) modified the section on error", tt.key, tt.value) t.Errorf(
"setValue(%q, %#v) modified the section on error", tt.key, tt.value,
)
} }
} }
} }
+19 -9
View File
@@ -33,7 +33,8 @@ func WithLogger(logger *slog.Logger) Option {
} }
// logger returns the logger Run logs to: the given one, or slog.Default(). // logger returns the logger Run logs to: the given one, or slog.Default().
// It isn't tagged here; each worker adds its source. // It isn't tagged here, as it is also given to the clients the workers
// create, which add their own component; see tagged.
func (opts options) logger() *slog.Logger { func (opts options) logger() *slog.Logger {
if opts.log == nil { if opts.log == nil {
return slog.Default() return slog.Default()
@@ -41,8 +42,15 @@ func (opts options) logger() *slog.Logger {
return opts.log return opts.log
} }
// runFunc runs one source's worker until ctx is cancelled, logging to // tagged returns logger tagged with the fetcher's component, for the lines
// logger. // Run and its workers log themselves.
func tagged(logger *slog.Logger) *slog.Logger {
return logger.With("component", "fetcher")
}
// runFunc runs one source's worker until ctx is cancelled. logger is
// untagged: the worker tags its own lines with tagged, and passes logger as
// it is to any client it creates.
type runFunc func( type runFunc func(
ctx context.Context, ctx context.Context,
logger *slog.Logger, logger *slog.Logger,
@@ -68,7 +76,7 @@ func Run(
for _, opt := range opts { for _, opt := range opts {
opt(&settings) opt(&settings)
} }
logger := settings.logger().With("component", "fetcher") logger := settings.logger()
var ( var (
wg sync.WaitGroup wg sync.WaitGroup
@@ -76,15 +84,16 @@ func Run(
errs []error errs []error
) )
start := func( start := func(name string, run runFunc) {
name string,
run runFunc,
) {
wg.Add(1) wg.Add(1)
go func() { go func() {
defer wg.Done() defer wg.Done()
if err := run(ctx, logger, conf, since); err != nil { if err := run(ctx, logger, conf, since); err != nil {
logger.Error("source stopped, shutting down", "source", name, "error", err) tagged(logger).Error(
"source stopped, shutting down",
"source", name,
"error", err,
)
mu.Lock() mu.Lock()
errs = append(errs, fmt.Errorf("%s: %w", name, err)) errs = append(errs, fmt.Errorf("%s: %w", name, err))
mu.Unlock() mu.Unlock()
@@ -104,5 +113,6 @@ func Run(
} }
wg.Wait() wg.Wait()
return errors.Join(errs...) return errors.Join(errs...)
} }
+40 -19
View File
@@ -20,7 +20,7 @@ import (
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
// Keep worker and fetch log lines out of the test output. // Keep worker and fetch log lines out of the test output.
slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, nil))) slog.SetDefault(slog.New(slog.DiscardHandler))
os.Exit(m.Run()) os.Exit(m.Run())
} }
@@ -38,18 +38,22 @@ const onePackage = `{
const emptyPackage = `{"uri": "x", "releases": []}` const emptyPackage = `{"uri": "x", "releases": []}`
// ftsServer serves body (with status) for every request and counts requests. // ftsServer serves body (with status) for every request and counts requests.
func ftsServer(t *testing.T, status int, body string) (*httptest.Server, *atomic.Int32) { func ftsServer(
t *testing.T, status int, body string,
) (*httptest.Server, *atomic.Int32) {
t.Helper() t.Helper()
var hits atomic.Int32 var hits atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { srv := httptest.NewServer(
hits.Add(1) http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/ocdsReleasePackages" { hits.Add(1)
t.Errorf("unexpected path %q", r.URL.Path) if r.URL.Path != "/ocdsReleasePackages" {
} t.Errorf("unexpected path %q", r.URL.Path)
w.Header().Set("Content-Type", "application/json") }
w.WriteHeader(status) w.Header().Set("Content-Type", "application/json")
io.WriteString(w, body) w.WriteHeader(status)
})) io.WriteString(w, body)
}),
)
t.Cleanup(srv.Close) t.Cleanup(srv.Close)
return srv, &hits return srv, &hits
} }
@@ -83,7 +87,8 @@ func TestFetchFunctions(t *testing.T) {
"nil", "nil",
}, },
{"no releases", http.StatusOK, emptyPackage, func(err error) bool { {"no releases", http.StatusOK, emptyPackage, func(err error) bool {
return errors.Is(err, ErrNoNewReleases) && errors.Is(err, util.ErrWorkerNoWork) return errors.Is(err, ErrNoNewReleases) &&
errors.Is(err, util.ErrWorkerNoWork)
}, "ErrNoNewReleases wrapping ErrWorkerNoWork"}, }, "ErrNoNewReleases wrapping ErrWorkerNoWork"},
{"bad body", http.StatusInternalServerError, "oops", func(err error) bool { {"bad body", http.StatusInternalServerError, "oops", func(err error) bool {
return err != nil && !errors.Is(err, util.ErrWorkerNoWork) return err != nil && !errors.Is(err, util.ErrWorkerNoWork)
@@ -119,8 +124,11 @@ func TestFetchUsesConfiguredAPIRoot(t *testing.T) {
fetchFindAOpportunity(context.Background(), slog.Default(), conf, time.Time{}) fetchFindAOpportunity(context.Background(), slog.Default(), conf, time.Time{})
if fatHits.Load() != 1 || mockHits.Load() != 0 { if fatHits.Load() != 1 || mockHits.Load() != 0 {
t.Fatalf("FindAOpportunity hits: fat=%d mock=%d", fatHits.Load(), mockHits.Load()) t.Fatalf(
"FindAOpportunity hits: fat=%d mock=%d", fatHits.Load(), mockHits.Load(),
)
} }
fetchFTSMock(context.Background(), slog.Default(), conf, time.Time{}) fetchFTSMock(context.Background(), slog.Default(), conf, time.Time{})
if fatHits.Load() != 1 || mockHits.Load() != 1 { if fatHits.Load() != 1 || mockHits.Load() != 1 {
t.Fatalf("FTSMock hits: fat=%d mock=%d", fatHits.Load(), mockHits.Load()) t.Fatalf("FTSMock hits: fat=%d mock=%d", fatHits.Load(), mockHits.Load())
@@ -144,9 +152,12 @@ func TestFetchCancelled(t *testing.T) {
func TestSleepCancelled(t *testing.T) { func TestSleepCancelled(t *testing.T) {
for _, block := range []bool{false, true} { for _, block := range []bool{false, true} {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond) ctx, cancel := context.WithTimeout(
context.Background(), 10*time.Millisecond,
)
defer cancel() defer cancel()
if err := sleep(ctx, block, time.Hour); !errors.Is(err, context.DeadlineExceeded) { err := sleep(ctx, block, time.Hour)
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("block=%v: err = %v, want context.DeadlineExceeded", block, err) t.Fatalf("block=%v: err = %v, want context.DeadlineExceeded", block, err)
} }
} }
@@ -198,7 +209,9 @@ func TestRunWorkersRejectZeroInterval(t *testing.T) {
"Sleeper": runSleeper, "Sleeper": runSleeper,
} }
for name, run := range runs { for name, run := range runs {
if err := run(context.Background(), slog.Default(), conf, time.Time{}); err == nil { if err := run(
context.Background(), slog.Default(), conf, time.Time{},
); err == nil {
t.Errorf("%s: expected an error for a zero interval", name) t.Errorf("%s: expected an error for a zero interval", name)
} }
} }
@@ -228,7 +241,9 @@ func TestRunLogsSourceErrors(t *testing.T) {
t.Errorf("returned error %v; want it to include %s", err, name) t.Errorf("returned error %v; want it to include %s", err, name)
} }
} }
if n := strings.Count(buf.String(), "interval must be greater than 0"); n != 3 { if n := strings.Count(
buf.String(), "interval must be greater than 0",
); n != 3 {
t.Errorf("logged %d interval errors; want 3: %s", n, &buf) t.Errorf("logged %d interval errors; want 3: %s", n, &buf)
} }
} }
@@ -236,11 +251,14 @@ func TestRunLogsSourceErrors(t *testing.T) {
func TestRunNoSourcesEnabled(t *testing.T) { func TestRunNoSourcesEnabled(t *testing.T) {
done := make(chan struct{}) done := make(chan struct{})
go func() { go func() {
if err := Run(context.Background(), &config.JobSearchConfig{}, time.Time{}); err != nil { if err := Run(
context.Background(), &config.JobSearchConfig{}, time.Time{},
); err != nil {
t.Errorf("Run = %v; want nil", err) t.Errorf("Run = %v; want nil", err)
} }
close(done) close(done)
}() }()
select { select {
case <-done: case <-done:
case <-time.After(time.Second): case <-time.After(time.Second):
@@ -275,6 +293,7 @@ func TestRunShutsDownOnWorkerError(t *testing.T) {
case <-time.After(5 * time.Second): case <-time.After(5 * time.Second):
t.Fatal("Run did not shut down after a worker failed") t.Fatal("Run did not shut down after a worker failed")
} }
// FTSMock either polled once before the shutdown or never started a run. // FTSMock either polled once before the shutdown or never started a run.
if n := mockHits.Load(); n > 1 { if n := mockHits.Load(); n > 1 {
t.Errorf("FTSMock polled %d times; want at most 1", n) t.Errorf("FTSMock polled %d times; want at most 1", n)
@@ -307,7 +326,9 @@ func TestRunEnabledSources(t *testing.T) {
deadline := time.Now().Add(5 * time.Second) deadline := time.Now().Add(5 * time.Second)
for fatHits.Load() == 0 || mockHits.Load() == 0 { for fatHits.Load() == 0 || mockHits.Load() == 0 {
if time.Now().After(deadline) { if time.Now().After(deadline) {
t.Fatalf("sources not polled: fat=%d mock=%d", fatHits.Load(), mockHits.Load()) t.Fatalf(
"sources not polled: fat=%d mock=%d", fatHits.Load(), mockHits.Load(),
)
} }
time.Sleep(5 * time.Millisecond) time.Sleep(5 * time.Millisecond)
} }
+12 -11
View File
@@ -21,7 +21,7 @@ func runFindAOpportunity(
since time.Time, since time.Time,
) error { ) error {
worker := util.Worker{ worker := util.Worker{
Logger: logger, Logger: tagged(logger),
Name: "FindAOpportunity", Name: "FindAOpportunity",
Interval: conf.Sources.FindATender.Interval, Interval: conf.Sources.FindATender.Interval,
Task: func(ctx context.Context, since time.Time) error { Task: func(ctx context.Context, since time.Time) error {
@@ -40,15 +40,16 @@ func fetchFindAOpportunity(
conf *config.JobSearchConfig, conf *config.JobSearchConfig,
since time.Time, since time.Time,
) error { ) error {
opportunities, err := util.AwaitTask(ctx, func() ([]*types.Opportunity, error) { opportunities, err := util.AwaitTask(
findATender := findatender.New( ctx,
conf, func() ([]*types.Opportunity, error) {
conf.Sources.FindATender.APIRoot, findATender := findatender.New(
findatender.WithLogger(logger), conf, conf.Sources.FindATender.APIRoot, findatender.WithLogger(logger),
) )
defer findATender.Shutdown() defer findATender.Shutdown()
return findATender.With(ctx).FetchAll(since, true) return findATender.With(ctx).FetchAll(since, true)
}) },
)
if errors.Is(err, findatender.ErrNoReleases) { if errors.Is(err, findatender.ErrNoReleases) {
return ErrNoNewReleases return ErrNoNewReleases
} }
@@ -56,7 +57,7 @@ func fetchFindAOpportunity(
return err return err
} }
logger.Info("fetched opportunities", "count", len(opportunities)) tagged(logger).Info("fetched opportunities", "count", len(opportunities))
return nil return nil
} }
+12 -11
View File
@@ -21,7 +21,7 @@ func runFTSMock(
since time.Time, since time.Time,
) error { ) error {
worker := util.Worker{ worker := util.Worker{
Logger: logger, Logger: tagged(logger),
Name: "FTSMock", Name: "FTSMock",
Interval: conf.Sources.FTSMock.Interval, Interval: conf.Sources.FTSMock.Interval,
Task: func(ctx context.Context, since time.Time) error { Task: func(ctx context.Context, since time.Time) error {
@@ -40,15 +40,16 @@ func fetchFTSMock(
conf *config.JobSearchConfig, conf *config.JobSearchConfig,
since time.Time, since time.Time,
) error { ) error {
opportunities, err := util.AwaitTask(ctx, func() ([]*types.Opportunity, error) { opportunities, err := util.AwaitTask(
findATender := findatender.New( ctx,
conf, func() ([]*types.Opportunity, error) {
conf.Sources.FTSMock.APIRoot, findATender := findatender.New(
findatender.WithLogger(logger), conf, conf.Sources.FTSMock.APIRoot, findatender.WithLogger(logger),
) )
defer findATender.Shutdown() defer findATender.Shutdown()
return findATender.With(ctx).FetchAll(since, true) return findATender.With(ctx).FetchAll(since, true)
}) },
)
if errors.Is(err, findatender.ErrNoReleases) { if errors.Is(err, findatender.ErrNoReleases) {
return ErrNoNewReleases return ErrNoNewReleases
} }
@@ -56,7 +57,7 @@ func fetchFTSMock(
return err return err
} }
logger.Info("fetched opportunities", "count", len(opportunities)) tagged(logger).Info("fetched opportunities", "count", len(opportunities))
return nil return nil
} }
+4 -3
View File
@@ -22,15 +22,14 @@ func runSleeper(
) error { ) error {
blockShutdown := conf.Sources.Sleeper.BlockShutdown blockShutdown := conf.Sources.Sleeper.BlockShutdown
worker := util.Worker{ worker := util.Worker{
Logger: logger, Logger: tagged(logger),
Name: "Sleeper", Name: "Sleeper",
Interval: conf.Sources.Sleeper.Interval, Interval: conf.Sources.Sleeper.Interval,
Task: func(taskCtx context.Context, since time.Time) error { Task: func(taskCtx context.Context, since time.Time) error {
if blockShutdown { if blockShutdown {
return sleep(taskCtx, true, sleepDuration) return sleep(taskCtx, true, sleepDuration)
} else {
return sleep(ctx, false, sleepDuration)
} }
return sleep(ctx, false, sleepDuration)
}, },
} }
return worker.Run(ctx, since) return worker.Run(ctx, since)
@@ -47,8 +46,10 @@ func sleep(
time.Sleep(duration) time.Sleep(duration)
return nil return nil
} }
timer := time.NewTimer(duration) timer := time.NewTimer(duration)
defer timer.Stop() defer timer.Stop()
select { select {
case <-timer.C: case <-timer.C:
return nil return nil
+46 -35
View File
@@ -22,17 +22,17 @@ import (
// Errors returned when fetching from the API. // Errors returned when fetching from the API.
var ( var (
// ErrNoNext means a release package has no next page link. // ErrNoNext means a release package has no next page link.
ErrNoNext = errors.New("no next link") ErrNoNext = errors.New("findatender: no next link")
// ErrNoReleases means the API returned no releases. // ErrNoReleases means the API returned no releases.
ErrNoReleases = errors.New("no releases") ErrNoReleases = errors.New("findatender: no releases")
// ErrRateLimitExceeded means the API refused a request with 429 Too Many // ErrRateLimitExceeded means the API refused a request with 429 Too Many
// Requests, and waiting for it was not allowed or did not help. // Requests, and waiting for it was not allowed or did not help.
ErrRateLimitExceeded = errors.New("rate limit has been exceeded") ErrRateLimitExceeded = errors.New("findatender: rate limit has been exceeded")
// ErrFetchCancelled means the fetch's context was cancelled while waiting // ErrFetchCancelled means the fetch's context was cancelled while waiting
// out a rate limit. // out a rate limit.
ErrFetchCancelled = errors.New("fetch cancelled") ErrFetchCancelled = errors.New("findatender: fetch cancelled")
// ErrUnexpectedStatus wraps an API response status other than 200 OK. // ErrUnexpectedStatus wraps an API response status other than 200 OK.
ErrUnexpectedStatus = errors.New("unexpected response status") ErrUnexpectedStatus = errors.New("findatender: unexpected response status")
) )
// RequestOptions selects the releases a fetch asks for. // RequestOptions selects the releases a fetch asks for.
@@ -95,7 +95,7 @@ func repairDates(raw json.RawMessage) (json.RawMessage, error) {
if _, err := decoder.Token(); !errors.Is(err, io.EOF) { if _, err := decoder.Token(); !errors.Is(err, io.EOF) {
return nil, errors.New( return nil, errors.New(
"repairing dates: unexpected data after the JSON value", "findatender: repairing dates: unexpected data after the JSON value",
) )
} }
@@ -106,8 +106,8 @@ func repairDates(raw json.RawMessage) (json.RawMessage, error) {
return json.Marshal(doc) return json.Marshal(doc)
} }
// recursiveRepairDates fixes date fields in value in place and reports whether it // recursiveRepairDates fixes date fields in value in place and reports whether
// changed anything. // it changed anything.
func recursiveRepairDates(value any) bool { func recursiveRepairDates(value any) bool {
changed := false changed := false
switch typed := value.(type) { switch typed := value.(type) {
@@ -155,10 +155,12 @@ func releaseIDs(raw json.RawMessage) (ocid, id string, err error) {
decoder := json.NewDecoder(bytes.NewReader(raw)) decoder := json.NewDecoder(bytes.NewReader(raw))
decoder.UseNumber() // a numeric id keeps its exact digits decoder.UseNumber() // a numeric id keeps its exact digits
if token, err := decoder.Token(); err != nil { token, err := decoder.Token()
if err != nil {
return "", "", fmt.Errorf("reading release: %w", err) return "", "", fmt.Errorf("reading release: %w", err)
} else if token != json.Delim('{') { }
return "", "", errors.New("reading release: not a JSON object") if token != json.Delim('{') {
return "", "", errors.New("findatender: reading release: not a JSON object")
} }
var haveOCID, haveID bool var haveOCID, haveID bool
@@ -205,7 +207,7 @@ func decodeReleaseID(decoder *json.Decoder) (string, error) {
case json.Number: case json.Number:
return typed.String(), nil return typed.String(), nil
default: default:
return "", fmt.Errorf("reading release id: unexpected %T", typed) return "", fmt.Errorf("findatender: reading release id: unexpected %T", typed)
} }
} }
@@ -213,11 +215,11 @@ func decodeReleaseID(decoder *json.Decoder) (string, error) {
func missingIDsError(haveOCID, haveID bool) error { func missingIDsError(haveOCID, haveID bool) error {
switch { switch {
case !haveOCID && !haveID: case !haveOCID && !haveID:
return errors.New("reading release: no ocid or id") return errors.New("findatender: reading release: no ocid or id")
case !haveOCID: case !haveOCID:
return errors.New("reading release: no ocid") return errors.New("findatender: reading release: no ocid")
default: default:
return errors.New("reading release: no id") return errors.New("findatender: reading release: no id")
} }
} }
@@ -227,8 +229,7 @@ func missingIDsError(haveOCID, haveID bool) error {
// //
// This method takes ownership of the response body and closes it when done. // This method takes ownership of the response body and closes it when done.
func safeUnpackReleases( func safeUnpackReleases(
logger *slog.Logger, logger *slog.Logger, response *resty.Response,
response *resty.Response,
) (*ocds.ReleasePackage, error) { ) (*ocds.ReleasePackage, error) {
var releasePackage safeReleasePackage var releasePackage safeReleasePackage
@@ -251,7 +252,8 @@ func safeUnpackReleases(
} }
// Correct PublishedDate of "" and "2006-01-02" // Correct PublishedDate of "" and "2006-01-02"
if releasePackage.PublishedDate != nil && len(*releasePackage.PublishedDate) > 0 { if releasePackage.PublishedDate != nil &&
*releasePackage.PublishedDate != "" {
published, err := time.Parse(time.RFC3339, *releasePackage.PublishedDate) published, err := time.Parse(time.RFC3339, *releasePackage.PublishedDate)
if err != nil { if err != nil {
published, err = time.Parse(time.DateOnly, *releasePackage.PublishedDate) published, err = time.Parse(time.DateOnly, *releasePackage.PublishedDate)
@@ -304,9 +306,9 @@ func closeBody(res *resty.Response) {
} }
} }
// fetchURI performs a single GET of uri, using findATender's context, and traces it at // fetchURI performs a single GET of uri, using findATender's context, and
// debug level. Unlike the method of the same name it does not check the // traces it at debug level. Unlike the method of the same name it does not
// response status. // check the response status.
func fetchURI(findATender *FindATender, uri string) (*resty.Response, error) { func fetchURI(findATender *FindATender, uri string) (*resty.Response, error) {
withTrace := findATender.config.Logging.LogLevel() <= slog.LevelDebug withTrace := findATender.config.Logging.LogLevel() <= slog.LevelDebug
logger := findATender.logger() logger := findATender.logger()
@@ -345,17 +347,21 @@ func logTrace(logger *slog.Logger, res *resty.Response) {
if logging.IsJSONHandler(logger.Handler()) { if logging.IsJSONHandler(logger.Handler()) {
trace = res.Request.TraceInfo().JSON() trace = res.Request.TraceInfo().JSON()
} }
logger.Debug("fetched", "trace", trace, "status", res.Status(), "response", res.String()) logger.Debug(
"fetched",
"trace", trace,
"status", res.Status(),
"response", res.String(),
)
} }
// fetchURI performs a fetch against the API. If the API returns with a rate limit // fetchURI performs a fetch against the API. If the API returns with a rate
// reached response and allowRateLimit is true, then wait and retry like a good // limit reached response and allowRateLimit is true, then wait and retry like a
// client. If allowRateLimit is false, return to the caller informing them of // good client. If allowRateLimit is false, return to the caller informing them
// the rate limit exceeded error. If, after waiting, the API still thinks we // of the rate limit exceeded error. If, after waiting, the API still thinks we
// have exceeded the rate limit, bail out anyway. // have exceeded the rate limit, bail out anyway.
func (ft *FindATender) fetchURI( func (ft *FindATender) fetchURI(
uri string, uri string, allowRateLimit bool,
allowRateLimit bool,
) (*resty.Response, error) { ) (*resty.Response, error) {
res, err := fetchURI(ft, uri) res, err := fetchURI(ft, uri)
if err != nil { if err != nil {
@@ -381,7 +387,10 @@ func (ft *FindATender) fetchURI(
} }
retryAfter := time.Duration(retryAfterSeconds) * time.Second retryAfter := time.Duration(retryAfterSeconds) * time.Second
ft.logger().Warn("api rate limit hit, waiting as asked", "retry_after", retryAfter) ft.logger().Warn(
"api rate limit hit, waiting as asked",
"retry_after", retryAfter,
)
if rateSleep(ctx, retryAfter) != nil { if rateSleep(ctx, retryAfter) != nil {
return nil, ErrFetchCancelled return nil, ErrFetchCancelled
} }
@@ -430,10 +439,9 @@ func (ft *FindATender) fetchWithOpts(
// ErrNoNext if there is no "Next" link; the caller can decide whether asking // ErrNoNext if there is no "Next" link; the caller can decide whether asking
// for it was an error. // for it was an error.
func (ft *FindATender) fetchNext( func (ft *FindATender) fetchNext(
releasePackage *ocds.ReleasePackage, releasePackage *ocds.ReleasePackage, allowRateLimit bool,
allowRateLimit bool,
) (*resty.Response, error) { ) (*resty.Response, error) {
if releasePackage.Links == nil || len(releasePackage.Links.Next) == 0 { if releasePackage.Links == nil || releasePackage.Links.Next == "" {
return nil, ErrNoNext return nil, ErrNoNext
} }
return ft.fetchURI(releasePackage.Links.Next, allowRateLimit) return ft.fetchURI(releasePackage.Links.Next, allowRateLimit)
@@ -478,16 +486,19 @@ func (ft *FindATender) fetchAll(opts RequestOptions) ([]ocds.Release, error) {
// one cannot loop forever. // one cannot loop forever.
seen := make(map[string]bool) seen := make(map[string]bool)
for { for {
if releasePackage.Links != nil && len(releasePackage.Links.Next) > 0 { if releasePackage.Links != nil && releasePackage.Links.Next != "" {
if seen[releasePackage.Links.Next] { if seen[releasePackage.Links.Next] {
ft.logger().Warn("stopping at repeated next link", "link", releasePackage.Links.Next) ft.logger().Warn(
"stopping at repeated next link",
"link", releasePackage.Links.Next,
)
break break
} }
seen[releasePackage.Links.Next] = true seen[releasePackage.Links.Next] = true
} }
res, err := ft.fetchNext(releasePackage, opts.AllowRateLimit) res, err := ft.fetchNext(releasePackage, opts.AllowRateLimit)
if err == ErrNoNext { if errors.Is(err, ErrNoNext) {
break break
} }
if err != nil { if err != nil {
+177 -72
View File
@@ -34,7 +34,7 @@ func testConfig(level string) *config.JobSearchConfig {
func quietLogs(t *testing.T, h slog.Handler) { func quietLogs(t *testing.T, h slog.Handler) {
t.Helper() t.Helper()
if h == nil { if h == nil {
h = slog.NewTextHandler(io.Discard, nil) h = slog.DiscardHandler
} }
prev := slog.Default() prev := slog.Default()
slog.SetDefault(slog.New(h)) slog.SetDefault(slog.New(h))
@@ -44,13 +44,18 @@ func quietLogs(t *testing.T, h slog.Handler) {
} }
// newTestFT starts h on a test server and returns a FindATender pointed at it. // newTestFT starts h on a test server and returns a FindATender pointed at it.
func newTestFT(t *testing.T, level string, h http.HandlerFunc) (*FindATender, *httptest.Server) { func newTestFT(
t *testing.T, level string, h http.HandlerFunc,
) (*FindATender, *httptest.Server) {
t.Helper() t.Helper()
quietLogs(t, nil) quietLogs(t, nil)
srv := httptest.NewServer(h) srv := httptest.NewServer(h)
t.Cleanup(srv.Close) t.Cleanup(srv.Close)
ft := New(testConfig(level), srv.URL) ft := New(testConfig(level), srv.URL)
t.Cleanup(ft.Shutdown) t.Cleanup(ft.Shutdown)
return ft, srv return ft, srv
} }
@@ -161,22 +166,40 @@ func TestFetchURIRateLimit(t *testing.T) {
}{ }{
{"not allowed", "0", false, ErrRateLimitExceeded, "", 1}, {"not allowed", "0", false, ErrRateLimitExceeded, "", 1},
{"allowed, retries", "0", true, nil, "", 2}, {"allowed, retries", "0", true, nil, "", 2},
{"allowed, bad retry-after", "soon", true, nil, "unknown retry-after value", 1}, {
{"allowed, missing retry-after", "", true, nil, "unknown retry-after value", 1}, "allowed, bad retry-after",
"soon",
true,
nil,
"unknown retry-after value",
1,
},
{
"allowed, missing retry-after",
"",
true,
nil,
"unknown retry-after value",
1,
},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
var hits atomic.Int32 var hits atomic.Int32
ft, srv := newTestFT(t, "warn", func(w http.ResponseWriter, r *http.Request) { ft, srv := newTestFT(
if hits.Add(1) == 1 { t,
if tt.retryAfter != "" { "warn",
w.Header().Set("Retry-After", tt.retryAfter) func(w http.ResponseWriter, r *http.Request) {
if hits.Add(1) == 1 {
if tt.retryAfter != "" {
w.Header().Set("Retry-After", tt.retryAfter)
}
w.WriteHeader(http.StatusTooManyRequests)
return
} }
w.WriteHeader(http.StatusTooManyRequests) io.WriteString(w, packageJSON("", releaseJSON("ocds-1", "1")))
return },
} )
io.WriteString(w, packageJSON("", releaseJSON("ocds-1", "1")))
})
res, err := ft.fetchURI(srv.URL+"/x", tt.allowRateLimit) res, err := ft.fetchURI(srv.URL+"/x", tt.allowRateLimit)
switch { switch {
@@ -217,7 +240,8 @@ func TestFetchURIRateLimitCancelled(t *testing.T) {
}() }()
start := time.Now() start := time.Now()
if _, err := withCtx.fetchURI(srv.URL, true); !errors.Is(err, ErrFetchCancelled) { _, err := withCtx.fetchURI(srv.URL, true)
if !errors.Is(err, ErrFetchCancelled) {
t.Fatalf("err = %v, want ErrFetchCancelled", err) t.Fatalf("err = %v, want ErrFetchCancelled", err)
} }
if time.Since(start) > 5*time.Second { if time.Since(start) > 5*time.Second {
@@ -233,7 +257,8 @@ func TestFetchURIRateLimitTwice(t *testing.T) {
w.Header().Set("Retry-After", "0") w.Header().Set("Retry-After", "0")
w.WriteHeader(http.StatusTooManyRequests) w.WriteHeader(http.StatusTooManyRequests)
}) })
if _, err := ft.fetchURI(srv.URL, true); !errors.Is(err, ErrRateLimitExceeded) { _, err := ft.fetchURI(srv.URL, true)
if !errors.Is(err, ErrRateLimitExceeded) {
t.Fatalf("err = %v, want ErrRateLimitExceeded", err) t.Fatalf("err = %v, want ErrRateLimitExceeded", err)
} }
if n := hits.Load(); n != 2 { if n := hits.Load(); n != 2 {
@@ -247,10 +272,14 @@ func TestFetchURIUnexpectedStatus(t *testing.T) {
http.StatusNotFound, http.StatusInternalServerError, http.StatusNotFound, http.StatusInternalServerError,
} { } {
t.Run(http.StatusText(status), func(t *testing.T) { t.Run(http.StatusText(status), func(t *testing.T) {
ft, srv := newTestFT(t, "warn", func(w http.ResponseWriter, r *http.Request) { ft, srv := newTestFT(
w.WriteHeader(status) t,
io.WriteString(w, packageJSON("", releaseJSON("ocds-1", "1"))) "warn",
}) func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(status)
io.WriteString(w, packageJSON("", releaseJSON("ocds-1", "1")))
},
)
_, err := ft.fetchURI(srv.URL, true) _, err := ft.fetchURI(srv.URL, true)
if !errors.Is(err, ErrUnexpectedStatus) { if !errors.Is(err, ErrUnexpectedStatus) {
t.Fatalf("err = %v, want ErrUnexpectedStatus", err) t.Fatalf("err = %v, want ErrUnexpectedStatus", err)
@@ -273,7 +302,8 @@ func TestFetchURIRateLimitThenError(t *testing.T) {
} }
w.WriteHeader(http.StatusServiceUnavailable) w.WriteHeader(http.StatusServiceUnavailable)
}) })
if _, err := ft.fetchURI(srv.URL, true); !errors.Is(err, ErrUnexpectedStatus) { _, err := ft.fetchURI(srv.URL, true)
if !errors.Is(err, ErrUnexpectedStatus) {
t.Fatalf("err = %v, want ErrUnexpectedStatus", err) t.Fatalf("err = %v, want ErrUnexpectedStatus", err)
} }
} }
@@ -314,9 +344,13 @@ func TestFetchURIDebugTrace(t *testing.T) {
} }
for name, mk := range handlers { for name, mk := range handlers {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
ft, _ := newTestFT(t, "debug", func(w http.ResponseWriter, r *http.Request) { ft, _ := newTestFT(
io.WriteString(w, packageJSON("", releaseJSON("ocds-1", "1"))) t,
}) "debug",
func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, packageJSON("", releaseJSON("ocds-1", "1")))
},
)
var logs strings.Builder var logs strings.Builder
quietLogs(t, mk(&logs)) quietLogs(t, mk(&logs))
@@ -343,8 +377,12 @@ func TestFetchURIDebugTrace(t *testing.T) {
} }
func TestFetchURIDebugTraceConnectionError(t *testing.T) { func TestFetchURIDebugTraceConnectionError(t *testing.T) {
ft, srv := newTestFT(t, "debug", func(w http.ResponseWriter, r *http.Request) { ft, srv := newTestFT(
}) t,
"debug",
func(w http.ResponseWriter, r *http.Request) {
},
)
srv.Close() srv.Close()
ft.httpClient.SetRetryCount(0) ft.httpClient.SetRetryCount(0)
if _, err := fetchURI(ft, srv.URL); err == nil { if _, err := fetchURI(ft, srv.URL); err == nil {
@@ -419,13 +457,18 @@ func TestSafeUnpackReleases(t *testing.T) {
{"ocid": "ocds-3", "id": "3", "date": "2026-10-08", "tag": [], "initiationType": "tender"} {"ocid": "ocds-3", "id": "3", "date": "2026-10-08", "tag": [], "initiationType": "tender"}
] ]
}` }`
pkg, err := safeUnpackReleases(slog.Default(), responseFor(t, body)) pkg, err := safeUnpackReleases(slog.Default(), responseFor(t, body))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if pkg.URI != "https://example/pkg" || pkg.Version != "1.1" || len(pkg.Extensions) != 1 ||
pkg.Publisher == nil || pkg.Publisher.Name != "FTS" || pkg.License != "OGL" || copied := pkg.URI == "https://example/pkg" && pkg.Version == "1.1" &&
pkg.PublicationPolicy != "policy" || pkg.Links == nil || pkg.Links.Next != "https://example/next" { len(pkg.Extensions) == 1 && pkg.Publisher != nil &&
pkg.Publisher.Name == "FTS" && pkg.License == "OGL" &&
pkg.PublicationPolicy == "policy" && pkg.Links != nil &&
pkg.Links.Next == "https://example/next"
if !copied {
t.Errorf("package fields not copied: %+v", pkg) t.Errorf("package fields not copied: %+v", pkg)
} }
if pkg.PublishedDate == nil || if pkg.PublishedDate == nil ||
@@ -434,7 +477,9 @@ func TestSafeUnpackReleases(t *testing.T) {
} }
if len(pkg.Releases) != 2 || pkg.Releases[0].OCID != "ocds-1" || if len(pkg.Releases) != 2 || pkg.Releases[0].OCID != "ocds-1" ||
pkg.Releases[1].OCID != "ocds-3" { pkg.Releases[1].OCID != "ocds-3" {
t.Errorf("releases = %+v, want ocds-1 and ocds-3 (bad one skipped)", pkg.Releases) t.Errorf(
"releases = %+v, want ocds-1 and ocds-3 (bad one skipped)", pkg.Releases,
)
} }
} }
@@ -446,8 +491,16 @@ func TestSafeUnpackReleasesPublishedDate(t *testing.T) {
}{ }{
{"omitted", "", nil}, {"omitted", "", nil},
{"empty string", `""`, nil}, {"empty string", `""`, nil},
{"RFC3339", `"2026-10-08T12:00:00Z"`, ptrTime(time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC))}, {
{"date only", `"2026-10-08"`, ptrTime(time.Date(2026, 10, 8, 0, 0, 0, 0, time.UTC))}, "RFC3339",
`"2026-10-08T12:00:00Z"`,
ptrTime(time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)),
},
{
"date only",
`"2026-10-08"`,
ptrTime(time.Date(2026, 10, 8, 0, 0, 0, 0, time.UTC)),
},
{"unparseable", `"yesterday"`, nil}, {"unparseable", `"yesterday"`, nil},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -463,7 +516,8 @@ func TestSafeUnpackReleasesPublishedDate(t *testing.T) {
switch { switch {
case tt.want == nil && pkg.PublishedDate != nil: case tt.want == nil && pkg.PublishedDate != nil:
t.Errorf("publishedDate = %v, want nil", pkg.PublishedDate) t.Errorf("publishedDate = %v, want nil", pkg.PublishedDate)
case tt.want != nil && (pkg.PublishedDate == nil || !pkg.PublishedDate.Equal(*tt.want)): case tt.want != nil && (pkg.PublishedDate == nil ||
!pkg.PublishedDate.Equal(*tt.want)):
t.Errorf("publishedDate = %v, want %v", pkg.PublishedDate, tt.want) t.Errorf("publishedDate = %v, want %v", pkg.PublishedDate, tt.want)
} }
}) })
@@ -480,7 +534,9 @@ func TestSafeUnpackReleasesInvalidJSON(t *testing.T) {
"publisher array": `{"publisher": []}`, "publisher array": `{"publisher": []}`,
} { } {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
if _, err := safeUnpackReleases(slog.Default(), responseFor(t, body)); err == nil { if _, err := safeUnpackReleases(
slog.Default(), responseFor(t, body),
); err == nil {
t.Error("expected a decode error") t.Error("expected a decode error")
} }
}) })
@@ -489,18 +545,23 @@ func TestSafeUnpackReleasesInvalidJSON(t *testing.T) {
// pagedServer serves pages[i] for ?page=i (page 0 at the release package // pagedServer serves pages[i] for ?page=i (page 0 at the release package
// path), linking each page to the next. // path), linking each page to the next.
func pagedServer(t *testing.T, pages ...[]string) (*FindATender, *atomic.Int32) { func pagedServer(
t *testing.T, pages ...[]string,
) (*FindATender, *atomic.Int32) {
t.Helper() t.Helper()
var hits atomic.Int32 var hits atomic.Int32
var srvURL string var srvURL string
ft, srv := newTestFT(t, "warn", func(w http.ResponseWriter, r *http.Request) { ft, srv := newTestFT(t, "warn", func(w http.ResponseWriter, r *http.Request) {
hits.Add(1) hits.Add(1)
i := 0 i := 0
fmt.Sscan(r.URL.Query().Get("page"), &i) fmt.Sscan(r.URL.Query().Get("page"), &i)
next := "" next := ""
if i+1 < len(pages) { if i+1 < len(pages) {
next = fmt.Sprintf("%s/ocdsReleasePackages?page=%d", srvURL, i+1) next = fmt.Sprintf("%s/ocdsReleasePackages?page=%d", srvURL, i+1)
} }
io.WriteString(w, packageJSON(next, pages[i]...)) io.WriteString(w, packageJSON(next, pages[i]...))
}) })
srvURL = srv.URL srvURL = srv.URL
@@ -513,12 +574,14 @@ func TestFetchOnce(t *testing.T) {
[]string{releaseJSON("ocds-1", "1"), releaseJSON("ocds-2", "1")}, []string{releaseJSON("ocds-1", "1"), releaseJSON("ocds-2", "1")},
[]string{releaseJSON("ocds-3", "1")}, []string{releaseJSON("ocds-3", "1")},
) )
releases, err := ft.fetchOnce(RequestOptions{ releases, err := ft.fetchOnce(RequestOptions{
Since: time.Now(), Since: time.Now(),
}) })
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if len(releases) != 2 { if len(releases) != 2 {
t.Errorf("got %d releases, want 2 (first page only)", len(releases)) t.Errorf("got %d releases, want 2 (first page only)", len(releases))
} }
@@ -551,10 +614,7 @@ func TestFetchOnceErrors(t *testing.T) {
RequestOptions{ RequestOptions{
Since: time.Now(), Since: time.Now(),
}, },
); !errors.Is( ); !errors.Is(err, ErrRateLimitExceeded) {
err,
ErrRateLimitExceeded,
) {
t.Errorf("rate limited: err = %v, want ErrRateLimitExceeded", err) t.Errorf("rate limited: err = %v, want ErrRateLimitExceeded", err)
} }
} }
@@ -566,12 +626,14 @@ func TestFetchAll(t *testing.T) {
[]string{}, []string{},
[]string{releaseJSON("ocds-1", "2")}, []string{releaseJSON("ocds-1", "2")},
) )
releases, err := ft.fetchAll(RequestOptions{ releases, err := ft.fetchAll(RequestOptions{
Since: time.Now(), Since: time.Now(),
}) })
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
var ids []string var ids []string
for _, r := range releases { for _, r := range releases {
ids = append(ids, r.OCID+"/"+r.ID) ids = append(ids, r.OCID+"/"+r.ID)
@@ -600,10 +662,7 @@ func TestFetchAllErrors(t *testing.T) {
RequestOptions{ RequestOptions{
Since: time.Now(), Since: time.Now(),
}, },
); !errors.Is( ); !errors.Is(err, ErrRateLimitExceeded) {
err,
ErrRateLimitExceeded,
) {
t.Errorf("first page rate limited: err = %v", err) t.Errorf("first page rate limited: err = %v", err)
} }
@@ -627,13 +686,19 @@ func TestFetchAllErrors(t *testing.T) {
} { } {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
var srvURL string var srvURL string
ft, srv := newTestFT(t, "warn", func(w http.ResponseWriter, r *http.Request) { ft, srv := newTestFT(
if r.URL.Query().Get("page") == "" { t,
io.WriteString(w, packageJSON(srvURL+"/next?page=1", releaseJSON("ocds-1", "1"))) "warn",
return func(w http.ResponseWriter, r *http.Request) {
} if r.URL.Query().Get("page") == "" {
fail(w, r) io.WriteString(
}) w, packageJSON(srvURL+"/next?page=1", releaseJSON("ocds-1", "1")),
)
return
}
fail(w, r)
},
)
srvURL = srv.URL srvURL = srv.URL
if _, err := ft.fetchAll(RequestOptions{ if _, err := ft.fetchAll(RequestOptions{
Since: time.Now(), Since: time.Now(),
@@ -663,17 +728,22 @@ func TestFetchAllRepeatedNextLink(t *testing.T) {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
var hits atomic.Int32 var hits atomic.Int32
var srvURL string var srvURL string
ft, srv := newTestFT(t, "error", func(w http.ResponseWriter, r *http.Request) { ft, srv := newTestFT(
if hits.Add(1) > 10 { t,
t.Error("fetchAll did not stop following next links") "error",
w.WriteHeader(http.StatusBadRequest) func(w http.ResponseWriter, r *http.Request) {
return if hits.Add(1) > 10 {
} t.Error("fetchAll did not stop following next links")
page := r.URL.Query().Get("page") w.WriteHeader(http.StatusBadRequest)
io.WriteString(w, packageJSON( return
nextFor(srvURL, page), releaseJSON("ocds-"+page, "1"), }
)) page := r.URL.Query().Get("page")
}) body := packageJSON(
nextFor(srvURL, page), releaseJSON("ocds-"+page, "1"),
)
io.WriteString(w, body)
},
)
srvURL = srv.URL srvURL = srv.URL
releases, err := ft.fetchAll(RequestOptions{ releases, err := ft.fetchAll(RequestOptions{
@@ -683,7 +753,11 @@ func TestFetchAllRepeatedNextLink(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
if len(releases) != int(hits.Load()) { if len(releases) != int(hits.Load()) {
t.Errorf("releases = %d, want one per request (%d)", len(releases), hits.Load()) t.Errorf(
"releases = %d, want one per request (%d)",
len(releases),
hits.Load(),
)
} }
}) })
} }
@@ -730,23 +804,28 @@ func TestUnmarshalLenientDates(t *testing.T) {
t.Errorf("date = %v", got.Date) t.Errorf("date = %v", got.Date)
} }
if got.PublishedDate != nil { if got.PublishedDate != nil {
t.Errorf("empty publishedDate should decode as nil, got %v", got.PublishedDate) t.Errorf(
"empty publishedDate should decode as nil, got %v", got.PublishedDate,
)
} }
wantStart := time.Date(2006, 10, 2, 15, 33, 2, 0, time.FixedZone("", 7*3600))
if start := got.Tender.TenderPeriod.StartDate; start == nil || if start := got.Tender.TenderPeriod.StartDate; start == nil ||
!start.Equal(time.Date(2006, 10, 2, 15, 33, 2, 0, time.FixedZone("", 7*3600))) { !start.Equal(wantStart) {
t.Errorf("full timestamp changed: %v", start) t.Errorf("full timestamp changed: %v", start)
} }
if end := got.Tender.TenderPeriod.EndDate; end == nil || !end.Equal(midnight(2006, 11, 30)) { if end := got.Tender.TenderPeriod.EndDate; end == nil ||
!end.Equal(midnight(2006, 11, 30)) {
t.Errorf("endDate = %v", end) t.Errorf("endDate = %v", end)
} }
if d := got.Tender.ExpressionOfInterestDeadline; d == nil || !d.Equal(midnight(2006, 10, 20)) { if d := got.Tender.ExpressionOfInterestDeadline; d == nil ||
!d.Equal(midnight(2006, 10, 20)) {
t.Errorf("expressionOfInterestDeadline = %v", d) t.Errorf("expressionOfInterestDeadline = %v", d)
} }
if got.Tender.Description != "Bids close on 2006-11-30" { if got.Tender.Description != "Bids close on 2006-11-30" {
t.Errorf("free text changed: %q", got.Tender.Description) t.Errorf("free text changed: %q", got.Tender.Description)
} }
if a := got.Awards[0]; a.Date == nil || !a.Date.Equal(midnight(2006, 12, 1)) || if a := got.Awards[0]; a.Date == nil ||
a.ID != "2006-12-01" { !a.Date.Equal(midnight(2006, 12, 1)) || a.ID != "2006-12-01" {
t.Errorf("award = %+v (date should be repaired, id left alone)", a) t.Errorf("award = %+v (date should be repaired, id left alone)", a)
} }
if got.Value != "1234567.891" { if got.Value != "1234567.891" {
@@ -829,9 +908,27 @@ func TestReleaseIDs(t *testing.T) {
{"missing ocid", `{"id":"r1"}`, "", "r1", "no ocid"}, {"missing ocid", `{"id":"r1"}`, "", "r1", "no ocid"},
{"missing id", `{"ocid":"ocds-1"}`, "ocds-1", "", "no id"}, {"missing id", `{"ocid":"ocds-1"}`, "ocds-1", "", "no id"},
{"ocid wrong type", `{"ocid":5,"id":"r1"}`, "", "", "reading release ocid"}, {"ocid wrong type", `{"ocid":5,"id":"r1"}`, "", "", "reading release ocid"},
{"id wrong type", `{"ocid":"ocds-1","id":true}`, "ocds-1", "", "unexpected bool"}, {
{"id is an object", `{"ocid":"ocds-1","id":{}}`, "ocds-1", "", "unexpected map"}, "id wrong type",
{"invalid id JSON", `{"ocid":"ocds-1","id":}`, "ocds-1", "", "reading release id"}, `{"ocid":"ocds-1","id":true}`,
"ocds-1",
"",
"unexpected bool",
},
{
"id is an object",
`{"ocid":"ocds-1","id":{}}`,
"ocds-1",
"",
"unexpected map",
},
{
"invalid id JSON",
`{"ocid":"ocds-1","id":}`,
"ocds-1",
"",
"reading release id",
},
{ {
"invalid field before ids", "invalid field before ids",
`{"date":nope,"ocid":"ocds-1","id":"r1"}`, `{"date":nope,"ocid":"ocds-1","id":"r1"}`,
@@ -839,7 +936,13 @@ func TestReleaseIDs(t *testing.T) {
"", "",
`reading release field "date"`, `reading release field "date"`,
}, },
{"truncated after ocid", `{"ocid":"ocds-1",`, "ocds-1", "", "reading release"}, {
"truncated after ocid",
`{"ocid":"ocds-1",`,
"ocds-1",
"",
"reading release",
},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
@@ -852,7 +955,9 @@ func TestReleaseIDs(t *testing.T) {
t.Fatalf("err = %v, want containing %q", err, tt.wantErr) t.Fatalf("err = %v, want containing %q", err, tt.wantErr)
} }
if ocid != tt.wantOCID || id != tt.wantID { if ocid != tt.wantOCID || id != tt.wantID {
t.Errorf("got (%q, %q), want (%q, %q)", ocid, id, tt.wantOCID, tt.wantID) t.Errorf(
"got (%q, %q), want (%q, %q)", ocid, id, tt.wantOCID, tt.wantID,
)
} }
}) })
} }
+8 -6
View File
@@ -34,7 +34,9 @@ func WithLogger(logger *slog.Logger) Option {
// New returns a client for the OCDS API at apiRoot. Requests time out after // New returns a client for the OCDS API at apiRoot. Requests time out after
// 60 seconds and are retried once. Call Shutdown when finished with it. It // 60 seconds and are retried once. Call Shutdown when finished with it. It
// logs to slog.Default() unless given WithLogger. // logs to slog.Default() unless given WithLogger.
func New(conf *config.JobSearchConfig, apiRoot string, opts ...Option) *FindATender { func New(
conf *config.JobSearchConfig, apiRoot string, opts ...Option,
) *FindATender {
findATender := &FindATender{ findATender := &FindATender{
config: conf, config: conf,
apiRoot: apiRoot, apiRoot: apiRoot,
@@ -42,9 +44,11 @@ func New(conf *config.JobSearchConfig, apiRoot string, opts ...Option) *FindATen
for _, opt := range opts { for _, opt := range opts {
opt(findATender) opt(findATender)
} }
findATender.httpClient = resty.New(). findATender.httpClient = resty.New().
SetTimeout(60 * time.Duration(time.Second)). SetTimeout(60 * time.Second).
SetRetryCount(1) SetRetryCount(1)
return findATender return findATender
} }
@@ -87,8 +91,7 @@ func (ft *FindATender) Shutdown() {
// converts them to opportunities. If allowRateLimit is true, a 429 response // converts them to opportunities. If allowRateLimit is true, a 429 response
// is waited out and retried; otherwise it fails with ErrRateLimitExceeded. // is waited out and retried; otherwise it fails with ErrRateLimitExceeded.
func (ft *FindATender) FetchOnce( func (ft *FindATender) FetchOnce(
since time.Time, since time.Time, allowRateLimit bool,
allowRateLimit bool,
) ([]*types.Opportunity, error) { ) ([]*types.Opportunity, error) {
releases, err := ft.fetchOnce(RequestOptions{ releases, err := ft.fetchOnce(RequestOptions{
Since: since, Since: since,
@@ -104,8 +107,7 @@ func (ft *FindATender) FetchOnce(
// each page's next link, and converts them to opportunities. allowRateLimit // each page's next link, and converts them to opportunities. allowRateLimit
// behaves as for FetchOnce. // behaves as for FetchOnce.
func (ft *FindATender) FetchAll( func (ft *FindATender) FetchAll(
since time.Time, since time.Time, allowRateLimit bool,
allowRateLimit bool,
) ([]*types.Opportunity, error) { ) ([]*types.Opportunity, error) {
releases, err := ft.fetchAll(RequestOptions{ releases, err := ft.fetchAll(RequestOptions{
Since: since, Since: since,
+15 -4
View File
@@ -24,8 +24,13 @@ func TestNew(t *testing.T) {
if ft.httpClient == nil { if ft.httpClient == nil {
t.Fatal("no http client") t.Fatal("no http client")
} }
if ft.httpClient.Timeout() != 60*time.Second || ft.httpClient.RetryCount() != 1 { if ft.httpClient.Timeout() != 60*time.Second ||
t.Errorf("client timeout %v, retries %d", ft.httpClient.Timeout(), ft.httpClient.RetryCount()) ft.httpClient.RetryCount() != 1 {
t.Errorf(
"client timeout %v, retries %d",
ft.httpClient.Timeout(),
ft.httpClient.RetryCount(),
)
} }
if ft.ctx != nil { if ft.ctx != nil {
t.Error("New should not set a context") t.Error("New should not set a context")
@@ -81,7 +86,11 @@ func opportunityIDs(ops []*types.Opportunity) []string {
func TestFetchOncePublic(t *testing.T) { func TestFetchOncePublic(t *testing.T) {
ft, _ := pagedServer( ft, _ := pagedServer(
t, t,
[]string{releaseJSON("ocds-1", "1"), releaseJSON("ocds-1", "2"), releaseJSON("ocds-2", "1")}, []string{
releaseJSON("ocds-1", "1"),
releaseJSON("ocds-1", "2"),
releaseJSON("ocds-2", "1"),
},
[]string{releaseJSON("ocds-3", "1")}, []string{releaseJSON("ocds-3", "1")},
) )
ops, err := ft.FetchOnce(time.Now(), false) ops, err := ft.FetchOnce(time.Now(), false)
@@ -108,7 +117,9 @@ func TestFetchAllPublic(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if got := strings.Join(opportunityIDs(ops), ","); got != "ocds-1,ocds-2,ocds-3" { if got := strings.Join(
opportunityIDs(ops), ",",
); got != "ocds-1,ocds-2,ocds-3" {
t.Errorf("opportunities = %s", got) t.Errorf("opportunities = %s", got)
} }
for _, op := range ops { for _, op := range ops {
+16 -13
View File
@@ -9,8 +9,8 @@ import (
"git.avsdev.uk/Go/job-search/internal/types" "git.avsdev.uk/Go/job-search/internal/types"
) )
// ocdsGetBuyerId returns the buyer's id, or "" if the release has no buyer. // ocdsGetBuyerID returns the buyer's id, or "" if the release has no buyer.
func ocdsGetBuyerId(release *ocds.Release) string { func ocdsGetBuyerID(release *ocds.Release) string {
if release.Buyer != nil { if release.Buyer != nil {
return string(release.Buyer.ID) return string(release.Buyer.ID)
} }
@@ -29,12 +29,12 @@ func ocdsGetBuyerName(release *ocds.Release) string {
// ocdsConvertClassification converts an OCDS classification, noting whether // ocdsConvertClassification converts an OCDS classification, noting whether
// it is a CPV code. // it is a CPV code.
func ocdsConvertClassification( func ocdsConvertClassification(
classif *ocds.Classification, classification *ocds.Classification,
) types.ClassificationCode { ) types.ClassificationCode {
return types.ClassificationCode{ return types.ClassificationCode{
IsCPV: strings.Compare(classif.Scheme, "CPV") == 0, IsCPV: strings.Compare(classification.Scheme, "CPV") == 0,
Code: string(classif.ID), Code: string(classification.ID),
Classification: classif.Description, Classification: classification.Description,
} }
} }
@@ -45,17 +45,19 @@ func ocdsGetClassifications(release *ocds.Release) []types.ClassificationCode {
if release.Tender == nil { if release.Tender == nil {
return classifications return classifications
} }
add := func(classif *ocds.Classification) {
newClassif := ocdsConvertClassification(classif) add := func(classification *ocds.Classification) {
converted := ocdsConvertClassification(classification)
found := slices.ContainsFunc( found := slices.ContainsFunc(
classifications, func(code types.ClassificationCode) bool { classifications, func(code types.ClassificationCode) bool {
return code.Code == newClassif.Code return code.Code == converted.Code
}, },
) )
if !found { if !found {
classifications = append(classifications, newClassif) classifications = append(classifications, converted)
} }
} }
if release.Tender.Classification != nil { if release.Tender.Classification != nil {
add(release.Tender.Classification) add(release.Tender.Classification)
} }
@@ -63,10 +65,11 @@ func ocdsGetClassifications(release *ocds.Release) []types.ClassificationCode {
if item.Classification != nil { if item.Classification != nil {
add(item.Classification) add(item.Classification)
} }
for _, classif := range item.AdditionalClassifications { for _, classification := range item.AdditionalClassifications {
add(&classif) add(&classification)
} }
} }
return classifications return classifications
} }
@@ -142,7 +145,7 @@ func convertOne(release *ocds.Release) *types.Opportunity {
opportunity := &types.Opportunity{ opportunity := &types.Opportunity{
UUID: release.OCID, UUID: release.OCID,
UUIDType: "ocid", UUIDType: "ocid",
BuyerId: ocdsGetBuyerId(release), BuyerID: ocdsGetBuyerID(release),
BuyerName: ocdsGetBuyerName(release), BuyerName: ocdsGetBuyerName(release),
Source: "FindATender", Source: "FindATender",
} }
+127 -43
View File
@@ -38,7 +38,7 @@ func TestOcdsGetBuyer(t *testing.T) {
r := &ocds.Release{ r := &ocds.Release{
Buyer: tt.buyer, Buyer: tt.buyer,
} }
if got := ocdsGetBuyerId(r); got != tt.wantID { if got := ocdsGetBuyerID(r); got != tt.wantID {
t.Errorf("buyer id = %q, want %q", got, tt.wantID) t.Errorf("buyer id = %q, want %q", got, tt.wantID)
} }
if got := ocdsGetBuyerName(r); got != tt.wantName { if got := ocdsGetBuyerName(r); got != tt.wantName {
@@ -101,6 +101,7 @@ func TestOcdsGetClassifications(t *testing.T) {
Description: "d" + id, Description: "d" + id,
} }
} }
codes := func(cs []types.ClassificationCode) []string { codes := func(cs []types.ClassificationCode) []string {
out := []string{} out := []string{}
for _, c := range cs { for _, c := range cs {
@@ -108,6 +109,7 @@ func TestOcdsGetClassifications(t *testing.T) {
} }
return out return out
} }
main := cpv("1") main := cpv("1")
tests := []struct { tests := []struct {
name string name string
@@ -122,13 +124,24 @@ func TestOcdsGetClassifications(t *testing.T) {
{"main and item additionals, deduplicated", &ocds.Tender{ {"main and item additionals, deduplicated", &ocds.Tender{
Classification: &main, Classification: &main,
Items: []ocds.Item{ Items: []ocds.Item{
{ID: "a", AdditionalClassifications: []ocds.Classification{cpv("2"), cpv("1")}}, {
{ID: "b"}, ID: "a",
{ID: "c", AdditionalClassifications: []ocds.Classification{cpv("3"), cpv("2")}}, AdditionalClassifications: []ocds.Classification{cpv("2"), cpv("1")},
},
{
ID: "b",
},
{
ID: "c",
AdditionalClassifications: []ocds.Classification{cpv("3"), cpv("2")},
},
}, },
}, []string{"1", "2", "3"}}, }, []string{"1", "2", "3"}},
{"item additionals without main", &ocds.Tender{ {"item additionals without main", &ocds.Tender{
Items: []ocds.Item{{ID: "a", AdditionalClassifications: []ocds.Classification{cpv("5")}}}, Items: []ocds.Item{{
ID: "a",
AdditionalClassifications: []ocds.Classification{cpv("5")},
}},
}, []string{"5"}}, }, []string{"5"}},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -151,9 +164,15 @@ func TestOcdsGetClassificationsIncludesItemClassification(t *testing.T) {
Scheme: "CPV", Scheme: "CPV",
ID: "9", ID: "9",
} }
got := ocdsGetClassifications(&ocds.Release{Tender: &ocds.Tender{ release := &ocds.Release{
Items: []ocds.Item{{ID: "a", Classification: &item}}, Tender: &ocds.Tender{
}}) Items: []ocds.Item{{
ID: "a",
Classification: &item,
}},
},
}
got := ocdsGetClassifications(release)
if len(got) != 1 || got[0].Code != "9" { if len(got) != 1 || got[0].Code != "9" {
t.Errorf("got %+v, want the item classification 9", got) t.Errorf("got %+v, want the item classification 9", got)
} }
@@ -231,12 +250,21 @@ func TestOcdsRelease(t *testing.T) {
} }
got := ocdsRelease(r) got := ocdsRelease(r)
if got.UUID != "ocds-x-1" || got.ReleaseID != "001" || !got.ReleaseDate.Equal(date) { if got.UUID != "ocds-x-1" || got.ReleaseID != "001" ||
!got.ReleaseDate.Equal(date) {
t.Errorf("release = %+v", got) t.Errorf("release = %+v", got)
} }
want := []types.OpportunityReleaseTag{ want := []types.OpportunityReleaseTag{
{UUID: "ocds-x-1", ReleaseID: "001", Tag: "planning"}, {
{UUID: "ocds-x-1", ReleaseID: "001", Tag: "tender"}, UUID: "ocds-x-1",
ReleaseID: "001",
Tag: "planning",
},
{
UUID: "ocds-x-1",
ReleaseID: "001",
Tag: "tender",
},
} }
if !reflect.DeepEqual(got.Tags, want) { if !reflect.DeepEqual(got.Tags, want) {
t.Errorf("tags = %+v, want %+v", got.Tags, want) t.Errorf("tags = %+v, want %+v", got.Tags, want)
@@ -250,7 +278,8 @@ func TestOcdsRelease(t *testing.T) {
t.Errorf("round tripped release = %+v", back) t.Errorf("round tripped release = %+v", back)
} }
if empty := ocdsRelease(&ocds.Release{}); empty.Tags == nil || len(empty.Tags) != 0 { if empty := ocdsRelease(&ocds.Release{}); empty.Tags == nil ||
len(empty.Tags) != 0 {
t.Errorf("no tags should give an empty, non-nil slice, got %#v", empty.Tags) t.Errorf("no tags should give an empty, non-nil slice, got %#v", empty.Tags)
} }
} }
@@ -289,7 +318,9 @@ func TestConvertOne(t *testing.T) {
Value: &ocds.Value{ Value: &ocds.Value{
Amount: &amount, Amount: &amount,
}, },
Documents: []ocds.Document{{ID: "doc"}}, Documents: []ocds.Document{{
ID: "doc",
}},
}, },
} }
@@ -300,7 +331,7 @@ func TestConvertOne(t *testing.T) {
}{ }{
{"UUID", got.UUID, "ocds-x-1"}, {"UUID", got.UUID, "ocds-x-1"},
{"UUIDType", got.UUIDType, "ocid"}, {"UUIDType", got.UUIDType, "ocid"},
{"BuyerId", got.BuyerId, "GB-1"}, {"BuyerID", got.BuyerID, "GB-1"},
{"BuyerName", got.BuyerName, "Council"}, {"BuyerName", got.BuyerName, "Council"},
{"Title", got.Title, "Website"}, {"Title", got.Title, "Website"},
{"Status", got.Status, "active"}, {"Status", got.Status, "active"},
@@ -344,10 +375,11 @@ func TestConvertOneMinimal(t *testing.T) {
ID: "1", ID: "1",
Tender: tt.tender, Tender: tt.tender,
}) })
if got.UUID != "ocds-x-2" || got.BuyerId != "" || got.BuyerName != "" { if got.UUID != "ocds-x-2" || got.BuyerID != "" || got.BuyerName != "" {
t.Errorf("got %+v", got) t.Errorf("got %+v", got)
} }
if got.GrossValue != 0 || got.StartDate != nil || got.EndDate != nil || got.HasDocuments { if got.GrossValue != 0 || got.StartDate != nil || got.EndDate != nil ||
got.HasDocuments {
t.Errorf("unexpected tender fields: %+v", got) t.Errorf("unexpected tender fields: %+v", got)
} }
if len(got.Releases) != 1 { if len(got.Releases) != 1 {
@@ -366,18 +398,46 @@ func TestConvert(t *testing.T) {
return time.Date(2026, 1, d, 0, 0, 0, 0, time.UTC) return time.Date(2026, 1, d, 0, 0, 0, 0, time.UTC)
} }
releases := []ocds.Release{ releases := []ocds.Release{
{OCID: "ocds-b", ID: "1", Date: day(1), Tender: &ocds.Tender{ {
Title: "B", OCID: "ocds-b",
}}, ID: "1",
{OCID: "ocds-a", ID: "2", Date: day(3), Tender: &ocds.Tender{ Date: day(1),
Title: "A latest", Lots: []ocds.Lot{{ID: "1"}, {ID: "2"}}, Tender: &ocds.Tender{
}}, Title: "B",
{OCID: "ocds-a", ID: "1", Date: day(2), Tender: &ocds.Tender{ },
Title: "A first", },
}}, {
{OCID: "ocds-a", ID: "3", Date: day(1), Tender: &ocds.Tender{ OCID: "ocds-a",
Title: "A oldest", ID: "2",
}}, Date: day(3),
Tender: &ocds.Tender{
Title: "A latest",
Lots: []ocds.Lot{
{
ID: "1",
},
{
ID: "2",
},
},
},
},
{
OCID: "ocds-a",
ID: "1",
Date: day(2),
Tender: &ocds.Tender{
Title: "A first",
},
},
{
OCID: "ocds-a",
ID: "3",
Date: day(1),
Tender: &ocds.Tender{
Title: "A oldest",
},
},
} }
got := Convert(releases) got := Convert(releases)
@@ -395,7 +455,9 @@ func TestConvert(t *testing.T) {
t.Errorf("ocds-a title = %q, want %q", a.Title, "A latest") t.Errorf("ocds-a title = %q, want %q", a.Title, "A latest")
} }
if !a.LatestReleaseDate.Equal(day(3)) { if !a.LatestReleaseDate.Equal(day(3)) {
t.Errorf("ocds-a LatestReleaseDate = %v, want %v", a.LatestReleaseDate, day(3)) t.Errorf(
"ocds-a LatestReleaseDate = %v, want %v", a.LatestReleaseDate, day(3),
)
} }
if !a.MultiLot { if !a.MultiLot {
t.Error("ocds-a MultiLot = false, want true for two lots") t.Error("ocds-a MultiLot = false, want true for two lots")
@@ -408,7 +470,8 @@ func TestConvert(t *testing.T) {
t.Errorf("ocds-a release ids = %v, want %v", ids, want) t.Errorf("ocds-a release ids = %v, want %v", ids, want)
} }
if b.Title != "B" || b.MultiLot || !b.LatestReleaseDate.Equal(day(1)) || len(b.Releases) != 1 { if b.Title != "B" || b.MultiLot || !b.LatestReleaseDate.Equal(day(1)) ||
len(b.Releases) != 1 {
t.Errorf("ocds-b = %+v", b) t.Errorf("ocds-b = %+v", b)
} }
} }
@@ -417,12 +480,22 @@ func TestConvert(t *testing.T) {
func TestConvertSameDate(t *testing.T) { func TestConvertSameDate(t *testing.T) {
date := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) date := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
got := Convert([]ocds.Release{ got := Convert([]ocds.Release{
{OCID: "ocds-a", ID: "1", Date: date, Tender: &ocds.Tender{ {
Title: "first", OCID: "ocds-a",
}}, ID: "1",
{OCID: "ocds-a", ID: "2", Date: date, Tender: &ocds.Tender{ Date: date,
Title: "second", Tender: &ocds.Tender{
}}, Title: "first",
},
},
{
OCID: "ocds-a",
ID: "2",
Date: date,
Tender: &ocds.Tender{
Title: "second",
},
},
}) })
if len(got) != 1 || got[0].Title != "second" { if len(got) != 1 || got[0].Title != "second" {
t.Fatalf("got %+v, want one opportunity titled %q", got, "second") t.Fatalf("got %+v, want one opportunity titled %q", got, "second")
@@ -446,13 +519,24 @@ func TestConvertSetsSource(t *testing.T) {
} }
for _, o := range Convert([]ocds.Release{ for _, o := range Convert([]ocds.Release{
{OCID: "ocds-a", ID: "1", Tender: &ocds.Tender{ {
Title: "A", OCID: "ocds-a",
}}, ID: "1",
{OCID: "ocds-a", ID: "2", Tender: &ocds.Tender{ Tender: &ocds.Tender{
Title: "A amended", Title: "A",
}}, },
{OCID: "ocds-b", ID: "1"}, },
{
OCID: "ocds-a",
ID: "2",
Tender: &ocds.Tender{
Title: "A amended",
},
},
{
OCID: "ocds-b",
ID: "1",
},
}) { }) {
if o.Source != want { if o.Source != want {
t.Errorf("Convert: %s Source = %q, want %q", o.UUID, o.Source, want) t.Errorf("Convert: %s Source = %q, want %q", o.UUID, o.Source, want)
+6 -2
View File
@@ -20,7 +20,8 @@ func TestWriteProblem(t *testing.T) {
if ct := rec.Header().Get("Content-Type"); ct != "application/problem+json" { if ct := rec.Header().Get("Content-Type"); ct != "application/problem+json" {
t.Errorf("Content-Type = %q", ct) t.Errorf("Content-Type = %q", ct)
} }
if nosniff := rec.Header().Get("X-Content-Type-Options"); nosniff != "nosniff" { nosniff := rec.Header().Get("X-Content-Type-Options")
if nosniff != "nosniff" {
t.Errorf("X-Content-Type-Options = %q", nosniff) t.Errorf("X-Content-Type-Options = %q", nosniff)
} }
var p Problem var p Problem
@@ -42,7 +43,10 @@ func TestWriteProblem(t *testing.T) {
// Without a request (or a request ID) the problem has no requestId, and an // Without a request (or a request ID) the problem has no requestId, and an
// empty detail is left out. // empty detail is left out.
func TestWriteProblemMinimal(t *testing.T) { func TestWriteProblemMinimal(t *testing.T) {
for _, req := range []*http.Request{nil, httptest.NewRequest("GET", "/", nil)} { for _, req := range []*http.Request{
nil,
httptest.NewRequest("GET", "/", nil),
} {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
WriteProblem(rec, req, http.StatusTooManyRequests, "") WriteProblem(rec, req, http.StatusTooManyRequests, "")
var body map[string]any var body map[string]any
+4 -1
View File
@@ -17,7 +17,9 @@ type Problem struct {
} }
// WriteProblem writes a problem+json error response. // WriteProblem writes a problem+json error response.
func WriteProblem(w http.ResponseWriter, r *http.Request, status int, detail string) { func WriteProblem(
w http.ResponseWriter, r *http.Request, status int, detail string,
) {
problem := Problem{ problem := Problem{
Type: "about:blank", Type: "about:blank",
Title: http.StatusText(status), Title: http.StatusText(status),
@@ -27,6 +29,7 @@ func WriteProblem(w http.ResponseWriter, r *http.Request, status int, detail str
if r != nil { if r != nil {
problem.RequestID = RequestID(r.Context()) problem.RequestID = RequestID(r.Context())
} }
w.Header().Set("Content-Type", "application/problem+json") w.Header().Set("Content-Type", "application/problem+json")
w.Header().Set("X-Content-Type-Options", "nosniff") w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(status) w.WriteHeader(status)
+23 -19
View File
@@ -19,19 +19,19 @@ type statusRecorder struct {
bytes int bytes int
} }
func (r *statusRecorder) WriteHeader(code int) { func (s *statusRecorder) WriteHeader(code int) {
if r.status == 0 { if s.status == 0 {
r.status = code s.status = code
} }
r.ResponseWriter.WriteHeader(code) s.ResponseWriter.WriteHeader(code)
} }
func (r *statusRecorder) Write(data []byte) (int, error) { func (s *statusRecorder) Write(data []byte) (int, error) {
if r.status == 0 { if s.status == 0 {
r.status = http.StatusOK s.status = http.StatusOK
} }
written, err := r.ResponseWriter.Write(data) written, err := s.ResponseWriter.Write(data)
r.bytes += written s.bytes += written
return written, err return written, err
} }
@@ -42,6 +42,7 @@ func logField(value string) string {
if value == "" { if value == "" {
return "-" return "-"
} }
var builder strings.Builder var builder strings.Builder
for i := 0; i < len(value); i++ { for i := 0; i < len(value); i++ {
char := value[i] char := value[i]
@@ -51,16 +52,17 @@ func logField(value string) string {
builder.WriteByte(char) builder.WriteByte(char)
} }
} }
return builder.String() return builder.String()
} }
// clientKey attempts to produce a client host name as a key identifier to split // clientKey attempts to produce a client host name as a key identifier to split
// requests in the log between sources. Could be extended to also check // requests in the log between sources. Could be extended to also check
// Forwarded-For or other proxy headers // Forwarded-For or other proxy headers
func clientKey(req *http.Request) string { func clientKey(r *http.Request) string {
host, _, err := net.SplitHostPort(req.RemoteAddr) host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil { if err != nil {
return req.RemoteAddr return r.RemoteAddr
} }
return host return host
} }
@@ -68,19 +70,21 @@ func clientKey(req *http.Request) string {
// AccessLog wraps next, writing a line to logger for each request in the // AccessLog wraps next, writing a line to logger for each request in the
// Apache/nginx Common Log Format, or the Combined Log Format (which adds the // Apache/nginx Common Log Format, or the Combined Log Format (which adds the
// referer and user agent) if combined is true. // referer and user agent) if combined is true.
func AccessLog(logger *log.Logger, combined bool, next http.Handler) http.Handler { func AccessLog(
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { logger *log.Logger, combined bool, next http.Handler,
) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now() start := time.Now()
recorder := &statusRecorder{ recorder := &statusRecorder{
ResponseWriter: w, ResponseWriter: w,
} }
next.ServeHTTP(recorder, req) next.ServeHTTP(recorder, r)
if recorder.status == 0 { if recorder.status == 0 {
recorder.status = http.StatusOK recorder.status = http.StatusOK
} }
user := "-" user := "-"
if username, _, ok := req.BasicAuth(); ok && username != "" { if username, _, ok := r.BasicAuth(); ok && username != "" {
user = logField(username) user = logField(username)
} }
size := "-" // CLF uses "-" for an empty body size := "-" // CLF uses "-" for an empty body
@@ -89,16 +93,16 @@ func AccessLog(logger *log.Logger, combined bool, next http.Handler) http.Handle
} }
line := fmt.Sprintf( line := fmt.Sprintf(
`%s - %s [%s] "%s" %d %s`, `%s - %s [%s] "%s" %d %s`,
clientKey(req), clientKey(r),
user, user,
start.Format("02/Jan/2006:15:04:05 -0700"), start.Format("02/Jan/2006:15:04:05 -0700"),
logField(req.Method+" "+req.RequestURI+" "+req.Proto), logField(r.Method+" "+r.RequestURI+" "+r.Proto),
recorder.status, recorder.status,
size, size,
) )
if combined { if combined {
line += fmt.Sprintf( line += fmt.Sprintf(
` "%s" "%s"`, logField(req.Referer()), logField(req.UserAgent()), ` "%s" "%s"`, logField(r.Referer()), logField(r.UserAgent()),
) )
} }
logger.Println(line) logger.Println(line)
+1
View File
@@ -65,6 +65,7 @@ func TestStatusRecorder(t *testing.T) {
rec := &statusRecorder{ rec := &statusRecorder{
ResponseWriter: inner, ResponseWriter: inner,
} }
rec.Write([]byte("hello")) rec.Write([]byte("hello"))
rec.Write([]byte(" world")) rec.Write([]byte(" world"))
if rec.status != http.StatusOK || rec.bytes != 11 { if rec.status != http.StatusOK || rec.bytes != 11 {
+16 -6
View File
@@ -29,7 +29,7 @@ func saveGlobals(t *testing.T) {
func TestLevels(t *testing.T) { func TestLevels(t *testing.T) {
saveGlobals(t) saveGlobals(t)
slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, nil))) slog.SetDefault(slog.New(slog.DiscardHandler))
SetLevel(slog.LevelInfo) SetLevel(slog.LevelInfo)
if IsDebugEnabled() { if IsDebugEnabled() {
@@ -57,6 +57,13 @@ func TestLevels(t *testing.T) {
type wrapper struct{ slog.Handler } type wrapper struct{ slog.Handler }
func TestIsJSONHandler(t *testing.T) { func TestIsJSONHandler(t *testing.T) {
// The handlers' types are what is tested, so slog.DiscardHandler won't do.
//nolint:sloglint // see above
jsonWithAttrs := slog.NewJSONHandler(io.Discard, nil).WithAttrs(
[]slog.Attr{slog.String("k", "v")},
)
//nolint:sloglint // as above
tests := []struct { tests := []struct {
name string name string
h slog.Handler h slog.Handler
@@ -66,9 +73,7 @@ func TestIsJSONHandler(t *testing.T) {
{"text", slog.NewTextHandler(io.Discard, nil), false}, {"text", slog.NewTextHandler(io.Discard, nil), false},
{"tint", tint.NewTextHandler(io.Discard, nil), false}, {"tint", tint.NewTextHandler(io.Discard, nil), false},
{"nil", nil, false}, {"nil", nil, false},
{"json with attrs", slog.NewJSONHandler(io.Discard, nil).WithAttrs( {"json with attrs", jsonWithAttrs, true},
[]slog.Attr{slog.String("k", "v")},
), true},
// Only slog's own JSON handler is recognised, not one wrapped in another. // Only slog's own JSON handler is recognised, not one wrapped in another.
{"wrapped json", wrapper{ {"wrapped json", wrapper{
Handler: slog.NewJSONHandler(io.Discard, nil), Handler: slog.NewJSONHandler(io.Discard, nil),
@@ -108,7 +113,8 @@ func TestLogToFileWritesJSON(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
s := string(data) s := string(data)
if !strings.Contains(s, `"msg":"hello file"`) || !strings.Contains(s, `"k":"v"`) { if !strings.Contains(s, `"msg":"hello file"`) ||
!strings.Contains(s, `"k":"v"`) {
t.Fatalf("log file missing entry: %s", s) t.Fatalf("log file missing entry: %s", s)
} }
if strings.Contains(s, "hidden") { if strings.Contains(s, "hidden") {
@@ -118,7 +124,9 @@ func TestLogToFileWritesJSON(t *testing.T) {
func TestLogToFileBadPath(t *testing.T) { func TestLogToFileBadPath(t *testing.T) {
saveGlobals(t) saveGlobals(t)
if err := LogToFile(filepath.Join(t.TempDir(), "missing", "app.log")); err == nil { if err := LogToFile(
filepath.Join(t.TempDir(), "missing", "app.log"),
); err == nil {
t.Fatal("expected an error for a path in a missing directory") t.Fatal("expected an error for a path in a missing directory")
} }
} }
@@ -131,6 +139,7 @@ func TestLogToFileTracksFile(t *testing.T) {
if !IsFileLogger() { if !IsFileLogger() {
t.Fatal("IsFileLogger() = false after LogToFile") t.Fatal("IsFileLogger() = false after LogToFile")
} }
if err := Shutdown(); err != nil { if err := Shutdown(); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -155,6 +164,7 @@ func TestLogToTerminal(t *testing.T) {
if IsJSONHandler(h) { if IsJSONHandler(h) {
t.Fatal("terminal handler reported as JSON") t.Fatal("terminal handler reported as JSON")
} }
SetLevel(slog.LevelWarn) SetLevel(slog.LevelWarn)
if h.Enabled(context.Background(), slog.LevelInfo) { if h.Enabled(context.Background(), slog.LevelInfo) {
t.Fatal("terminal handler ignores the package level") t.Fatal("terminal handler ignores the package level")
+11 -9
View File
@@ -53,7 +53,9 @@ func RateLimit(limiter Limiter, key KeyFunc) func(http.Handler) http.Handler {
seconds = 1 seconds = 1
} }
w.Header().Set("Retry-After", strconv.Itoa(seconds)) w.Header().Set("Retry-After", strconv.Itoa(seconds))
httpx.WriteProblem(w, r, http.StatusTooManyRequests, "rate limit exceeded") httpx.WriteProblem(
w, r, http.StatusTooManyRequests, "rate limit exceeded",
)
return return
} }
next.ServeHTTP(w, r) next.ServeHTTP(w, r)
@@ -85,10 +87,7 @@ type limiterEntry struct {
// requests per second with the given burst. A janitor goroutine evicts // requests per second with the given burst. A janitor goroutine evicts
// entries idle for longer than ttl until ctx is cancelled. // entries idle for longer than ttl until ctx is cancelled.
func NewKeyedLimiter( func NewKeyedLimiter(
ctx context.Context, ctx context.Context, requestsPerSecond float64, burst int, ttl time.Duration,
requestsPerSecond float64,
burst int,
ttl time.Duration,
) *KeyedLimiter { ) *KeyedLimiter {
limiter := &KeyedLimiter{ limiter := &KeyedLimiter{
limit: rate.Limit(requestsPerSecond), limit: rate.Limit(requestsPerSecond),
@@ -99,9 +98,11 @@ func NewKeyedLimiter(
for i := range limiter.shards { for i := range limiter.shards {
limiter.shards[i].entries = make(map[string]*limiterEntry) limiter.shards[i].entries = make(map[string]*limiterEntry)
} }
if ttl > 0 { if ttl > 0 {
go limiter.janitor(ctx) go limiter.janitor(ctx)
} }
return limiter return limiter
} }
@@ -125,14 +126,15 @@ func (k *KeyedLimiter) Allow(key string) (bool, time.Duration) {
limiter := entry.limiter limiter := entry.limiter
shard.mu.Unlock() shard.mu.Unlock()
res := limiter.ReserveN(now, 1) reservation := limiter.ReserveN(now, 1)
if !res.OK() { if !reservation.OK() {
return false, time.Second return false, time.Second
} }
if delay := res.DelayFrom(now); delay > 0 { if delay := reservation.DelayFrom(now); delay > 0 {
res.CancelAt(now) reservation.CancelAt(now)
return false, delay return false, delay
} }
return true, 0 return true, 0
} }
+7 -3
View File
@@ -8,17 +8,21 @@ import (
func TestRateLimit(t *testing.T) { func TestRateLimit(t *testing.T) {
l := NewKeyedLimiter(t.Context(), 1, 2, 0) l := NewKeyedLimiter(t.Context(), 1, 2, 0)
h := RateLimit(l, nil)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { next := http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
})) })
h := RateLimit(l, nil)(next)
var last int var last int
for i := 0; i < 4; i++ { for i := 0; i < 4; i++ {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "/", nil)) h.ServeHTTP(rec, httptest.NewRequest("GET", "/", nil))
last = rec.Code last = rec.Code
if last == http.StatusTooManyRequests && rec.Header().Get("Retry-After") == "" { if last == http.StatusTooManyRequests &&
rec.Header().Get("Retry-After") == "" {
t.Fatal("missing Retry-After") t.Fatal("missing Retry-After")
} }
} }
if last != http.StatusTooManyRequests { if last != http.StatusTooManyRequests {
t.Fatalf("want 429, got %d", last) t.Fatalf("want 429, got %d", last)
} }
+2 -1
View File
@@ -116,7 +116,8 @@ func RealIP(opts RealIPOptions) func(http.Handler) http.Handler {
break break
} }
} }
if proto := forwardedProto(r.Header); proto == "http" || proto == "https" { proto := forwardedProto(r.Header)
if proto == "http" || proto == "https" {
info.Scheme = proto info.Scheme = proto
} }
} else if opts.StripUntrusted { } else if opts.StripUntrusted {
+21 -7
View File
@@ -9,18 +9,23 @@ import (
// serveRealIP runs a request from peer with the given headers through RealIP // serveRealIP runs a request from peer with the given headers through RealIP
// and returns the ClientInfo the next handler sees. // and returns the ClientInfo the next handler sees.
func serveRealIP(t *testing.T, opts RealIPOptions, peer string, h map[string]string) ClientInfo { func serveRealIP(
t *testing.T, opts RealIPOptions, peer string, h map[string]string,
) ClientInfo {
t.Helper() t.Helper()
var got ClientInfo var got ClientInfo
next := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { next := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
got = Client(r) got = Client(r)
}) })
req := httptest.NewRequest("GET", "/", nil) req := httptest.NewRequest("GET", "/", nil)
req.RemoteAddr = peer req.RemoteAddr = peer
for k, v := range h { for k, v := range h {
req.Header.Set(k, v) req.Header.Set(k, v)
} }
RealIP(opts)(next).ServeHTTP(httptest.NewRecorder(), req) RealIP(opts)(next).ServeHTTP(httptest.NewRecorder(), req)
return got return got
} }
@@ -32,15 +37,19 @@ func TestRealIP(t *testing.T) {
Headers: defaultHeaders, Headers: defaultHeaders,
} }
// Spoofed left-most entry is ignored; right-most untrusted wins. // Spoofed left-most entry is ignored; right-most untrusted wins.
c := serveRealIP(t, opts, "127.0.0.1:1234", map[string]string{ headers := map[string]string{
"X-Forwarded-For": "6.6.6.6, 203.0.113.9, 127.0.0.1", "X-Forwarded-Proto": "https", "X-Forwarded-For": "6.6.6.6, 203.0.113.9, 127.0.0.1",
}) "X-Forwarded-Proto": "https",
}
c := serveRealIP(t, opts, "127.0.0.1:1234", headers)
if c.IP.String() != "203.0.113.9" || c.Scheme != "https" || !c.ViaProxy { if c.IP.String() != "203.0.113.9" || c.Scheme != "https" || !c.ViaProxy {
t.Fatalf("xff: %+v", c) t.Fatalf("xff: %+v", c)
} }
c = serveRealIP(t, opts, "127.0.0.1:1234", map[string]string{
headers = map[string]string{
"Forwarded": `for="[2001:db8::7]:4711";proto=https`, "Forwarded": `for="[2001:db8::7]:4711";proto=https`,
}) }
c = serveRealIP(t, opts, "127.0.0.1:1234", headers)
if c.IP.String() != "2001:db8::7" { if c.IP.String() != "2001:db8::7" {
t.Fatalf("forwarded: %+v", c) t.Fatalf("forwarded: %+v", c)
} }
@@ -51,7 +60,12 @@ func TestUntrustedPeerHeadersIgnored(t *testing.T) {
Trusted: []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")}, Trusted: []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")},
Headers: defaultHeaders, Headers: defaultHeaders,
} }
c := serveRealIP(t, opts, "127.0.0.1:1234", map[string]string{"X-Forwarded-For": "203.0.113.9"}) c := serveRealIP(
t,
opts,
"127.0.0.1:1234",
map[string]string{"X-Forwarded-For": "203.0.113.9"},
)
if c.IP.String() != "127.0.0.1" || c.ViaProxy { if c.IP.String() != "127.0.0.1" || c.ViaProxy {
t.Fatalf("untrusted XFF honoured: %+v", c) t.Fatalf("untrusted XFF honoured: %+v", c)
} }
+26 -32
View File
@@ -79,33 +79,19 @@ func parseCatalog(data []byte, name string) (*Catalog, error) {
func (c *Catalog) validate() error { func (c *Catalog) validate() error {
var errs []error var errs []error
for _, pool := range []struct { pools := []struct {
name string name string
n int count int
}{ }{
{ {name: "buyers", count: len(c.Buyers)},
name: "buyers", {name: "suppliers", count: len(c.Suppliers)},
n: len(c.Buyers), {name: "subjects", count: len(c.Subjects)},
}, {name: "methods", count: len(c.Methods)},
{ {name: "classifications", count: len(c.Classifications)},
name: "suppliers", }
n: len(c.Suppliers), for _, pool := range pools {
}, if pool.count == 0 {
{ errs = append(errs, fmt.Errorf("mock: no %s", pool.name))
name: "subjects",
n: len(c.Subjects),
},
{
name: "methods",
n: len(c.Methods),
},
{
name: "classifications",
n: len(c.Classifications),
},
} {
if pool.n == 0 {
errs = append(errs, fmt.Errorf("no %s", pool.name))
} }
} }
@@ -136,8 +122,9 @@ func (c *Catalog) validate() error {
keys = append(keys[:5], "...") keys = append(keys[:5], "...")
} }
errs = append(errs, fmt.Errorf( errs = append(errs, fmt.Errorf(
"%d classification key(s) not found in classifications: %s", "mock: %d classification key(s) not found in classifications: %s",
len(missing), strings.Join(keys, ", "), len(missing),
strings.Join(keys, ", "),
)) ))
} }
@@ -164,13 +151,19 @@ func (c *Catalog) subject(rng *rand.Rand) ocds.Tender {
subject := clone(c.Subjects[rng.Intn(len(c.Subjects))]) subject := clone(c.Subjects[rng.Intn(len(c.Subjects))])
tender := subject.Tender tender := subject.Tender
tender.Classification = c.resolveOne(subject.Classification) tender.Classification = c.resolveOne(subject.Classification)
tender.AdditionalClassifications = c.resolve(subject.AdditionalClassifications) tender.AdditionalClassifications = c.resolve(
subject.AdditionalClassifications,
)
tender.Items = nil tender.Items = nil
for _, subjectItem := range subject.Items { for _, subjectItem := range subject.Items {
item := subjectItem.Item item := subjectItem.Item
item.AdditionalClassifications = c.resolve(subjectItem.AdditionalClassifications) item.AdditionalClassifications = c.resolve(
subjectItem.AdditionalClassifications,
)
tender.Items = append(tender.Items, item) tender.Items = append(tender.Items, item)
} }
return tender return tender
} }
@@ -195,12 +188,13 @@ func (c *Catalog) resolve(keys []string) []ocds.Classification {
return out return out
} }
// clone deep-copies a value through JSON, which is all the generated types need. // clone deep-copies a value through JSON, which is all the generated types
// need.
func clone[T any](value T) T { func clone[T any](value T) T {
var out T var out T
data, err := json.Marshal(value) data, err := json.Marshal(value)
if err == nil { if err == nil {
json.Unmarshal(data, &out) //nolint:errcheck,gosec // b was just marshalled from a T json.Unmarshal(data, &out) //nolint:errcheck,gosec // data was just marshalled from a T
} }
return out return out
} }
+12 -7
View File
@@ -31,8 +31,8 @@ func TestEmbeddedCatalogIsValid(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if len(cat.Buyers) == 0 || len(cat.Suppliers) == 0 || len(cat.Subjects) == 0 || if len(cat.Buyers) == 0 || len(cat.Suppliers) == 0 ||
len(cat.Methods) == 0 { len(cat.Subjects) == 0 || len(cat.Methods) == 0 {
t.Fatalf( t.Fatalf(
"embedded catalog has an empty pool: %d buyers, %d suppliers, %d subjects, %d methods", "embedded catalog has an empty pool: %d buyers, %d suppliers, %d subjects, %d methods",
len(cat.Buyers), len(cat.Buyers),
@@ -45,8 +45,8 @@ func TestEmbeddedCatalogIsValid(t *testing.T) {
func TestParseCatalog(t *testing.T) { func TestParseCatalog(t *testing.T) {
cat := loadTestCatalog(t) cat := loadTestCatalog(t)
if len(cat.Buyers) != 2 || len(cat.Suppliers) != 1 || len(cat.Subjects) != 2 || if len(cat.Buyers) != 2 || len(cat.Suppliers) != 1 ||
len(cat.Methods) != 2 { len(cat.Subjects) != 2 || len(cat.Methods) != 2 {
t.Fatalf( t.Fatalf(
"got %d buyers, %d suppliers, %d subjects, %d methods", "got %d buyers, %d suppliers, %d subjects, %d methods",
len(cat.Buyers), len(cat.Buyers),
@@ -55,18 +55,22 @@ func TestParseCatalog(t *testing.T) {
len(cat.Methods), len(cat.Methods),
) )
} }
if d := cat.Buyers[0].Details; d == nil || len(d.Classifications) != 1 || if d := cat.Buyers[0].Details; d == nil || len(d.Classifications) != 1 ||
d.Classifications[0].Scheme != "UK_CA_TYPE" { d.Classifications[0].Scheme != "UK_CA_TYPE" {
t.Fatalf("buyer details.classifications not read: %+v", d) t.Fatalf("buyer details.classifications not read: %+v", d)
} }
s := cat.Subjects[0] s := cat.Subjects[0]
if s.Classification != "CPV-77314000" || len(s.Items) != 1 { if s.Classification != "CPV-77314000" || len(s.Items) != 1 {
t.Fatalf("subject classification key/items not read: %+v", s) t.Fatalf("subject classification key/items not read: %+v", s)
} }
if ac := s.Items[0].AdditionalClassifications; len(ac) != 1 || ac[0] != "CPV-77310000" { if ac := s.Items[0].AdditionalClassifications; len(ac) != 1 ||
ac[0] != "CPV-77310000" {
t.Fatalf("items.additionalClassifications keys not read: %+v", ac) t.Fatalf("items.additionalClassifications keys not read: %+v", ac)
} }
if v := s.Lots[0].Value; v == nil || v.AmountGross == nil || *v.AmountGross != 120000 { if v := s.Lots[0].Value; v == nil || v.AmountGross == nil ||
*v.AmountGross != 120000 {
t.Fatalf("lot value not read: %+v", v) t.Fatalf("lot value not read: %+v", v)
} }
} }
@@ -86,7 +90,8 @@ func TestSubjectResolvesClassifications(t *testing.T) {
t.Fatalf("main classification = %+v", cl) t.Fatalf("main classification = %+v", cl)
} }
ac := tender.Items[0].AdditionalClassifications ac := tender.Items[0].AdditionalClassifications
if len(ac) != 1 || ac[0].Description != "Planting and maintenance services of green areas" { if len(ac) != 1 ||
ac[0].Description != "Planting and maintenance services of green areas" {
t.Fatalf("item classifications = %+v", ac) t.Fatalf("item classifications = %+v", ac)
} }
// The catalog itself still holds keys, untouched by picking. // The catalog itself still holds keys, untouched by picking.
+60 -53
View File
@@ -17,61 +17,63 @@ const ocidPrefix = "ocds-mock-"
// moves an existing procedure on (planning -> tender -> award) so the same ocid // moves an existing procedure on (planning -> tender -> award) so the same ocid
// appears in several releases, as it does on the real service. // appears in several releases, as it does on the real service.
type Generator struct { type Generator struct {
cat *Catalog catalog *Catalog
store *Store store *Store
rnd *rand.Rand rng *rand.Rand
opts options opts options
} }
// NewGenerator returns a generator drawing from catalog and storing releases in // NewGenerator returns a generator drawing from catalog and storing releases in
// store. The same seed and store contents give the same releases. It logs to // store. The same seed and store contents give the same releases. It logs to
// slog.Default() unless given WithLogger. // slog.Default() unless given WithLogger.
func NewGenerator(catalog *Catalog, store *Store, seed int64, opts ...Option) *Generator { func NewGenerator(
catalog *Catalog, store *Store, seed int64, opts ...Option,
) *Generator {
return &Generator{ return &Generator{
cat: catalog, catalog: catalog,
store: store, store: store,
rnd: rand.New(rand.NewSource(seed)), //nolint:gosec // seeded for repeatable mock data rng: rand.New(rand.NewSource(seed)), //nolint:gosec // seeded for repeatable mock data
opts: applyOptions(opts), opts: applyOptions(opts),
} }
} }
// Make returns a new release dated at (to the second). About a third of the // Make returns a new release with the given date (to the second). About a third
// time it moves on a procedure still open from the last 60 days; otherwise it // of the time it moves on a procedure still open from the last 60 days;
// starts a new one. // otherwise it starts a new one.
func (g *Generator) Make(at time.Time) ocds.Release { func (g *Generator) Make(date time.Time) ocds.Release {
at = at.UTC().Truncate(time.Second) date = date.UTC().Truncate(time.Second)
if g.rnd.Float64() < 0.35 { if g.rng.Float64() < 0.35 {
if prev, ok := g.store.RandomOpen(g.rnd, at.AddDate(0, 0, -60)); ok { if previous, ok := g.store.RandomOpen(g.rng, date.AddDate(0, 0, -60)); ok {
return g.advance(prev, at) return g.advance(previous, date)
} }
} }
return g.newProcedure(at) return g.newProcedure(date)
} }
// newProcedure returns the first release of a new procedure: a tender (or, // newProcedure returns the first release of a new procedure: a tender (or,
// one time in five, a planning notice) from a random catalog buyer and // one time in five, a planning notice) from a random catalog buyer and
// subject, using a random procurement method. Values are scaled by a random // subject, using a random procurement method. Values are scaled by a random
// factor, and a tender without lots gets a single lot covering it all. // factor, and a tender without lots gets a single lot covering it all.
func (g *Generator) newProcedure(at time.Time) ocds.Release { func (g *Generator) newProcedure(date time.Time) ocds.Release {
buyer := g.cat.buyer(g.rnd) buyer := g.catalog.buyer(g.rng)
ensurePartyID(&buyer) ensurePartyID(&buyer)
buyer.Roles = []string{"buyer"} buyer.Roles = []string{"buyer"}
method := g.cat.method(g.rnd) method := g.catalog.method(g.rng)
tag, status := "tender", "active" tag, status := "tender", "active"
if g.rnd.Float64() < 0.2 { if g.rng.Float64() < 0.2 {
tag, status = "planning", "planned" tag, status = "planning", "planned"
} }
ocid := g.newOCID() ocid := g.newOCID()
tender := g.cat.subject(g.rnd) tender := g.catalog.subject(g.rng)
tender.ID = ocds.ID(ocid) tender.ID = ocds.ID(ocid)
tender.ProcurementMethod = method.Method tender.ProcurementMethod = method.Method
tender.ProcurementMethodDetails = method.Details tender.ProcurementMethodDetails = method.Details
factor := 0.5 + g.rnd.Float64()*1.5 factor := 0.5 + g.rng.Float64()*1.5
if tender.Value == nil { if tender.Value == nil {
amount := float64((g.rnd.Intn(5000) + 10) * 1000) amount := float64((g.rng.Intn(5000) + 10) * 1000)
tender.Value = &ocds.Value{ tender.Value = &ocds.Value{
Amount: &amount, Amount: &amount,
Currency: "GBP", Currency: "GBP",
@@ -96,14 +98,14 @@ func (g *Generator) newProcedure(at time.Time) ocds.Release {
setStatus(&tender, status) setStatus(&tender, status)
if status == "active" { if status == "active" {
tender.TenderPeriod = &ocds.Period{ tender.TenderPeriod = &ocds.Period{
EndDate: ptr(at.AddDate(0, 0, 30+g.rnd.Intn(30))), EndDate: ptr(date.AddDate(0, 0, 30+g.rng.Intn(30))),
} }
} }
return ocds.Release{ return ocds.Release{
OCID: ocid, OCID: ocid,
ID: g.store.NextNoticeID(at.Year()), ID: g.store.NextNoticeID(date.Year()),
Date: at, Date: date,
Tag: []string{tag}, Tag: []string{tag},
InitiationType: "tender", InitiationType: "tender",
Parties: []ocds.Organization{buyer}, Parties: []ocds.Organization{buyer},
@@ -116,13 +118,15 @@ func (g *Generator) newProcedure(at time.Time) ocds.Release {
} }
} }
// advance returns the next release in prev's procedure: planning moves to // advance returns the next release in previous's procedure: planning moves to
// tender, and a tender either has its deadline extended (tenderUpdate) or is // tender, and a tender either has its deadline extended (tenderUpdate) or is
// awarded to a supplier from the catalog. // awarded to a supplier from the catalog.
func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release { func (g *Generator) advance(
release := clone(prev) previous ocds.Release, date time.Time,
release.ID = g.store.NextNoticeID(at.Year()) ) ocds.Release {
release.Date = at release := clone(previous)
release.ID = g.store.NextNoticeID(date.Year())
release.Date = date
release.Awards = nil release.Awards = nil
if release.Tender == nil { if release.Tender == nil {
release.Tender = &ocds.Tender{ release.Tender = &ocds.Tender{
@@ -131,11 +135,11 @@ func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release {
} }
next := "award" next := "award"
switch prev.Tag[0] { switch previous.Tag[0] {
case "planning": case "planning":
next = "tender" next = "tender"
case "tender": case "tender":
if g.rnd.Float64() < 0.25 { if g.rng.Float64() < 0.25 {
next = "tenderUpdate" next = "tenderUpdate"
} }
} }
@@ -145,12 +149,13 @@ func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release {
case "tender": case "tender":
setStatus(release.Tender, "active") setStatus(release.Tender, "active")
release.Tender.TenderPeriod = &ocds.Period{ release.Tender.TenderPeriod = &ocds.Period{
EndDate: ptr(at.AddDate(0, 0, 30+g.rnd.Intn(30))), EndDate: ptr(date.AddDate(0, 0, 30+g.rng.Intn(30))),
} }
case "tenderUpdate": case "tenderUpdate":
// A typical update: the deadline is extended by two weeks. // A typical update: the deadline is extended by two weeks.
end := at.AddDate(0, 0, 30) end := date.AddDate(0, 0, 30)
if period := release.Tender.TenderPeriod; period != nil && period.EndDate != nil { if period := release.Tender.TenderPeriod; period != nil &&
period.EndDate != nil {
end = *period.EndDate end = *period.EndDate
} }
release.Tender.TenderPeriod = &ocds.Period{ release.Tender.TenderPeriod = &ocds.Period{
@@ -158,7 +163,7 @@ func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release {
} }
case "award": case "award":
setStatus(release.Tender, "complete") setStatus(release.Tender, "complete")
supplier := g.cat.supplier(g.rnd) supplier := g.catalog.supplier(g.rng)
ensurePartyID(&supplier) ensurePartyID(&supplier)
supplier.Roles = []string{"supplier"} supplier.Roles = []string{"supplier"}
release.Parties = append(release.Parties, supplier) release.Parties = append(release.Parties, supplier)
@@ -166,7 +171,7 @@ func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release {
var value *ocds.Value var value *ocds.Value
if release.Tender.Value != nil { if release.Tender.Value != nil {
scaled := clone(*release.Tender.Value) scaled := clone(*release.Tender.Value)
scaleValue(&scaled, 0.8+g.rnd.Float64()*0.2) scaleValue(&scaled, 0.8+g.rng.Float64()*0.2)
value = &scaled value = &scaled
} }
@@ -180,7 +185,7 @@ func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release {
release.Awards = []ocds.Award{{ release.Awards = []ocds.Award{{
ID: "1", ID: "1",
Status: "active", Status: "active",
Date: ptr(at), Date: ptr(date),
Value: value, Value: value,
Suppliers: []ocds.OrganizationReference{ Suppliers: []ocds.OrganizationReference{
{ {
@@ -198,26 +203,26 @@ func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release {
// newOCID returns a random ocid not already in the store. // newOCID returns a random ocid not already in the store.
func (g *Generator) newOCID() string { func (g *Generator) newOCID() string {
for { for {
id := fmt.Sprintf("%s%06x", ocidPrefix, g.rnd.Intn(1<<24)) id := fmt.Sprintf("%s%06x", ocidPrefix, g.rng.Intn(1<<24))
if !g.store.HasOCID(id) { if !g.store.HasOCID(id) {
return id return id
} }
} }
} }
// gap returns a random duration from min to max inclusive, or min if max is // gap returns a random duration from minGap to maxGap inclusive, or minGap if
// not larger. // maxGap is not larger.
func (g *Generator) gap(min, max time.Duration) time.Duration { func (g *Generator) gap(minGap, maxGap time.Duration) time.Duration {
if max <= min { if maxGap <= minGap {
return min return minGap
} }
return min + time.Duration(g.rnd.Int63n(int64(max-min)+1)) return minGap + time.Duration(g.rng.Int63n(int64(maxGap-minGap)+1))
} }
// CatchUp fills the period since the last stored release (or the last backfill // CatchUp fills the period since the last stored release (or the last backfill
// period, whichever is shorter) so restarts leave no gap in the timeline. // period, whichever is shorter) so restarts leave no gap in the timeline.
func (g *Generator) CatchUp( func (g *Generator) CatchUp(
now time.Time, backfill, min, max time.Duration, now time.Time, backfill, minGap, maxGap time.Duration,
) (int, error) { ) (int, error) {
start := now.Add(-backfill) start := now.Add(-backfill)
if last, ok := g.store.Last(); ok && last.After(start) { if last, ok := g.store.Last(); ok && last.After(start) {
@@ -225,7 +230,8 @@ func (g *Generator) CatchUp(
} }
count := 0 count := 0
for when := start.Add(g.gap(min, max)); !when.After(now); when = when.Add(g.gap(min, max)) { when := start.Add(g.gap(minGap, maxGap))
for ; !when.After(now); when = when.Add(g.gap(minGap, maxGap)) {
if err := g.store.AppendNoSync(g.Make(when)); err != nil { if err := g.store.AppendNoSync(g.Make(when)); err != nil {
return count, err return count, err
} }
@@ -235,11 +241,12 @@ func (g *Generator) CatchUp(
return count, g.store.Sync() return count, g.store.Sync()
} }
// Run publishes a new release after a random gap between min and max until ctx ends. // Run publishes a new release after a random gap between minGap and maxGap
func (g *Generator) Run(ctx context.Context, min, max time.Duration) { // until ctx ends.
func (g *Generator) Run(ctx context.Context, minGap, maxGap time.Duration) {
logger := g.opts.logger("generator") logger := g.opts.logger("generator")
for { for {
gap := g.gap(min, max) gap := g.gap(minGap, maxGap)
logger.Info("next release", "in", gap.Round(time.Second)) logger.Info("next release", "in", gap.Round(time.Second))
timer := time.NewTimer(gap) timer := time.NewTimer(gap)
+27 -22
View File
@@ -2,8 +2,7 @@ package mock
import ( import (
"context" "context"
"io" "log/slog"
"log"
"math/rand" "math/rand"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -13,19 +12,13 @@ import (
"git.avsdev.uk/Go/job-search/internal/mock/findatender/ocds" "git.avsdev.uk/Go/job-search/internal/mock/findatender/ocds"
) )
func quietLog(t *testing.T) { // quiet discards the logging of the Store and Generator it is given to.
t.Helper() var quiet = WithLogger(slog.New(slog.DiscardHandler))
w := log.Writer()
log.SetOutput(io.Discard)
t.Cleanup(func() {
log.SetOutput(w)
})
}
func TestGap(t *testing.T) { func TestGap(t *testing.T) {
//nolint:gosec // G404: seeded math/rand keeps the test deterministic //nolint:gosec // G404: seeded math/rand keeps the test deterministic
g := &Generator{ g := &Generator{
rnd: rand.New(rand.NewSource(1)), rng: rand.New(rand.NewSource(1)),
} }
if d := g.gap(time.Second, time.Second); d != time.Second { if d := g.gap(time.Second, time.Second); d != time.Second {
t.Errorf("min == max: %s", d) t.Errorf("min == max: %s", d)
@@ -34,7 +27,8 @@ func TestGap(t *testing.T) {
t.Errorf("max < min: %s, want min", d) t.Errorf("max < min: %s, want min", d)
} }
for i := 0; i < 1000; i++ { for i := 0; i < 1000; i++ {
if d := g.gap(time.Second, 2*time.Second); d < time.Second || d > 2*time.Second { if d := g.gap(time.Second, 2*time.Second); d < time.Second ||
d > 2*time.Second {
t.Fatalf("gap out of range: %s", d) t.Fatalf("gap out of range: %s", d)
} }
} }
@@ -135,7 +129,14 @@ func TestAdvance(t *testing.T) {
Amount: ptr(10000.0), Amount: ptr(10000.0),
Currency: "GBP", Currency: "GBP",
}, },
Lots: []ocds.Lot{{ID: "1"}, {ID: ""}}, Lots: []ocds.Lot{
{
ID: "1",
},
{
ID: "",
},
},
TenderPeriod: &ocds.Period{ TenderPeriod: &ocds.Period{
EndDate: &end, EndDate: &end,
}, },
@@ -147,7 +148,8 @@ func TestAdvance(t *testing.T) {
seen[r.Tag[0]] = true seen[r.Tag[0]] = true
switch r.Tag[0] { switch r.Tag[0] {
case "tenderUpdate": case "tenderUpdate":
if got := *r.Tender.TenderPeriod.EndDate; !got.Equal(end.AddDate(0, 0, 14)) { got := *r.Tender.TenderPeriod.EndDate
if !got.Equal(end.AddDate(0, 0, 14)) {
t.Fatalf("updated deadline %s", got) t.Fatalf("updated deadline %s", got)
} }
case "award": case "award":
@@ -155,13 +157,15 @@ func TestAdvance(t *testing.T) {
t.Fatalf("award: status %s, %d awards", r.Tender.Status, len(r.Awards)) t.Fatalf("award: status %s, %d awards", r.Tender.Status, len(r.Awards))
} }
aw := r.Awards[0] aw := r.Awards[0]
if aw.Value == nil || *aw.Value.Amount < 8000 || *aw.Value.Amount > 10000 { if aw.Value == nil || *aw.Value.Amount < 8000 ||
*aw.Value.Amount > 10000 {
t.Fatalf("award value %+v", aw.Value) t.Fatalf("award value %+v", aw.Value)
} }
if len(aw.RelatedLots) != 1 || aw.RelatedLots[0] != "1" { if len(aw.RelatedLots) != 1 || aw.RelatedLots[0] != "1" {
t.Fatalf("related lots %v", aw.RelatedLots) t.Fatalf("related lots %v", aw.RelatedLots)
} }
if len(aw.Suppliers) != 1 || len(r.Parties) != 1 || r.Parties[0].Roles[0] != "supplier" { if len(aw.Suppliers) != 1 || len(r.Parties) != 1 ||
r.Parties[0].Roles[0] != "supplier" {
t.Fatalf("suppliers %v, parties %v", aw.Suppliers, r.Parties) t.Fatalf("suppliers %v, parties %v", aw.Suppliers, r.Parties)
} }
default: default:
@@ -188,7 +192,8 @@ func TestAdvance(t *testing.T) {
r := g.advance(prev, at) r := g.advance(prev, at)
switch r.Tag[0] { switch r.Tag[0] {
case "tenderUpdate": case "tenderUpdate":
if got := *r.Tender.TenderPeriod.EndDate; !got.Equal(at.AddDate(0, 0, 44)) { got := *r.Tender.TenderPeriod.EndDate
if !got.Equal(at.AddDate(0, 0, 44)) {
t.Fatalf("default deadline %s", got) t.Fatalf("default deadline %s", got)
} }
case "award": case "award":
@@ -200,9 +205,8 @@ func TestAdvance(t *testing.T) {
} }
func TestRunPublishesUntilCancelled(t *testing.T) { func TestRunPublishesUntilCancelled(t *testing.T) {
quietLog(t)
cat := loadTestCatalog(t) cat := loadTestCatalog(t)
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl")) store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"), quiet)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -210,8 +214,9 @@ func TestRunPublishesUntilCancelled(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{}) done := make(chan struct{})
generator := NewGenerator(cat, store, 1, quiet)
go func() { go func() {
NewGenerator(cat, store, 1).Run(ctx, time.Millisecond, 2*time.Millisecond) generator.Run(ctx, time.Millisecond, 2*time.Millisecond)
close(done) close(done)
}() }()
@@ -234,7 +239,6 @@ func TestRunPublishesUntilCancelled(t *testing.T) {
// A failing store is logged and Run carries on until cancelled. // A failing store is logged and Run carries on until cancelled.
func TestRunSurvivesStoreErrors(t *testing.T) { func TestRunSurvivesStoreErrors(t *testing.T) {
quietLog(t)
cat := loadTestCatalog(t) cat := loadTestCatalog(t)
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl")) store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
if err != nil { if err != nil {
@@ -244,7 +248,8 @@ func TestRunSurvivesStoreErrors(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel() defer cancel()
NewGenerator(cat, store, 1).Run(ctx, time.Millisecond, time.Millisecond) generator := NewGenerator(cat, store, 1, quiet)
generator.Run(ctx, time.Millisecond, time.Millisecond)
if store.Len() != 0 { if store.Len() != 0 {
t.Errorf("Len = %d, want 0", store.Len()) t.Errorf("Len = %d, want 0", store.Len())
} }
+28 -24
View File
@@ -31,8 +31,10 @@ const (
type server struct { type server struct {
store *Store store *Store
baseURL string baseURL string
loc *time.Location // zone for updatedFrom/updatedTo without an offset; nil means UTC // location is the zone for updatedFrom/updatedTo without an offset; nil
opts options // means UTC.
location *time.Location
opts options
} }
// NewHandler serves release packages from store. baseURL is the public base // NewHandler serves release packages from store. baseURL is the public base
@@ -42,16 +44,13 @@ type server struct {
// Stored release dates are UTC; comparisons are on absolute instants. It // Stored release dates are UTC; comparisons are on absolute instants. It
// logs to slog.Default() unless given WithLogger. // logs to slog.Default() unless given WithLogger.
func NewHandler( func NewHandler(
store *Store, store *Store, baseURL string, location *time.Location, opts ...Option,
baseURL string,
location *time.Location,
opts ...Option,
) http.HandlerFunc { ) http.HandlerFunc {
handler := &server{ handler := &server{
store: store, store: store,
baseURL: baseURL, baseURL: baseURL,
loc: location, location: location,
opts: applyOptions(opts), opts: applyOptions(opts),
} }
return handler.handle return handler.handle
} }
@@ -86,8 +85,8 @@ func badRequest(w http.ResponseWriter, msg string) {
} }
// parseTime reads a query timestamp. Values without an offset (the documented // parseTime reads a query timestamp. Values without an offset (the documented
// yyyy-MM-ddTHH:mm:ss, or a plain date) are local time in location; values with an // yyyy-MM-ddTHH:mm:ss, or a plain date) are local time in location; values with
// offset or Z (RFC 3339) mean exactly what they say. // an offset or Z (RFC 3339) mean exactly what they say.
func parseTime(value string, location *time.Location) (time.Time, error) { func parseTime(value string, location *time.Location) (time.Time, error) {
if location == nil { if location == nil {
location = time.UTC location = time.UTC
@@ -98,7 +97,8 @@ func parseTime(value string, location *time.Location) (time.Time, error) {
} }
for _, layout := range []string{timeLayout, "2006-01-02"} { for _, layout := range []string{timeLayout, "2006-01-02"} {
if parsed, err := time.ParseInLocation(layout, value, location); err == nil { parsed, err := time.ParseInLocation(layout, value, location)
if err == nil {
return parsed, nil return parsed, nil
} }
} }
@@ -115,22 +115,24 @@ type pageQuery struct {
// handle answers a GET for a page of releases as an OCDS release package, // handle answers a GET for a page of releases as an OCDS release package,
// with a links.next when more releases match. // with a links.next when more releases match.
func (s *server) handle(w http.ResponseWriter, req *http.Request) { func (s *server) handle(w http.ResponseWriter, r *http.Request) {
if req.Method != http.MethodGet { if r.Method != http.MethodGet {
w.WriteHeader(http.StatusMethodNotAllowed) w.WriteHeader(http.StatusMethodNotAllowed)
return return
} }
query := req.URL.Query() query := r.URL.Query()
page, err := s.parseQuery(query) page, err := s.parseQuery(query)
if err != nil { if err != nil {
badRequest(w, err.Error()) badRequest(w, err.Error())
return return
} }
releases, next, more := s.store.Page(page.from, page.upper, page.start, page.limit) releases, next, more := s.store.Page(
page.from, page.upper, page.start, page.limit,
)
releasePackage := ocds.ReleasePackage{ releasePackage := ocds.ReleasePackage{
URI: s.baseURL + req.URL.RequestURI(), URI: s.baseURL + r.URL.RequestURI(),
Version: "1.1", Version: "1.1",
Extensions: ocds.DeclaredExtensions, Extensions: ocds.DeclaredExtensions,
PublishedDate: ptr(time.Now().UTC().Truncate(time.Second)), PublishedDate: ptr(time.Now().UTC().Truncate(time.Second)),
@@ -167,13 +169,15 @@ func (s *server) parseQuery(query url.Values) (pageQuery, error) {
if value := query.Get("limit"); value != "" { if value := query.Get("limit"); value != "" {
number, err := strconv.Atoi(value) number, err := strconv.Atoi(value)
if err != nil || number < 1 || number > maxLimit { if err != nil || number < 1 || number > maxLimit {
return page, fmt.Errorf("limit must be an integer between 1 and %d", maxLimit) return page, fmt.Errorf(
"limit must be an integer between 1 and %d", maxLimit,
)
} }
page.limit = number page.limit = number
} }
if value := query.Get("updatedFrom"); value != "" { if value := query.Get("updatedFrom"); value != "" {
from, err := parseTime(value, s.loc) from, err := parseTime(value, s.location)
if err != nil { if err != nil {
return page, fmt.Errorf("updatedFrom %w", err) return page, fmt.Errorf("updatedFrom %w", err)
} }
@@ -181,15 +185,15 @@ func (s *server) parseQuery(query url.Values) (pageQuery, error) {
} }
if value := query.Get("updatedTo"); value != "" { if value := query.Get("updatedTo"); value != "" {
to, err := parseTime(value, s.loc) until, err := parseTime(value, s.location)
if err != nil { if err != nil {
return page, fmt.Errorf("updatedTo %w", err) return page, fmt.Errorf("updatedTo %w", err)
} }
if query.Get("updatedFrom") != "" && to.Before(page.from) { if query.Get("updatedFrom") != "" && until.Before(page.from) {
return page, errors.New("updatedTo must not be before updatedFrom") return page, errors.New("updatedTo must not be before updatedFrom")
} }
if to.Before(page.upper) { if until.Before(page.upper) {
page.upper = to page.upper = until
} }
} }
+8 -2
View File
@@ -120,7 +120,8 @@ func TestHandlerRejectsBadRequests(t *testing.T) {
t.Errorf("%s: Content-Type %q", q, ct) t.Errorf("%s: Content-Type %q", q, ct)
} }
var body map[string]string var body map[string]string
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil || body["error"] != want { if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil ||
body["error"] != want {
t.Errorf("%s: body %s, want error %q", q, rec.Body.String(), want) t.Errorf("%s: body %s, want error %q", q, rec.Body.String(), want)
} }
} }
@@ -147,7 +148,12 @@ func TestParseTimeFormats(t *testing.T) {
t.Errorf("parseTime(%q) = %s, %v; want %s", v, got, err, want) t.Errorf("parseTime(%q) = %s, %v; want %s", v, got, err, want)
} }
} }
for _, v := range []string{"", "08/10/2026", "2026-10-08 12:00:00", "2026-10-08T25:00:00"} { for _, v := range []string{
"",
"08/10/2026",
"2026-10-08 12:00:00",
"2026-10-08T25:00:00",
} {
if _, err := parseTime(v, nil); err == nil { if _, err := parseTime(v, nil); err == nil {
t.Errorf("parseTime(%q): expected error", v) t.Errorf("parseTime(%q): expected error", v)
} }
+14 -9
View File
@@ -66,7 +66,8 @@ func (rl *RateLimiter) allow(key string) (bool, int) {
// Move to whichever fixed window, counted from the anchor, now falls in. // Move to whichever fixed window, counted from the anchor, now falls in.
index := now.Sub(client.anchor) / rl.window index := now.Sub(client.anchor) / rl.window
if windowStart := client.anchor.Add(index * rl.window); !windowStart.Equal(client.windowStart) { windowStart := client.anchor.Add(index * rl.window)
if !windowStart.Equal(client.windowStart) {
client.windowStart = windowStart client.windowStart = windowStart
client.count = 0 client.count = 0
} }
@@ -82,10 +83,12 @@ func (rl *RateLimiter) allow(key string) (bool, int) {
if nextHour := hour.Add(time.Hour); nextHour.Before(next) { if nextHour := hour.Add(time.Hour); nextHour.Before(next) {
next = nextHour next = nextHour
} }
seconds := int((next.Sub(now) + time.Second - 1) / time.Second) // round up seconds := int((next.Sub(now) + time.Second - 1) / time.Second) // round up
if seconds < 1 { if seconds < 1 {
seconds = 1 seconds = 1
} }
return false, seconds return false, seconds
} }
@@ -100,10 +103,10 @@ func (rl *RateLimiter) prune(hour time.Time) {
// clientKey produces an identifier for the client based on remote address. // clientKey produces an identifier for the client based on remote address.
// Could be updated to support proxy and forwarded-for headers. // Could be updated to support proxy and forwarded-for headers.
func clientKey(req *http.Request) string { func clientKey(r *http.Request) string {
host, _, err := net.SplitHostPort(req.RemoteAddr) host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil { if err != nil {
return req.RemoteAddr return r.RemoteAddr
} }
return host return host
} }
@@ -112,18 +115,20 @@ func clientKey(req *http.Request) string {
// to make a request in their current window or returns a "TooManyRequests" // to make a request in their current window or returns a "TooManyRequests"
// status and the time the client should wait before making another request. // status and the time the client should wait before making another request.
func (rl *RateLimiter) Middleware(next http.HandlerFunc) http.HandlerFunc { func (rl *RateLimiter) Middleware(next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, req *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
ok, retry := rl.allow(clientKey(req)) ok, retry := rl.allow(clientKey(r))
if !ok { if !ok {
w.Header().Set("Retry-After", strconv.Itoa(retry)) w.Header().Set("Retry-After", strconv.Itoa(retry))
w.Header().Set("Content-Type", "text/plain; charset=utf-8") w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusTooManyRequests) w.WriteHeader(http.StatusTooManyRequests)
fmt.Fprintf( //nolint:errcheck // the client may have gone; nothing to do fmt.Fprintf( //nolint:errcheck // the client may have gone; nothing to do
w, "Rate limit of %d exceeded. Please retry after %d seconds.", w,
rl.limit, retry, "Rate limit of %d exceeded. Please retry after %d seconds.",
rl.limit,
retry,
) )
return return
} }
next(w, req) next(w, r)
} }
} }
+2 -1
View File
@@ -10,7 +10,8 @@ import (
) )
func TestRateLimitWindows(t *testing.T) { func TestRateLimitWindows(t *testing.T) {
now := time.Date(2026, 10, 6, 10, 3, 20, 0, time.UTC) // first request of the hour // The first request of the hour.
now := time.Date(2026, 10, 6, 10, 3, 20, 0, time.UTC)
rl := NewRateLimiter(12, 5*time.Minute) rl := NewRateLimiter(12, 5*time.Minute)
rl.now = func() time.Time { rl.now = func() time.Time {
return now return now
+66 -36
View File
@@ -22,17 +22,17 @@ import (
// Store keeps every release in memory and appends each new one to a JSON Lines // Store keeps every release in memory and appends each new one to a JSON Lines
// file, so releases survive restarts. Releases are kept in date order. // file, so releases survive restarts. Releases are kept in date order.
type Store struct { type Store struct {
mu sync.RWMutex mu sync.RWMutex
f *os.File file *os.File
releases []ocds.Release releases []ocds.Release
times []time.Time times []time.Time
latest map[string]int // ocid -> index of its most recent release latest map[string]int // ocid -> index of its most recent release
lastNum map[int]int // year -> highest notice number used lastNumber map[int]int // year -> highest notice number used
// Procedures not yet awarded, as a slice plus positions so that adding, // Procedures not yet awarded, as a slice plus positions so that adding,
// removing and picking one at random are all constant time. // removing and picking one at random are all constant time.
open []string open []string
openPos map[string]int openPosition map[string]int
opts options opts options
} }
@@ -44,11 +44,12 @@ type Store struct {
// It logs to slog.Default() unless given WithLogger. // It logs to slog.Default() unless given WithLogger.
func OpenStore(path string, opts ...Option) (*Store, error) { func OpenStore(path string, opts ...Option) (*Store, error) {
store := &Store{ store := &Store{
latest: map[string]int{}, latest: map[string]int{},
lastNum: map[int]int{}, lastNumber: map[int]int{},
openPos: map[string]int{}, openPosition: map[string]int{},
opts: applyOptions(opts), opts: applyOptions(opts),
} }
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil { if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
return nil, err return nil, err
} }
@@ -57,6 +58,7 @@ func OpenStore(path string, opts ...Option) (*Store, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
// The file is written in order, but sort anyway in case it was hand-edited. // The file is written in order, but sort anyway in case it was hand-edited.
sort.SliceStable(releases, func(i, other int) bool { sort.SliceStable(releases, func(i, other int) bool {
return releases[i].Date.Before(releases[other].Date) return releases[i].Date.Before(releases[other].Date)
@@ -69,9 +71,10 @@ func OpenStore(path string, opts ...Option) (*Store, error) {
store.index(i, release) store.index(i, release)
} }
if store.f, err = openForAppend(path); err != nil { if store.file, err = openForAppend(path); err != nil {
return nil, err return nil, err
} }
return store, nil return store, nil
} }
@@ -106,6 +109,7 @@ func readReleases(logger *slog.Logger, path string) ([]ocds.Release, error) {
if err := scanner.Err(); err != nil { if err := scanner.Err(); err != nil {
return nil, fmt.Errorf("reading %s: %w", path, err) return nil, fmt.Errorf("reading %s: %w", path, err)
} }
return releases, nil return releases, nil
} }
@@ -113,18 +117,25 @@ func readReleases(logger *slog.Logger, path string) ([]ocds.Release, error) {
// mid-write, it ends the partial record so the next one starts on a fresh // mid-write, it ends the partial record so the next one starts on a fresh
// line. // line.
func openForAppend(path string) (*os.File, error) { func openForAppend(path string) (*os.File, error) {
file, err := os.OpenFile(filepath.Clean(path), os.O_CREATE|os.O_RDWR|os.O_APPEND, 0o600) file, err := os.OpenFile(
filepath.Clean(path), os.O_CREATE|os.O_RDWR|os.O_APPEND, 0o600,
)
if err != nil { if err != nil {
return nil, err return nil, err
} }
if info, err := file.Stat(); err == nil && info.Size() > 0 { if info, err := file.Stat(); err == nil && info.Size() > 0 {
last := make([]byte, 1) last := make([]byte, 1)
if _, err := file.ReadAt(last, info.Size()-1); err == nil && last[0] != '\n' { _, readErr := file.ReadAt(last, info.Size()-1)
if readErr == nil && last[0] != '\n' {
if _, err := file.Write([]byte("\n")); err != nil { if _, err := file.Write([]byte("\n")); err != nil {
return nil, errors.Join(fmt.Errorf("repairing %s: %w", path, err), file.Close()) return nil, errors.Join(
fmt.Errorf("repairing %s: %w", path, err), file.Close(),
)
} }
} }
} }
return file, nil return file, nil
} }
@@ -133,51 +144,58 @@ func openForAppend(path string) (*os.File, error) {
// highest notice number used each year. // highest notice number used each year.
func (s *Store) index(i int, release ocds.Release) { func (s *Store) index(i int, release ocds.Release) {
s.latest[release.OCID] = i s.latest[release.OCID] = i
if len(release.Tag) > 0 && release.Tag[0] == "award" { if len(release.Tag) > 0 && release.Tag[0] == "award" {
s.closeProcedure(release.OCID) s.closeProcedure(release.OCID)
} else if _, ok := s.openPos[release.OCID]; !ok { } else if _, ok := s.openPosition[release.OCID]; !ok {
s.openPos[release.OCID] = len(s.open) s.openPosition[release.OCID] = len(s.open)
s.open = append(s.open, release.OCID) s.open = append(s.open, release.OCID)
} }
if numberText, year, ok := strings.Cut(release.ID, "-"); ok { if numberText, year, ok := strings.Cut(release.ID, "-"); ok {
number, numberErr := strconv.Atoi(numberText) number, numberErr := strconv.Atoi(numberText)
yearNumber, yearErr := strconv.Atoi(year) yearNumber, yearErr := strconv.Atoi(year)
if numberErr == nil && yearErr == nil && number > s.lastNum[yearNumber] { if numberErr == nil && yearErr == nil && number > s.lastNumber[yearNumber] {
s.lastNum[yearNumber] = number s.lastNumber[yearNumber] = number
} }
} }
} }
// closeProcedure removes ocid from the open procedures, if it is there. // closeProcedure removes ocid from the open procedures, if it is there.
func (s *Store) closeProcedure(ocid string) { func (s *Store) closeProcedure(ocid string) {
position, ok := s.openPos[ocid] position, ok := s.openPosition[ocid]
if !ok { if !ok {
return return
} }
last := len(s.open) - 1 last := len(s.open) - 1
s.open[position] = s.open[last] s.open[position] = s.open[last]
s.openPos[s.open[position]] = position s.openPosition[s.open[position]] = position
s.open = s.open[:last] s.open = s.open[:last]
delete(s.openPos, ocid) delete(s.openPosition, ocid)
} }
// Append persists a release and flushes it to disk. // Append persists a release and flushes it to disk.
func (s *Store) Append(release ocds.Release) error { return s.append(release, true) } func (s *Store) Append(release ocds.Release) error {
return s.append(release, true)
}
// AppendNoSync persists a release without an fsync; call Sync afterwards. // AppendNoSync persists a release without an fsync; call Sync afterwards.
// Used for bulk catch-up, where a sync per record would be very slow. // Used for bulk catch-up, where a sync per record would be very slow.
func (s *Store) AppendNoSync(release ocds.Release) error { return s.append(release, false) } func (s *Store) AppendNoSync(release ocds.Release) error {
return s.append(release, false)
}
// Sync flushes appended releases to disk. Call it after AppendNoSync. // Sync flushes appended releases to disk. Call it after AppendNoSync.
func (s *Store) Sync() error { func (s *Store) Sync() error {
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
return s.f.Sync() return s.file.Sync()
} }
// append stores release, syncing the file to disk if sync is true. A date earlier // append stores release, syncing the file to disk if sync is true. A date
// than the newest stored release (for example after a clock change) is moved // earlier than the newest stored release (for example after a clock change) is
// up so the order stays valid. // moved up so the order stays valid.
func (s *Store) append(release ocds.Release, sync bool) error { func (s *Store) append(release ocds.Release, sync bool) error {
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
@@ -186,27 +204,34 @@ func (s *Store) append(release ocds.Release, sync bool) error {
if count := len(s.times); count > 0 && release.Date.Before(s.times[count-1]) { if count := len(s.times); count > 0 && release.Date.Before(s.times[count-1]) {
release.Date = s.times[count-1] release.Date = s.times[count-1]
} }
data, err := json.Marshal(release) data, err := json.Marshal(release)
if err != nil { if err != nil {
return err return err
} }
if _, err := s.f.Write(append(data, '\n')); err != nil { if _, err := s.file.Write(append(data, '\n')); err != nil {
return err return err
} }
if sync { if sync {
if err := s.f.Sync(); err != nil { if err := s.file.Sync(); err != nil {
return err return err
} }
} }
s.releases = append(s.releases, release) s.releases = append(s.releases, release)
s.times = append(s.times, release.Date) s.times = append(s.times, release.Date)
s.index(len(s.releases)-1, release) s.index(len(s.releases)-1, release)
return nil return nil
} }
// Page returns up to limit releases dated in [from, upper], starting no earlier // Page returns up to limit releases dated in [from, upper], starting no earlier
// than position start, plus the position to continue from and whether more exist. // than position start, plus the position to continue from and whether more
func (s *Store) Page(from, upper time.Time, start, limit int) ([]ocds.Release, int, bool) { // exist.
func (s *Store) Page(
from, upper time.Time,
start, limit int,
) ([]ocds.Release, int, bool) {
s.mu.RLock() s.mu.RLock()
defer s.mu.RUnlock() defer s.mu.RUnlock()
@@ -216,12 +241,15 @@ func (s *Store) Page(from, upper time.Time, start, limit int) ([]ocds.Release, i
if start > i { if start > i {
i = start i = start
} }
out := make([]ocds.Release, 0, limit) out := make([]ocds.Release, 0, limit)
for i < len(s.releases) && len(out) < limit && !s.times[i].After(upper) { for i < len(s.releases) && len(out) < limit && !s.times[i].After(upper) {
out = append(out, s.releases[i]) out = append(out, s.releases[i])
i++ i++
} }
more := i < len(s.releases) && !s.times[i].After(upper) more := i < len(s.releases) && !s.times[i].After(upper)
return out, i, more return out, i, more
} }
@@ -255,13 +283,15 @@ func (s *Store) HasOCID(ocid string) bool {
func (s *Store) NextNoticeID(year int) string { func (s *Store) NextNoticeID(year int) string {
s.mu.RLock() s.mu.RLock()
defer s.mu.RUnlock() defer s.mu.RUnlock()
return fmt.Sprintf("%06d-%d", s.lastNum[year]+1, year) return fmt.Sprintf("%06d-%d", s.lastNumber[year]+1, year)
} }
// RandomOpen picks a procedure that has not been awarded and whose latest // RandomOpen picks a procedure that has not been awarded and whose latest
// release is after since. Procedures idle since before then are dropped from // release is after since. Procedures idle since before then are dropped from
// the open set, as if they had been abandoned. // the open set, as if they had been abandoned.
func (s *Store) RandomOpen(rng *rand.Rand, since time.Time) (ocds.Release, bool) { func (s *Store) RandomOpen(
rng *rand.Rand, since time.Time,
) (ocds.Release, bool) {
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
for len(s.open) > 0 { for len(s.open) > 0 {
@@ -279,5 +309,5 @@ func (s *Store) RandomOpen(rng *rand.Rand, since time.Time) (ocds.Release, bool)
func (s *Store) Close() error { func (s *Store) Close() error {
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
return s.f.Close() return s.file.Close()
} }
+20 -9
View File
@@ -56,7 +56,9 @@ func TestStorePersistsAndGeneratorProgresses(t *testing.T) {
t.Fatalf("after reopen: %d releases, want %d", store2.Len(), len(all)) t.Fatalf("after reopen: %d releases, want %d", store2.Len(), len(all))
} }
gen2 := NewGenerator(cat, store2, 7) gen2 := NewGenerator(cat, store2, 7)
m, _ := gen2.CatchUp(now.Add(time.Hour), 48*time.Hour, 3*time.Minute, 15*time.Minute) m, _ := gen2.CatchUp(
now.Add(time.Hour), 48*time.Hour, 3*time.Minute, 15*time.Minute,
)
if m < 4 || m > 20 { if m < 4 || m > 20 {
t.Fatalf("one hour gap produced %d releases", m) t.Fatalf("one hour gap produced %d releases", m)
} }
@@ -66,9 +68,9 @@ func TestStorePersistsAndGeneratorProgresses(t *testing.T) {
// The HTTP handler pages through the store without gaps or repeats. // The HTTP handler pages through the store without gaps or repeats.
srv := &server{ srv := &server{
store: store2, store: store2,
baseURL: "", baseURL: "",
loc: time.UTC, location: time.UTC,
} }
seen := map[string]bool{} seen := map[string]bool{}
next := BasePath + "?updatedFrom=2026-10-07T00:00:00&limit=7" next := BasePath + "?updatedFrom=2026-10-07T00:00:00&limit=7"
@@ -102,7 +104,9 @@ func TestStorePersistsAndGeneratorProgresses(t *testing.T) {
func writeLines(t *testing.T, path string, lines ...string) { func writeLines(t *testing.T, path string, lines ...string) {
t.Helper() t.Helper()
if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")), 0o600); err != nil { if err := os.WriteFile(
path, []byte(strings.Join(lines, "\n")), 0o600,
); err != nil {
t.Fatal(err) t.Fatal(err)
} }
} }
@@ -122,7 +126,8 @@ func releaseLine(t *testing.T, ocid, id, tag string, at time.Time) string {
} }
// Unreadable lines are skipped, blank lines ignored, a hand-edited file is put // Unreadable lines are skipped, blank lines ignored, a hand-edited file is put
// back in date order, and the index (open procedures, notice numbers) is rebuilt. // back in date order, and the index (open procedures, notice numbers) is
// rebuilt.
func TestOpenStoreRecoversFromMessyFile(t *testing.T) { func TestOpenStoreRecoversFromMessyFile(t *testing.T) {
path := filepath.Join(t.TempDir(), "r.jsonl") path := filepath.Join(t.TempDir(), "r.jsonl")
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC) t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
@@ -148,7 +153,8 @@ func TestOpenStoreRecoversFromMessyFile(t *testing.T) {
for _, r := range all { for _, r := range all {
ids = append(ids, r.ID) ids = append(ids, r.ID)
} }
if got, want := strings.Join(ids, ","), "000003-2026,000005-2026,000007-2026"; got != want { want := "000003-2026,000005-2026,000007-2026"
if got := strings.Join(ids, ","); got != want {
t.Fatalf("releases %s, want %s", got, want) t.Fatalf("releases %s, want %s", got, want)
} }
if got := store.NextNoticeID(2026); got != "000008-2026" { if got := store.NextNoticeID(2026); got != "000008-2026" {
@@ -199,7 +205,11 @@ func TestOpenStoreAfterTruncatedWrite(t *testing.T) {
} }
defer store2.Close() defer store2.Close()
if store2.Len() != 2 || !store2.HasOCID("ocds-c") { if store2.Len() != 2 || !store2.HasOCID("ocds-c") {
t.Fatalf("after reopen: Len = %d, has ocds-c = %v", store2.Len(), store2.HasOCID("ocds-c")) t.Fatalf(
"after reopen: Len = %d, has ocds-c = %v",
store2.Len(),
store2.HasOCID("ocds-c"),
)
} }
} }
@@ -294,7 +304,8 @@ func TestRandomOpenDropsStaleProcedures(t *testing.T) {
//nolint:gosec // G404: seeded math/rand keeps the test deterministic //nolint:gosec // G404: seeded math/rand keeps the test deterministic
rnd := rand.New(rand.NewSource(1)) rnd := rand.New(rand.NewSource(1))
for i := 0; i < 10; i++ { for i := 0; i < 10; i++ {
if r, ok := store.RandomOpen(rnd, t0.Add(time.Minute)); !ok || r.OCID != "new" { if r, ok := store.RandomOpen(rnd, t0.Add(time.Minute)); !ok ||
r.OCID != "new" {
t.Fatalf("RandomOpen = %s, %v; want new", r.OCID, ok) t.Fatalf("RandomOpen = %s, %v; want new", r.OCID, ok)
} }
} }
+12
View File
@@ -78,6 +78,18 @@ Reads select whole rows (`SELECT *`, `t.*`, `c.*`) which map to a Go struct. sql
`ListOpportunities` is assembled by `listOpportunitiesQuery` from fragments in `queries.go`: a base `SELECT`, one `filterOpportunity*` clause per filter that is set, and the ordering and paging tail. Each filter fragment takes exactly one argument, except the `filterOpportunityMark*` fragments, which take none (`MarkUnset` puts `NOT` in front of them). Adding a filter means one fragment and one line in `listOpportunitiesQuery`. The classifications and the marks for a page of opportunities are then loaded with one `IN (...)` query each. `ListOpportunities` is assembled by `listOpportunitiesQuery` from fragments in `queries.go`: a base `SELECT`, one `filterOpportunity*` clause per filter that is set, and the ordering and paging tail. Each filter fragment takes exactly one argument, except the `filterOpportunityMark*` fragments, which take none (`MarkUnset` puts `NOT` in front of them). Adding a filter means one fragment and one line in `listOpportunitiesQuery`. The classifications and the marks for a page of opportunities are then loaded with one `IN (...)` query each.
### SQL style
A soft convention for the SQL in `queries.go`, followed so the queries look alike and read easily. A query that reads better another way can depart from it.
- A statement starts on the line after the opening backtick. A fragment combined into another query (`filterOpportunity*`) starts straight after it.
- Keywords are uppercase (`SELECT`, `INSERT INTO`, `ON CONFLICT`); table and column names are the lowercase snake_case of the schema.
- A short statement may stay on one line (`SELECT t.* FROM opportunities t WHERE t.uuid = ?`). A longer one has one clause per line, each at the start of the line: `SELECT`, `FROM`, `JOIN`, `WHERE`, `ORDER BY`, `LIMIT`.
- A column list that doesn't fit on one line goes inside its brackets, indented one tab and filled to about 80 characters per line. The `VALUES` list wraps at the same columns, so each line of values matches the line of columns above it.
- `ON CONFLICT … DO UPDATE SET` puts one `column = excluded.column` per line, indented one tab.
- A subquery (`EXISTS (…)`, `IN (…)`) is indented one tab inside its brackets, with the closing bracket at the start of a line.
- Tables are aliased consistently: `t` for `opportunities`, `c` for `classification_codes`, `tc` for `opportunity_classifications`, `m` for `opportunity_marks`.
## Store API ## Store API
```go ```go
+5 -17
View File
@@ -17,21 +17,6 @@ import (
//go:embed migrations/*.sql //go:embed migrations/*.sql
var migrationsFS embed.FS var migrationsFS embed.FS
// ErrNeedsMigration is returned by Open when the database has pending
// migrations. Apply them with the migrate command, or pass
// WithAutoMigrate.
var ErrNeedsMigration = errors.New("store: database needs migrating")
// ErrSchemaTooNew is returned by Open when the database has migrations
// this build doesn't know about, i.e. it was migrated by a newer build.
var ErrSchemaTooNew = errors.New("store: database schema is newer than this build")
// ErrWouldEmptyDatabase is returned by StepDown when the migration would drop
// all of the tables and thus empty the database.
var ErrWouldEmptyDatabase = errors.New(
"migration cancelled; database would be emptied by undoing all migrations",
)
// NewMigrator returns a goose Provider over the store's embedded migrations, // NewMigrator returns a goose Provider over the store's embedded migrations,
// for applying, rolling back or inspecting them. Each migration runs in its // for applying, rolling back or inspecting them. Each migration runs in its
// own transaction, so a failed migration leaves the database at the previous // own transaction, so a failed migration leaves the database at the previous
@@ -96,13 +81,15 @@ func checkNotNewer(ctx context.Context, provider *goose.Provider) error {
if err != nil { if err != nil {
return err return err
} }
if latest := LatestVersion(provider); current > latest { if latest := LatestVersion(provider); current > latest {
return fmt.Errorf( return fmt.Errorf(
"database is at version %d but this build only knows up to %d; use a newer build", "store: database is at version %d but this build only knows up to %d; use a newer build",
current, current,
latest, latest,
) )
} }
return nil return nil
} }
@@ -116,7 +103,7 @@ func prepareSchema(ctx context.Context, db *sql.DB, opts options) error {
if opts.autoMigrate { if opts.autoMigrate {
if _, err := provider.Up(ctx); err != nil { if _, err := provider.Up(ctx); err != nil {
return fmt.Errorf("store: migrating database: %w", err) return fmt.Errorf("migrating database: %w", err)
} }
} }
@@ -184,6 +171,7 @@ func StepDown(
applied++ applied++
} }
} }
all := steps == 0 || steps >= applied all := steps == 0 || steps >= applied
if all && !confirmDrop { if all && !confirmDrop {
return results, ErrWouldEmptyDatabase return results, ErrWouldEmptyDatabase
+20 -18
View File
@@ -29,10 +29,12 @@ func newMigrator(t *testing.T) (*goose.Provider, *sql.DB) {
t.Cleanup(func() { t.Cleanup(func() {
db.Close() db.Close()
}) })
m, err := store.NewMigrator(db) m, err := store.NewMigrator(db)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
return m, db return m, db
} }
@@ -105,7 +107,8 @@ func TestOpenRejectsNewerSchema(t *testing.T) {
"check": nil, "check": nil,
"auto-migrate": {store.WithAutoMigrate()}, "auto-migrate": {store.WithAutoMigrate()},
} { } {
if _, err := store.Open(ctx, path, opts...); !errors.Is(err, store.ErrSchemaTooNew) { _, err := store.Open(ctx, path, opts...)
if !errors.Is(err, store.ErrSchemaTooNew) {
t.Errorf("%s: err = %v, want ErrSchemaTooNew", name, err) t.Errorf("%s: err = %v, want ErrSchemaTooNew", name, err)
} }
} }
@@ -135,10 +138,12 @@ func TestMigrateDownAllAndUpAgain(t *testing.T) {
t.Fatalf("down to 0: %v", err) t.Fatalf("down to 0: %v", err)
} }
var tables int var tables int
if err := db.QueryRow(` query := `
SELECT count(*) FROM sqlite_master SELECT count(*) FROM sqlite_master
WHERE type IN ('table', 'index', 'trigger') WHERE type IN ('table', 'index', 'trigger')
AND name NOT LIKE 'sqlite_%' AND name != 'goose_db_version'`).Scan(&tables); err != nil { AND name NOT LIKE 'sqlite_%'
AND name != 'goose_db_version'`
if err := db.QueryRow(query).Scan(&tables); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if tables != 0 { if tables != 0 {
@@ -229,7 +234,10 @@ func TestDownAllRequiresConfirmation(t *testing.T) {
} }
hasErrored := false hasErrored := false
for name, steps := range map[string]int{"steps 0": 0, "more steps than applied": 99} { for name, steps := range map[string]int{
"steps 0": 0,
"more steps than applied": 99,
} {
_, err := store.StepDown(ctx, m, steps, false) _, err := store.StepDown(ctx, m, steps, false)
if !errors.Is(err, store.ErrWouldEmptyDatabase) { if !errors.Is(err, store.ErrWouldEmptyDatabase) {
t.Errorf("%s: %s", name, err) t.Errorf("%s: %s", name, err)
@@ -277,20 +285,11 @@ func TestNewerDatabaseIsRefused(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
if _, err := store.StepUp( if _, err := store.StepUp(ctx, m, 0); err == nil ||
ctx,
m,
0,
); err == nil ||
!strings.Contains(err.Error(), "newer build") { !strings.Contains(err.Error(), "newer build") {
t.Errorf("up: err = %v, want a newer-build error", err) t.Errorf("up: err = %v, want a newer-build error", err)
} }
if _, err := store.StepDown( if _, err := store.StepDown(ctx, m, 1, false); err == nil ||
ctx,
m,
1,
false,
); err == nil ||
!strings.Contains(err.Error(), "newer build") { !strings.Contains(err.Error(), "newer build") {
t.Errorf("down: err = %v, want a newer-build error", err) t.Errorf("down: err = %v, want a newer-build error", err)
} }
@@ -307,7 +306,9 @@ func TestFailedMigrationIsRolledBack(t *testing.T) {
// Occupy migration 4's table name with a view. Its CREATE TABLE IF NOT // Occupy migration 4's table name with a view. Its CREATE TABLE IF NOT
// EXISTS then skips, and the CREATE INDEX that follows fails because // EXISTS then skips, and the CREATE INDEX that follows fails because
// views can't be indexed, part-way through the migration. // views can't be indexed, part-way through the migration.
if _, err := db.Exec(`CREATE VIEW classification_codes AS SELECT 1 AS x`); err != nil { if _, err := db.Exec(
`CREATE VIEW classification_codes AS SELECT 1 AS x`,
); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -319,8 +320,9 @@ func TestFailedMigrationIsRolledBack(t *testing.T) {
t.Errorf("version after failure: %d, want 3", v) t.Errorf("version after failure: %d, want 3", v)
} }
var indexes int var indexes int
db.QueryRow(`SELECT count(*) FROM sqlite_master WHERE name LIKE 'classification_codes_%'`). query := `SELECT count(*) FROM sqlite_master
Scan(&indexes) WHERE name LIKE 'classification_codes_%'`
db.QueryRow(query).Scan(&indexes)
if indexes != 0 { if indexes != 0 {
t.Errorf("failed migration left %d objects behind", indexes) t.Errorf("failed migration left %d objects behind", indexes)
} }
+18 -12
View File
@@ -18,23 +18,27 @@ package store
// ---------- opportunities ---------- // ---------- opportunities ----------
// selectOpportunityLatestReleaseDate reads the date SaveOpportunity compares against. // selectOpportunityLatestReleaseDate reads the date SaveOpportunity compares
// against.
const selectOpportunityLatestReleaseDate = ` const selectOpportunityLatestReleaseDate = `
SELECT latest_release_date FROM opportunities WHERE uuid = ?` SELECT latest_release_date FROM opportunities WHERE uuid = ?`
// upsertOpportunity inserts or updates a opportunity. Named parameters (types.Opportunity). // upsertOpportunity inserts or updates a opportunity. Named parameters
// (types.Opportunity).
// //
// An UPDATE on conflict, not a REPLACE: REPLACE would delete the row first, // An UPDATE on conflict, not a REPLACE: REPLACE would delete the row first,
// cascading to the opportunity's releases and changing its id. // cascading to the opportunity's releases and changing its id.
const upsertOpportunity = ` const upsertOpportunity = `
INSERT INTO opportunities ( INSERT INTO opportunities (
uuid, uuid_type, title, status, description, buyer_name, buyer_id, uuid, uuid_type, title, status, description, buyer_name, buyer_id,
main_category, procurement_method, multi_lot, sme_suitable, vcse_suitable, main_category, procurement_method, multi_lot, sme_suitable,
start_date, end_date, gross_value, has_documents, latest_release_date vcse_suitable, start_date, end_date, gross_value, has_documents,
latest_release_date
) VALUES ( ) VALUES (
:uuid, :uuid_type, :title, :status, :description, :buyer_name, :buyer_id, :uuid, :uuid_type, :title, :status, :description, :buyer_name, :buyer_id,
:main_category, :procurement_method, :multi_lot, :sme_suitable, :vcse_suitable, :main_category, :procurement_method, :multi_lot, :sme_suitable,
:start_date, :end_date, :gross_value, :has_documents, :latest_release_date :vcse_suitable, :start_date, :end_date, :gross_value, :has_documents,
:latest_release_date
) )
ON CONFLICT (uuid) DO UPDATE SET ON CONFLICT (uuid) DO UPDATE SET
uuid_type = excluded.uuid_type, uuid_type = excluded.uuid_type,
@@ -162,7 +166,8 @@ INSERT INTO opportunity_release_tags (uuid, release_id, tag)
VALUES (:uuid, :release_id, :tag) VALUES (:uuid, :release_id, :tag)
ON CONFLICT DO NOTHING` ON CONFLICT DO NOTHING`
// selectReleaseTagsByOpportunity reads the tags of all of a opportunity's releases. // selectReleaseTagsByOpportunity reads the tags of all of a opportunity's
// releases.
const selectReleaseTagsByOpportunity = ` const selectReleaseTagsByOpportunity = `
SELECT * SELECT *
FROM opportunity_release_tags FROM opportunity_release_tags
@@ -180,13 +185,14 @@ ORDER BY release_id, tag`
const upsertClassificationCodeReturningID = ` const upsertClassificationCodeReturningID = `
INSERT INTO classification_codes (is_cpv, code, classification) INSERT INTO classification_codes (is_cpv, code, classification)
VALUES (:is_cpv, :code, :classification) VALUES (:is_cpv, :code, :classification)
ON CONFLICT (is_cpv, code, classification) DO UPDATE SET is_cpv = excluded.is_cpv ON CONFLICT (is_cpv, code, classification)
DO UPDATE SET is_cpv = excluded.is_cpv
RETURNING id` RETURNING id`
// ---------- opportunity_classifications ---------- // ---------- opportunity_classifications ----------
// deleteOpportunityClassifications clears a opportunity's mappings before they are // deleteOpportunityClassifications clears a opportunity's mappings before they
// rewritten. // are rewritten.
const deleteOpportunityClassifications = ` const deleteOpportunityClassifications = `
DELETE FROM opportunity_classifications WHERE opportunity_uuid = ?` DELETE FROM opportunity_classifications WHERE opportunity_uuid = ?`
@@ -197,8 +203,8 @@ VALUES (?, ?)
ON CONFLICT DO NOTHING` ON CONFLICT DO NOTHING`
// selectClassificationsForOpportunities reads the classifications of several // selectClassificationsForOpportunities reads the classifications of several
// opportunities. "IN (?)" is expanded by sqlx.In, which must be followed by Rebind. // opportunities. "IN (?)" is expanded by sqlx.In, which must be followed by
// c.* must map entirely onto types.ClassificationCode. // Rebind. c.* must map entirely onto types.ClassificationCode.
const selectClassificationsForOpportunities = ` const selectClassificationsForOpportunities = `
SELECT tc.opportunity_uuid, c.* SELECT tc.opportunity_uuid, c.*
FROM opportunity_classifications tc FROM opportunity_classifications tc
+81 -28
View File
@@ -30,26 +30,50 @@ type queryCheck struct {
// missing here. // missing here.
var checks = map[string]queryCheck{ var checks = map[string]queryCheck{
// opportunities // opportunities
"selectOpportunityLatestReleaseDate": {sql: selectOpportunityLatestReleaseDate}, "selectOpportunityLatestReleaseDate": {
"upsertOpportunity": {sql: upsertOpportunity, named: true}, sql: selectOpportunityLatestReleaseDate,
"selectOpportunityByUUID": {sql: selectOpportunityByUUID}, },
"upsertOpportunity": {sql: upsertOpportunity, named: true},
"selectOpportunityByUUID": {sql: selectOpportunityByUUID},
"listOpportunitiesSelect": { "listOpportunitiesSelect": {
sql: listOpportunitiesSelect + listOpportunitiesOrderAndPage, sql: listOpportunitiesSelect + listOpportunitiesOrderAndPage,
}, },
"listOpportunitiesOrderAndPage": { "listOpportunitiesOrderAndPage": {
sql: listOpportunitiesSelect + listOpportunitiesOrderAndPage, sql: listOpportunitiesSelect + listOpportunitiesOrderAndPage,
}, },
"filterOpportunityStatus": {sql: withFilter(filterOpportunityStatus)}, "filterOpportunityStatus": {
"filterOpportunityClassificationCode": {sql: withFilter(filterOpportunityClassificationCode)}, sql: withFilter(filterOpportunityStatus),
"filterOpportunityTitleMatch": {sql: withFilter(filterOpportunityTitleMatch)}, },
"filterOpportunityReleasedFrom": {sql: withFilter(filterOpportunityReleasedFrom)}, "filterOpportunityClassificationCode": {
"filterOpportunityReleasedBefore": {sql: withFilter(filterOpportunityReleasedBefore)}, sql: withFilter(filterOpportunityClassificationCode),
"filterOpportunityMarkHidden": {sql: withFilter(filterOpportunityMarkHidden)}, },
"filterOpportunityMarkSkipped": {sql: withFilter(filterOpportunityMarkSkipped)}, "filterOpportunityTitleMatch": {
"filterOpportunityMarkInterested": {sql: withFilter(filterOpportunityMarkInterested)}, sql: withFilter(filterOpportunityTitleMatch),
"filterOpportunityMarkBidSubmitted": {sql: withFilter(filterOpportunityMarkBidSubmitted)}, },
"filterOpportunityMarkWon": {sql: withFilter(filterOpportunityMarkWon)}, "filterOpportunityReleasedFrom": {
"filterOpportunityMarkLost": {sql: withFilter(filterOpportunityMarkLost)}, sql: withFilter(filterOpportunityReleasedFrom),
},
"filterOpportunityReleasedBefore": {
sql: withFilter(filterOpportunityReleasedBefore),
},
"filterOpportunityMarkHidden": {
sql: withFilter(filterOpportunityMarkHidden),
},
"filterOpportunityMarkSkipped": {
sql: withFilter(filterOpportunityMarkSkipped),
},
"filterOpportunityMarkInterested": {
sql: withFilter(filterOpportunityMarkInterested),
},
"filterOpportunityMarkBidSubmitted": {
sql: withFilter(filterOpportunityMarkBidSubmitted),
},
"filterOpportunityMarkWon": {
sql: withFilter(filterOpportunityMarkWon),
},
"filterOpportunityMarkLost": {
sql: withFilter(filterOpportunityMarkLost),
},
// opportunity_ocds_releases // opportunity_ocds_releases
"insertRelease": {sql: insertRelease, named: true}, "insertRelease": {sql: insertRelease, named: true},
@@ -61,12 +85,22 @@ var checks = map[string]queryCheck{
"selectReleaseTagsByOpportunity": {sql: selectReleaseTagsByOpportunity}, "selectReleaseTagsByOpportunity": {sql: selectReleaseTagsByOpportunity},
// classification_codes // classification_codes
"upsertClassificationCodeReturningID": {sql: upsertClassificationCodeReturningID, named: true}, "upsertClassificationCodeReturningID": {
sql: upsertClassificationCodeReturningID,
named: true,
},
// opportunity_classifications // opportunity_classifications
"deleteOpportunityClassifications": {sql: deleteOpportunityClassifications}, "deleteOpportunityClassifications": {
"insertOpportunityClassification": {sql: insertOpportunityClassification}, sql: deleteOpportunityClassifications,
"selectClassificationsForOpportunities": {sql: selectClassificationsForOpportunities, in: true}, },
"insertOpportunityClassification": {
sql: insertOpportunityClassification,
},
"selectClassificationsForOpportunities": {
sql: selectClassificationsForOpportunities,
in: true,
},
// opportunity_marks // opportunity_marks
"insertOpportunityMarks": {sql: insertOpportunityMarks, named: true}, "insertOpportunityMarks": {sql: insertOpportunityMarks, named: true},
@@ -75,12 +109,17 @@ var checks = map[string]queryCheck{
} }
func withFilter(fragment string) string { func withFilter(fragment string) string {
return listOpportunitiesSelect + "\nWHERE " + fragment + listOpportunitiesOrderAndPage return listOpportunitiesSelect + "\nWHERE " + fragment +
listOpportunitiesOrderAndPage
} }
func openTestDB(t *testing.T) *sqlx.DB { func openTestDB(t *testing.T) *sqlx.DB {
t.Helper() t.Helper()
s, err := Open(context.Background(), filepath.Join(t.TempDir(), "queries.db"), WithAutoMigrate()) s, err := Open(
context.Background(),
filepath.Join(t.TempDir(), "queries.db"),
WithAutoMigrate(),
)
if err != nil { if err != nil {
t.Fatalf("Open: %v", err) t.Fatalf("Open: %v", err)
} }
@@ -144,11 +183,13 @@ func TestListOpportunitiesQueryAllFilters(t *testing.T) {
Limit: 10, Limit: 10,
Offset: 5, Offset: 5,
}) })
stmt, err := db.Preparex(query) stmt, err := db.Preparex(query)
if err != nil { if err != nil {
t.Fatalf("prepare:\n%s\n%v", query, err) t.Fatalf("prepare:\n%s\n%v", query, err)
} }
defer stmt.Close() defer stmt.Close()
var rows []opportunityRow var rows []opportunityRow
if err := stmt.Select(&rows, args...); err != nil { if err := stmt.Select(&rows, args...); err != nil {
t.Errorf("run with %d args:\n%s\n%v", len(args), query, err) t.Errorf("run with %d args:\n%s\n%v", len(args), query, err)
@@ -162,6 +203,7 @@ func TestQueriesAllListed(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
var declared []string var declared []string
for _, decl := range file.Decls { for _, decl := range file.Decls {
gen, ok := decl.(*ast.GenDecl) gen, ok := decl.(*ast.GenDecl)
@@ -174,17 +216,21 @@ func TestQueriesAllListed(t *testing.T) {
} }
} }
} }
sort.Strings(declared) sort.Strings(declared)
if len(declared) == 0 { if len(declared) == 0 {
t.Fatal("found no constants in queries.go") t.Fatal("found no constants in queries.go")
} }
for _, name := range declared { for _, name := range declared {
if _, ok := checks[name]; !ok { if _, ok := checks[name]; !ok {
t.Errorf("queries.go constant %s is not listed in checks", name) t.Errorf("queries.go constant %s is not listed in checks", name)
} }
} }
for name := range checks { for name := range checks {
if i := sort.SearchStrings(declared, name); i == len(declared) || declared[i] != name { if i := sort.SearchStrings(declared, name); i == len(declared) ||
declared[i] != name {
t.Errorf("checks lists %s, which queries.go does not declare", name) t.Errorf("checks lists %s, which queries.go does not declare", name)
} }
} }
@@ -194,20 +240,27 @@ func TestQueriesAllListed(t *testing.T) {
// together, for writes that bypass OpportunityMarks.Validate. // together, for writes that bypass OpportunityMarks.Validate.
func TestMarksCheckConstraint(t *testing.T) { func TestMarksCheckConstraint(t *testing.T) {
db := openTestDB(t) db := openTestDB(t)
if _, err := db.Exec(`INSERT INTO opportunities insert := `INSERT INTO opportunities
(uuid, uuid_type, title, status, latest_release_date) (uuid, uuid_type, title, status, latest_release_date)
VALUES ('ocds-1', 'ocid', 'T', 'active', '2026-01-01T00:00:00Z')`); err != nil { VALUES ('ocds-1', 'ocid', 'T', 'active', '2026-01-01T00:00:00Z')`
if _, err := db.Exec(insert); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := db.Exec(`INSERT INTO opportunity_marks (uuid, won, lost)
VALUES ('ocds-1', 1, 1)`); err == nil { insertWonAndLost := `INSERT INTO opportunity_marks (uuid, won, lost)
VALUES ('ocds-1', 1, 1)`
if _, err := db.Exec(insertWonAndLost); err == nil {
t.Error("inserting won and lost together succeeded") t.Error("inserting won and lost together succeeded")
} }
if _, err := db.Exec(`INSERT INTO opportunity_marks (uuid, won)
VALUES ('ocds-1', 1)`); err != nil { insertWon := `INSERT INTO opportunity_marks (uuid, won)
VALUES ('ocds-1', 1)`
if _, err := db.Exec(insertWon); err != nil {
t.Fatalf("inserting won: %v", err) t.Fatalf("inserting won: %v", err)
} }
if _, err := db.Exec(`UPDATE opportunity_marks SET lost = 1 WHERE uuid = 'ocds-1'`); err == nil {
updateLost := `UPDATE opportunity_marks SET lost = 1 WHERE uuid = 'ocds-1'`
if _, err := db.Exec(updateLost); err == nil {
t.Error("updating to won and lost together succeeded") t.Error("updating to won and lost together succeeded")
} }
} }
+187 -113
View File
@@ -18,8 +18,25 @@ import (
"git.avsdev.uk/Go/job-search/internal/types" "git.avsdev.uk/Go/job-search/internal/types"
) )
// ErrNotFound is returned when a requested opportunity does not exist. // Errors returned by the store.
var ErrNotFound = errors.New("store: not found") var (
// ErrNotFound is returned when a requested opportunity does not exist.
ErrNotFound = errors.New("store: not found")
// ErrNeedsMigration is returned by Open when the database has pending
// migrations. Apply them with the migrate command, or pass
// WithAutoMigrate.
ErrNeedsMigration = errors.New("store: database needs migrating")
// ErrSchemaTooNew is returned by Open when the database has migrations
// this build doesn't know about, i.e. it was migrated by a newer build.
ErrSchemaTooNew = errors.New(
"store: database schema is newer than this build",
)
// ErrWouldEmptyDatabase is returned by StepDown when the migration would
// drop all of the tables and thus empty the database.
ErrWouldEmptyDatabase = errors.New(
"store: migration cancelled; database would be emptied by undoing all migrations",
)
)
// DefaultListLimit is used when OpportunityFilter.Limit is zero or negative. // DefaultListLimit is used when OpportunityFilter.Limit is zero or negative.
const DefaultListLimit = 50 const DefaultListLimit = 50
@@ -32,14 +49,15 @@ type Store struct {
// SaveMultipleResult reports what SaveOpportunities changed. // SaveMultipleResult reports what SaveOpportunities changed.
type SaveMultipleResult struct { type SaveMultipleResult struct {
// NewOpportunities is the number of opportunities that were not already stored. // NewOpportunities is the number of opportunities that were not already
// stored.
NewOpportunities int NewOpportunities int
// NewReleases is the number of releases that were not already stored. // NewReleases is the number of releases that were not already stored.
NewReleases int NewReleases int
// UpdatedOpportunities is incremented for each opportunity row and its classifications // UpdatedOpportunities is incremented for each opportunity row and its
// that are updated (when every release in the save was older than the // classifications that are updated (when every release in the save was older
// one the stored opportunity was last updated from). It does not include new // than the one the stored opportunity was last updated from). It does not
// opportunities. // include new opportunities.
UpdatedOpportunities int UpdatedOpportunities int
// OpportunityResults is a slice of the results of each opportunity saved. // OpportunityResults is a slice of the results of each opportunity saved.
OpportunityResults []SaveResult OpportunityResults []SaveResult
@@ -47,8 +65,8 @@ type SaveMultipleResult struct {
// SaveResult reports what SaveOpportunity changed. // SaveResult reports what SaveOpportunity changed.
type SaveResult struct { type SaveResult struct {
// OpportunityUpdated is true if the opportunity row and its classifications were // OpportunityUpdated is true if the opportunity row and its classifications
// written. It is false when every release in the save was older than the // were written. It is false when every release in the save was older than the
// one the stored opportunity was last updated from. // one the stored opportunity was last updated from.
OpportunityUpdated bool OpportunityUpdated bool
// NewReleases is the number of releases that were not already stored. // NewReleases is the number of releases that were not already stored.
@@ -58,6 +76,7 @@ type SaveResult struct {
// MarkFilter filters ListOpportunities on one of an opportunity's marks. // MarkFilter filters ListOpportunities on one of an opportunity's marks.
type MarkFilter int type MarkFilter int
// Values of MarkFilter.
const ( const (
// MarkAny does not filter. It is the zero value. // MarkAny does not filter. It is the zero value.
MarkAny MarkFilter = iota MarkAny MarkFilter = iota
@@ -68,8 +87,8 @@ const (
MarkUnset MarkUnset
) )
// OpportunityFilter selects opportunities for ListOpportunities. Zero-valued fields do not // OpportunityFilter selects opportunities for ListOpportunities. Zero-valued
// filter. // fields do not filter.
type OpportunityFilter struct { type OpportunityFilter struct {
Status string Status string
// ClassificationCode matches opportunities with a classification of this code // ClassificationCode matches opportunities with a classification of this code
@@ -117,14 +136,18 @@ func prepareOpportunity(opportunity *types.Opportunity) error {
opportunity.Marks.UUID = opportunity.UUID opportunity.Marks.UUID = opportunity.UUID
if err := opportunity.Marks.Validate(); err != nil { if err := opportunity.Marks.Validate(); err != nil {
return fmt.Errorf("store: opportunity %s: %w", opportunity.UUID, err) return fmt.Errorf("opportunity %s: %w", opportunity.UUID, err)
} }
latest := opportunity.LatestReleaseDate latest := opportunity.LatestReleaseDate
for i := range opportunity.Releases { for i := range opportunity.Releases {
release := &opportunity.Releases[i] release := &opportunity.Releases[i]
if release.ReleaseID == "" { if release.ReleaseID == "" {
return fmt.Errorf("store: opportunity %s: release %d has no release id", opportunity.UUID, i) return fmt.Errorf(
"store: opportunity %s: release %d has no release id",
opportunity.UUID,
i,
)
} }
release.UUID = opportunity.UUID release.UUID = opportunity.UUID
release.ReleaseDate = release.ReleaseDate.UTC() release.ReleaseDate = release.ReleaseDate.UTC()
@@ -138,7 +161,9 @@ func prepareOpportunity(opportunity *types.Opportunity) error {
} }
if latest.IsZero() { if latest.IsZero() {
return fmt.Errorf("store: opportunity %s has no release date", opportunity.UUID) return fmt.Errorf(
"store: opportunity %s has no release date", opportunity.UUID,
)
} }
opportunity.LatestReleaseDate = latest.UTC() opportunity.LatestReleaseDate = latest.UTC()
@@ -188,12 +213,15 @@ func Open(ctx context.Context, path string, opts ...Option) (*Store, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
if err := sqldb.PingContext(ctx); err != nil { if err := sqldb.PingContext(ctx); err != nil {
return nil, errors.Join(err, sqldb.Close()) return nil, errors.Join(err, sqldb.Close())
} }
if err := prepareSchema(ctx, sqldb, settings); err != nil { if err := prepareSchema(ctx, sqldb, settings); err != nil {
return nil, errors.Join(err, sqldb.Close()) return nil, errors.Join(err, sqldb.Close())
} }
return &Store{ return &Store{
db: sqlx.NewDb(sqldb, "sqlite3"), db: sqlx.NewDb(sqldb, "sqlite3"),
}, nil }, nil
@@ -207,10 +235,10 @@ func (s *Store) Close() error { return s.db.Close() }
// reads fail loudly ("missing destination name ...") until it is mapped, // reads fail loudly ("missing destination name ...") until it is mapped,
// instead of being silently ignored. // instead of being silently ignored.
// //
// opportunityRow gives the opportunities table's internal id column a home, so it stays // opportunityRow gives the opportunities table's internal id column a home, so
// out of types.Opportunity. sqlx maps the embedded Opportunity's fields as if declared // it stays out of types.Opportunity. sqlx maps the embedded Opportunity's
// here. The other tables have no internal columns and read straight into // fields as if declared here. The other tables have no internal columns and
// internal/types. // read straight into internal/types.
type opportunityRow struct { type opportunityRow struct {
ID int64 `db:"id"` ID int64 `db:"id"`
types.Opportunity types.Opportunity
@@ -230,8 +258,7 @@ type opportunityRow struct {
// opportunity is normalised in place (see prepareOpportunity), and when it is // opportunity is normalised in place (see prepareOpportunity), and when it is
// written its classifications' IDs are filled in. // written its classifications' IDs are filled in.
func (s *Store) SaveOpportunity( func (s *Store) SaveOpportunity(
ctx context.Context, ctx context.Context, opportunity *types.Opportunity,
opportunity *types.Opportunity,
) (SaveResult, error) { ) (SaveResult, error) {
if err := prepareOpportunity(opportunity); err != nil { if err := prepareOpportunity(opportunity); err != nil {
return SaveResult{}, err return SaveResult{}, err
@@ -243,14 +270,16 @@ func (s *Store) SaveOpportunity(
} }
defer tx.Rollback() //nolint:errcheck // no-op after Commit defer tx.Rollback() //nolint:errcheck // no-op after Commit
res, _, err := saveOpportunity(ctx, tx, opportunity) result, _, err := saveOpportunity(ctx, tx, opportunity)
if err != nil { if err != nil {
return SaveResult{}, err return SaveResult{}, err
} }
if err := tx.Commit(); err != nil { if err := tx.Commit(); err != nil {
return SaveResult{}, err return SaveResult{}, err
} }
return res, nil
return result, nil
} }
// SaveOpportunities stores each of opportunities and its releases, all in one // SaveOpportunities stores each of opportunities and its releases, all in one
@@ -282,28 +311,29 @@ func (s *Store) SaveOpportunities(
} }
defer tx.Rollback() //nolint:errcheck // no-op after Commit defer tx.Rollback() //nolint:errcheck // no-op after Commit
allRes := SaveMultipleResult{ total := SaveMultipleResult{
OpportunityResults: make([]SaveResult, len(opportunities)), OpportunityResults: make([]SaveResult, len(opportunities)),
} }
for i, opportunity := range opportunities { for i, opportunity := range opportunities {
res, isNew, err := saveOpportunity(ctx, tx, opportunity) result, isNew, err := saveOpportunity(ctx, tx, opportunity)
if err != nil { if err != nil {
return SaveMultipleResult{}, err return SaveMultipleResult{}, err
} }
allRes.OpportunityResults[i] = res total.OpportunityResults[i] = result
allRes.NewReleases += res.NewReleases total.NewReleases += result.NewReleases
switch { switch {
case isNew: case isNew:
allRes.NewOpportunities++ total.NewOpportunities++
case res.OpportunityUpdated: case result.OpportunityUpdated:
allRes.UpdatedOpportunities++ total.UpdatedOpportunities++
} }
} }
if err := tx.Commit(); err != nil { if err := tx.Commit(); err != nil {
return SaveMultipleResult{}, err return SaveMultipleResult{}, err
} }
return allRes, nil
return total, nil
} }
// saveOpportunity does the work of SaveOpportunity and SaveOpportunities for // saveOpportunity does the work of SaveOpportunity and SaveOpportunities for
@@ -311,28 +341,31 @@ func (s *Store) SaveOpportunities(
// not already stored. // not already stored.
func saveOpportunity( func saveOpportunity(
ctx context.Context, tx *sqlx.Tx, opportunity *types.Opportunity, ctx context.Context, tx *sqlx.Tx, opportunity *types.Opportunity,
) (res SaveResult, isNew bool, err error) { ) (SaveResult, bool, error) {
var stored time.Time var (
err = tx.GetContext(ctx, &stored, selectOpportunityLatestReleaseDate, opportunity.UUID) result SaveResult
isNew bool
stored time.Time
)
err := tx.GetContext(
ctx, &stored, selectOpportunityLatestReleaseDate, opportunity.UUID,
)
switch { switch {
case errors.Is(err, sql.ErrNoRows): case errors.Is(err, sql.ErrNoRows):
isNew = true isNew = true
res.OpportunityUpdated = true result.OpportunityUpdated = true
case err != nil: case err != nil:
return SaveResult{}, false, fmt.Errorf( return SaveResult{}, false, fmt.Errorf(
"store: reading opportunity %s: %w", "reading opportunity %s: %w", opportunity.UUID, err,
opportunity.UUID,
err,
) )
default: default:
res.OpportunityUpdated = !opportunity.LatestReleaseDate.Before(stored) result.OpportunityUpdated = !opportunity.LatestReleaseDate.Before(stored)
} }
if res.OpportunityUpdated {
if result.OpportunityUpdated {
if err := writeOpportunity(ctx, tx, opportunity); err != nil { if err := writeOpportunity(ctx, tx, opportunity); err != nil {
return SaveResult{}, false, fmt.Errorf( return SaveResult{}, false, fmt.Errorf(
"store: writing opportunity %s: %w", "writing opportunity %s: %w", opportunity.UUID, err,
opportunity.UUID,
err,
) )
} }
} }
@@ -343,10 +376,11 @@ func saveOpportunity(
return SaveResult{}, false, err return SaveResult{}, false, err
} }
if inserted { if inserted {
res.NewReleases++ result.NewReleases++
} }
} }
return res, isNew, nil
return result, isNew, nil
} }
// insertReleaseWithTags stores release and its tags unless it is already // insertReleaseWithTags stores release and its tags unless it is already
@@ -357,77 +391,96 @@ func insertReleaseWithTags(
result, err := tx.NamedExecContext(ctx, insertRelease, release) result, err := tx.NamedExecContext(ctx, insertRelease, release)
if err != nil { if err != nil {
return false, fmt.Errorf( return false, fmt.Errorf(
"store: inserting release %s/%s: %w", "inserting release %s/%s: %w",
release.UUID, release.UUID,
release.ReleaseID, release.ReleaseID,
err, err,
) )
} }
if affected, err := result.RowsAffected(); err != nil || affected == 0 { if affected, err := result.RowsAffected(); err != nil || affected == 0 {
return false, err // n == 0: already stored return false, err // affected == 0: already stored
} }
for _, tag := range release.Tags { for _, tag := range release.Tags {
if _, err := tx.NamedExecContext(ctx, insertReleaseTag, tag); err != nil { if _, err := tx.NamedExecContext(ctx, insertReleaseTag, tag); err != nil {
return false, fmt.Errorf( return false, fmt.Errorf(
"store: inserting tag for %s/%s: %w", release.UUID, release.ReleaseID, err, "inserting tag for %s/%s: %w",
release.UUID,
release.ReleaseID,
err,
) )
} }
} }
return true, nil return true, nil
} }
// writeOpportunity upserts opportunity's row, gives it a marks row if it has none // writeOpportunity upserts opportunity's row, gives it a marks row if it has
// (existing marks are kept), and replaces its classifications, filling in // none (existing marks are kept), and replaces its classifications, filling in
// their IDs. // their IDs.
func writeOpportunity( func writeOpportunity(
ctx context.Context, ctx context.Context, tx *sqlx.Tx, opportunity *types.Opportunity,
tx *sqlx.Tx,
opportunity *types.Opportunity,
) (err error) { ) (err error) {
if _, err := tx.NamedExecContext(ctx, upsertOpportunity, opportunity); err != nil { _, err = tx.NamedExecContext(ctx, upsertOpportunity, opportunity)
return err if err != nil {
}
if _, err := tx.NamedExecContext(ctx, insertOpportunityMarks, opportunity.Marks); err != nil {
return err
}
if _, err := tx.ExecContext(ctx, deleteOpportunityClassifications, opportunity.UUID); err != nil {
return err return err
} }
statement, err := tx.PrepareNamedContext(ctx, upsertClassificationCodeReturningID) _, err = tx.NamedExecContext(ctx, insertOpportunityMarks, opportunity.Marks)
if err != nil {
return err
}
_, err = tx.ExecContext(
ctx, deleteOpportunityClassifications, opportunity.UUID,
)
if err != nil {
return err
}
statement, err := tx.PrepareNamedContext(
ctx, upsertClassificationCodeReturningID,
)
if err != nil { if err != nil {
return err return err
} }
defer func() { defer func() {
err = errors.Join(err, statement.Close()) err = errors.Join(err, statement.Close())
}() }()
for i := range opportunity.Classifications { for i := range opportunity.Classifications {
classification := &opportunity.Classifications[i] classification := &opportunity.Classifications[i]
if err := statement.GetContext(ctx, &classification.ID, classification); err != nil {
err = statement.GetContext(ctx, &classification.ID, classification)
if err != nil {
return err return err
} }
if _, err := tx.ExecContext(
ctx, _, err = tx.ExecContext(
insertOpportunityClassification, ctx, insertOpportunityClassification, opportunity.UUID, classification.ID,
opportunity.UUID, )
classification.ID, if err != nil {
); err != nil {
return err return err
} }
} }
return nil return nil
} }
// SetMarks replaces the marks of the opportunity marks.UUID, or returns // SetMarks replaces the marks of the opportunity marks.UUID, or returns
// ErrNotFound. Marks that fail marks.Validate (e.g. types.ErrWonAndLost) are // ErrNotFound. Marks that fail marks.Validate (e.g. types.ErrWonAndLost) are
// not stored. // not stored.
func (s *Store) SetMarks(ctx context.Context, marks types.OpportunityMarks) error { func (s *Store) SetMarks(
ctx context.Context, marks types.OpportunityMarks,
) error {
if marks.UUID == "" { if marks.UUID == "" {
return errors.New("store: marks have no UUID") return errors.New("store: marks have no UUID")
} }
if err := marks.Validate(); err != nil { if err := marks.Validate(); err != nil {
return fmt.Errorf("store: opportunity %s: %w", marks.UUID, err) return fmt.Errorf("opportunity %s: %w", marks.UUID, err)
} }
_, err := s.db.NamedExecContext(ctx, upsertOpportunityMarks, marks) _, err := s.db.NamedExecContext(ctx, upsertOpportunityMarks, marks)
if isForeignKeyError(err) { if isForeignKeyError(err) {
return ErrNotFound return ErrNotFound
@@ -439,12 +492,15 @@ func (s *Store) SetMarks(ctx context.Context, marks types.OpportunityMarks) erro
// breaks a foreign key, e.g. one referring to a missing opportunity. // breaks a foreign key, e.g. one referring to a missing opportunity.
func isForeignKeyError(err error) bool { func isForeignKeyError(err error) bool {
var sqliteErr sqlite3.Error var sqliteErr sqlite3.Error
return errors.As(err, &sqliteErr) && sqliteErr.ExtendedCode == sqlite3.ErrConstraintForeignKey return errors.As(err, &sqliteErr) &&
sqliteErr.ExtendedCode == sqlite3.ErrConstraintForeignKey
} }
// GetOpportunity returns the opportunity with its classifications, marks and // GetOpportunity returns the opportunity with its classifications, marks and
// all of its releases (with tags), or ErrNotFound. // all of its releases (with tags), or ErrNotFound.
func (s *Store) GetOpportunity(ctx context.Context, uuid string) (*types.Opportunity, error) { func (s *Store) GetOpportunity(
ctx context.Context, uuid string,
) (*types.Opportunity, error) {
var row opportunityRow var row opportunityRow
err := s.db.GetContext(ctx, &row, selectOpportunityByUUID, uuid) err := s.db.GetContext(ctx, &row, selectOpportunityByUUID, uuid)
if errors.Is(err, sql.ErrNoRows) { if errors.Is(err, sql.ErrNoRows) {
@@ -453,6 +509,7 @@ func (s *Store) GetOpportunity(ctx context.Context, uuid string) (*types.Opportu
if err != nil { if err != nil {
return nil, err return nil, err
} }
opportunities := []types.Opportunity{row.Opportunity} opportunities := []types.Opportunity{row.Opportunity}
if err := s.loadClassifications(ctx, opportunities); err != nil { if err := s.loadClassifications(ctx, opportunities); err != nil {
return nil, err return nil, err
@@ -460,9 +517,11 @@ func (s *Store) GetOpportunity(ctx context.Context, uuid string) (*types.Opportu
if err := s.loadMarks(ctx, opportunities); err != nil { if err := s.loadMarks(ctx, opportunities); err != nil {
return nil, err return nil, err
} }
if opportunities[0].Releases, err = s.ListReleases(ctx, uuid); err != nil { if opportunities[0].Releases, err = s.ListReleases(ctx, uuid); err != nil {
return nil, err return nil, err
} }
return &opportunities[0], nil return &opportunities[0], nil
} }
@@ -470,29 +529,32 @@ func (s *Store) GetOpportunity(ctx context.Context, uuid string) (*types.Opportu
// classifications and marks but without releases (use ListReleases or // classifications and marks but without releases (use ListReleases or
// GetOpportunity for those). // GetOpportunity for those).
func (s *Store) ListOpportunities( func (s *Store) ListOpportunities(
ctx context.Context, ctx context.Context, filter OpportunityFilter,
filter OpportunityFilter,
) ([]types.Opportunity, error) { ) ([]types.Opportunity, error) {
query, args := listOpportunitiesQuery(filter) query, args := listOpportunitiesQuery(filter)
var rows []opportunityRow var rows []opportunityRow
if err := s.db.SelectContext(ctx, &rows, query, args...); err != nil { if err := s.db.SelectContext(ctx, &rows, query, args...); err != nil {
return nil, err return nil, err
} }
opportunities := make([]types.Opportunity, len(rows)) opportunities := make([]types.Opportunity, len(rows))
for i, row := range rows { for i, row := range rows {
opportunities[i] = row.Opportunity opportunities[i] = row.Opportunity
} }
if err := s.loadClassifications(ctx, opportunities); err != nil { if err := s.loadClassifications(ctx, opportunities); err != nil {
return nil, err return nil, err
} }
if err := s.loadMarks(ctx, opportunities); err != nil { if err := s.loadMarks(ctx, opportunities); err != nil {
return nil, err return nil, err
} }
return opportunities, nil return opportunities, nil
} }
// listOpportunitiesQuery assembles the ListOpportunities query from the fragments in // listOpportunitiesQuery assembles the ListOpportunities query from the
// queries.go, with one argument per filter that is set (none for mark filters). // fragments in queries.go, with one argument per filter that is set (none for
// mark filters).
func listOpportunitiesQuery(filter OpportunityFilter) (string, []any) { func listOpportunitiesQuery(filter OpportunityFilter) (string, []any) {
var where []string var where []string
var args []any var args []any
@@ -500,6 +562,7 @@ func listOpportunitiesQuery(filter OpportunityFilter) (string, []any) {
where = append(where, clause) where = append(where, clause)
args = append(args, argument) args = append(args, argument)
} }
if filter.Status != "" { if filter.Status != "" {
add(filterOpportunityStatus, filter.Status) add(filterOpportunityStatus, filter.Status)
} }
@@ -515,35 +578,19 @@ func listOpportunitiesQuery(filter OpportunityFilter) (string, []any) {
if !filter.ReleasedBefore.IsZero() { if !filter.ReleasedBefore.IsZero() {
add(filterOpportunityReleasedBefore, filter.ReleasedBefore.UTC()) add(filterOpportunityReleasedBefore, filter.ReleasedBefore.UTC())
} }
for _, mark := range []struct {
marks := []struct {
filter MarkFilter filter MarkFilter
clause string clause string
}{ }{
{ {filter: filter.Hidden, clause: filterOpportunityMarkHidden},
filter: filter.Hidden, {filter: filter.Skipped, clause: filterOpportunityMarkSkipped},
clause: filterOpportunityMarkHidden, {filter: filter.Interested, clause: filterOpportunityMarkInterested},
}, {filter: filter.BidSubmitted, clause: filterOpportunityMarkBidSubmitted},
{ {filter: filter.Won, clause: filterOpportunityMarkWon},
filter: filter.Skipped, {filter: filter.Lost, clause: filterOpportunityMarkLost},
clause: filterOpportunityMarkSkipped, }
}, for _, mark := range marks {
{
filter: filter.Interested,
clause: filterOpportunityMarkInterested,
},
{
filter: filter.BidSubmitted,
clause: filterOpportunityMarkBidSubmitted,
},
{
filter: filter.Won,
clause: filterOpportunityMarkWon,
},
{
filter: filter.Lost,
clause: filterOpportunityMarkLost,
},
} {
switch mark.filter { switch mark.filter {
case MarkSet: case MarkSet:
where = append(where, mark.clause) where = append(where, mark.clause)
@@ -557,14 +604,19 @@ func listOpportunitiesQuery(filter OpportunityFilter) (string, []any) {
query += "\nWHERE " + strings.Join(where, "\n\tAND ") query += "\nWHERE " + strings.Join(where, "\n\tAND ")
} }
query += listOpportunitiesOrderAndPage query += listOpportunitiesOrderAndPage
return query, append(args, filter.limit(), filter.Offset) return query, append(args, filter.limit(), filter.Offset)
} }
// loadClassifications fills in Classifications for all opportunities with one query. // loadClassifications fills in Classifications for all opportunities with one
func (s *Store) loadClassifications(ctx context.Context, opportunities []types.Opportunity) error { // query.
func (s *Store) loadClassifications(
ctx context.Context, opportunities []types.Opportunity,
) error {
if len(opportunities) == 0 { if len(opportunities) == 0 {
return nil return nil
} }
uuids := make([]string, len(opportunities)) uuids := make([]string, len(opportunities))
byUUID := make(map[string]*types.Opportunity, len(opportunities)) byUUID := make(map[string]*types.Opportunity, len(opportunities))
for i := range opportunities { for i := range opportunities {
@@ -576,27 +628,36 @@ func (s *Store) loadClassifications(ctx context.Context, opportunities []types.O
if err != nil { if err != nil {
return err return err
} }
// opportunity_uuid says which opportunity each row belongs to. // opportunity_uuid says which opportunity each row belongs to.
var rows []struct { var rows []struct {
OpportunityUUID string `db:"opportunity_uuid"` OpportunityUUID string `db:"opportunity_uuid"`
types.ClassificationCode types.ClassificationCode
} }
if err := s.db.SelectContext(ctx, &rows, s.db.Rebind(query), args...); err != nil { err = s.db.SelectContext(ctx, &rows, s.db.Rebind(query), args...)
if err != nil {
return err return err
} }
for _, releaseRow := range rows {
opportunity := byUUID[releaseRow.OpportunityUUID] for _, classificationRow := range rows {
opportunity.Classifications = append(opportunity.Classifications, releaseRow.ClassificationCode) opportunity := byUUID[classificationRow.OpportunityUUID]
opportunity.Classifications = append(
opportunity.Classifications, classificationRow.ClassificationCode,
)
} }
return nil return nil
} }
// loadMarks fills in Marks for all opportunities with one query. An // loadMarks fills in Marks for all opportunities with one query. An
// opportunity with no marks row gets zero marks carrying its UUID. // opportunity with no marks row gets zero marks carrying its UUID.
func (s *Store) loadMarks(ctx context.Context, opportunities []types.Opportunity) error { func (s *Store) loadMarks(
ctx context.Context, opportunities []types.Opportunity,
) error {
if len(opportunities) == 0 { if len(opportunities) == 0 {
return nil return nil
} }
uuids := make([]string, len(opportunities)) uuids := make([]string, len(opportunities))
byUUID := make(map[string]*types.Opportunity, len(opportunities)) byUUID := make(map[string]*types.Opportunity, len(opportunities))
for i := range opportunities { for i := range opportunities {
@@ -611,42 +672,55 @@ func (s *Store) loadMarks(ctx context.Context, opportunities []types.Opportunity
if err != nil { if err != nil {
return err return err
} }
var rows []types.OpportunityMarks var rows []types.OpportunityMarks
if err := s.db.SelectContext(ctx, &rows, s.db.Rebind(query), args...); err != nil { err = s.db.SelectContext(ctx, &rows, s.db.Rebind(query), args...)
if err != nil {
return err return err
} }
for _, row := range rows { for _, row := range rows {
byUUID[row.UUID].Marks = row byUUID[row.UUID].Marks = row
} }
return nil return nil
} }
// ListReleases returns a opportunity's releases with their tags, oldest first. // ListReleases returns a opportunity's releases with their tags, oldest first.
func (s *Store) ListReleases(ctx context.Context, uuid string) ([]types.OpportunityRelease, error) { func (s *Store) ListReleases(
ctx context.Context, uuid string,
) ([]types.OpportunityRelease, error) {
releases := []types.OpportunityRelease{} releases := []types.OpportunityRelease{}
if err := s.db.SelectContext(ctx, &releases, selectReleasesByOpportunity, uuid); err != nil { err := s.db.SelectContext(ctx, &releases, selectReleasesByOpportunity, uuid)
if err != nil {
return nil, err return nil, err
} }
byID := make(map[string]*types.OpportunityRelease, len(releases)) byID := make(map[string]*types.OpportunityRelease, len(releases))
for i := range releases { for i := range releases {
byID[releases[i].ReleaseID] = &releases[i] byID[releases[i].ReleaseID] = &releases[i]
} }
var tags []types.OpportunityReleaseTag var tags []types.OpportunityReleaseTag
if err := s.db.SelectContext(ctx, &tags, selectReleaseTagsByOpportunity, uuid); err != nil { err = s.db.SelectContext(ctx, &tags, selectReleaseTagsByOpportunity, uuid)
if err != nil {
return nil, err return nil, err
} }
for _, tag := range tags { for _, tag := range tags {
if release := byID[tag.ReleaseID]; release != nil { if release := byID[tag.ReleaseID]; release != nil {
release.Tags = append(release.Tags, tag) release.Tags = append(release.Tags, tag)
} }
} }
return releases, nil return releases, nil
} }
// ReleaseExists reports whether a release is already stored, so a fetcher // ReleaseExists reports whether a release is already stored, so a fetcher
// can skip converting it. // can skip converting it.
func (s *Store) ReleaseExists(ctx context.Context, uuid, releaseID string) (bool, error) { func (s *Store) ReleaseExists(
ctx context.Context, uuid, releaseID string,
) (bool, error) {
var exists bool var exists bool
err := s.db.GetContext(ctx, &exists, selectReleaseExists, uuid, releaseID) err := s.db.GetContext(ctx, &exists, selectReleaseExists, uuid, releaseID)
return exists, err return exists, err
+146 -65
View File
@@ -30,7 +30,11 @@ func day(d int) time.Time {
return time.Date(2026, time.March, d, 9, 30, 0, 0, time.UTC) return time.Date(2026, time.March, d, 9, 30, 0, 0, time.UTC)
} }
func opportunity(uuid, title string, releaseDay int, releaseIDs ...string) *types.Opportunity { func opportunity(
uuid, title string,
releaseDay int,
releaseIDs ...string,
) *types.Opportunity {
start, end := day(20), day(28) start, end := day(20), day(28)
t := &types.Opportunity{ t := &types.Opportunity{
UUID: uuid, UUID: uuid,
@@ -39,7 +43,7 @@ func opportunity(uuid, title string, releaseDay int, releaseIDs ...string) *type
Status: "active", Status: "active",
Description: "A description", Description: "A description",
BuyerName: "Example Council", BuyerName: "Example Council",
BuyerId: "GB-EXAMPLE", BuyerID: "GB-EXAMPLE",
MainCategory: "services", MainCategory: "services",
ProcurementMethod: "open", ProcurementMethod: "open",
MultiLot: true, MultiLot: true,
@@ -49,8 +53,16 @@ func opportunity(uuid, title string, releaseDay int, releaseIDs ...string) *type
GrossValue: 125000.50, GrossValue: 125000.50,
HasDocuments: true, HasDocuments: true,
Classifications: []types.ClassificationCode{ Classifications: []types.ClassificationCode{
{IsCPV: true, Code: "72000000", Classification: "IT services"}, {
{IsCPV: true, Code: "03000000", Classification: "Agricultural products"}, IsCPV: true,
Code: "72000000",
Classification: "IT services",
},
{
IsCPV: true,
Code: "03000000",
Classification: "Agricultural products",
},
}, },
// No UUID: SaveOpportunity must fill it in. // No UUID: SaveOpportunity must fill it in.
Marks: types.OpportunityMarks{ Marks: types.OpportunityMarks{
@@ -58,18 +70,29 @@ func opportunity(uuid, title string, releaseDay int, releaseIDs ...string) *type
BidSubmitted: true, BidSubmitted: true,
}, },
} }
for i, id := range releaseIDs { for i, id := range releaseIDs {
t.Releases = append(t.Releases, types.OpportunityRelease{ t.Releases = append(t.Releases, types.OpportunityRelease{
ReleaseID: id, ReleaseID: id,
ReleaseDate: day(releaseDay).Add(time.Duration(i) * time.Minute), ReleaseDate: day(releaseDay).Add(time.Duration(i) * time.Minute),
JSON: `{"id":"` + id + `"}`, JSON: `{"id":"` + id + `"}`,
Tags: []types.OpportunityReleaseTag{{Tag: "opportunity"}, {Tag: "planning"}}, Tags: []types.OpportunityReleaseTag{
{
Tag: "opportunity",
},
{
Tag: "planning",
},
},
}) })
} }
return t return t
} }
func mustSave(t *testing.T, s *store.Store, td *types.Opportunity) store.SaveResult { func mustSave(
t *testing.T, s *store.Store, td *types.Opportunity,
) store.SaveResult {
t.Helper() t.Helper()
res, err := s.SaveOpportunity(context.Background(), td) res, err := s.SaveOpportunity(context.Background(), td)
if err != nil { if err != nil {
@@ -116,24 +139,30 @@ func TestSaveAndGetRoundTrip(t *testing.T) {
} }
got := mustGet(t, s, "ocds-1") got := mustGet(t, s, "ocds-1")
if got.Title != in.Title || got.BuyerName != in.BuyerName || got.BuyerId != in.BuyerId || if got.Title != in.Title || got.BuyerName != in.BuyerName ||
got.MainCategory != in.MainCategory || got.ProcurementMethod != in.ProcurementMethod || got.BuyerID != in.BuyerID || got.MainCategory != in.MainCategory ||
got.UUIDType != "ocid" || got.Status != "active" || got.Description != in.Description || got.ProcurementMethod != in.ProcurementMethod || got.UUIDType != "ocid" ||
!got.MultiLot || !got.SMESuitable || got.VCSESuitable || !got.HasDocuments || got.Status != "active" || got.Description != in.Description ||
got.GrossValue != in.GrossValue { !got.MultiLot || !got.SMESuitable || got.VCSESuitable ||
!got.HasDocuments || got.GrossValue != in.GrossValue {
t.Errorf("opportunity fields differ:\n got %+v\nwant %+v", got, in) t.Errorf("opportunity fields differ:\n got %+v\nwant %+v", got, in)
} }
if got.StartDate == nil || !got.StartDate.Equal(day(20)) || got.EndDate == nil || if got.StartDate == nil || !got.StartDate.Equal(day(20)) ||
!got.EndDate.Equal(day(28)) { got.EndDate == nil || !got.EndDate.Equal(day(28)) {
t.Errorf("dates: got %v / %v", got.StartDate, got.EndDate) t.Errorf("dates: got %v / %v", got.StartDate, got.EndDate)
} }
if want := day(1).Add(time.Minute); !got.LatestReleaseDate.Equal(want) { if want := day(1).Add(time.Minute); !got.LatestReleaseDate.Equal(want) {
t.Errorf("LatestReleaseDate = %v, want %v", got.LatestReleaseDate, want) t.Errorf("LatestReleaseDate = %v, want %v", got.LatestReleaseDate, want)
} }
if len(got.Classifications) != 2 || got.Classifications[0].Code != "03000000" || if len(got.Classifications) != 2 ||
got.Classifications[1].Code != "72000000" || got.Classifications[0].ID == 0 { got.Classifications[0].Code != "03000000" ||
t.Errorf("classifications: got %+v (leading zero must survive)", got.Classifications) got.Classifications[1].Code != "72000000" ||
got.Classifications[0].ID == 0 {
t.Errorf(
"classifications: got %+v (leading zero must survive)",
got.Classifications,
)
} }
wantMarks := types.OpportunityMarks{ wantMarks := types.OpportunityMarks{
@@ -153,9 +182,8 @@ func TestSaveAndGetRoundTrip(t *testing.T) {
!r.ReleaseDate.Equal(day(1)) { !r.ReleaseDate.Equal(day(1)) {
t.Errorf("release: got %+v", r) t.Errorf("release: got %+v", r)
} }
if len(r.Tags) != 2 || if len(r.Tags) != 2 || r.Tags[0].Tag != "opportunity" ||
r.Tags[0].Tag != "opportunity" || r.Tags[1].Tag != "planning" || r.Tags[1].Tag != "planning" || r.Tags[0].ReleaseID != "r1" ||
r.Tags[0].ReleaseID != "r1" ||
r.Tags[0].UUID != "ocds-1" { r.Tags[0].UUID != "ocds-1" {
t.Errorf("tags: got %+v", r.Tags) t.Errorf("tags: got %+v", r.Tags)
} }
@@ -178,7 +206,9 @@ func TestReleasesAreWriteOnce(t *testing.T) {
again := opportunity("ocds-1", "Title", 1, "r1") again := opportunity("ocds-1", "Title", 1, "r1")
again.Releases[0].JSON = `{"changed":true}` again.Releases[0].JSON = `{"changed":true}`
again.Releases[0].Tags = []types.OpportunityReleaseTag{{Tag: "award"}} again.Releases[0].Tags = []types.OpportunityReleaseTag{{
Tag: "award",
}}
if res := mustSave(t, s, again); res.NewReleases != 0 { if res := mustSave(t, s, again); res.NewReleases != 0 {
t.Errorf("re-save: NewReleases = %d, want 0", res.NewReleases) t.Errorf("re-save: NewReleases = %d, want 0", res.NewReleases)
} }
@@ -206,7 +236,9 @@ func TestOlderReleaseDoesNotOverwriteOpportunity(t *testing.T) {
t.Errorf("opportunity was overwritten by an older release: %+v", got) t.Errorf("opportunity was overwritten by an older release: %+v", got)
} }
if len(got.Releases) != 2 || got.Releases[0].ReleaseID != "r1" { if len(got.Releases) != 2 || got.Releases[0].ReleaseID != "r1" {
t.Errorf("older release should still be stored, oldest first: %+v", got.Releases) t.Errorf(
"older release should still be stored, oldest first: %+v", got.Releases,
)
} }
} }
@@ -220,7 +252,11 @@ func TestNewerReleaseUpdatesOpportunityAndSearchIndex(t *testing.T) {
newer := opportunity("ocds-1", "Website redesign", 2, "r2") newer := opportunity("ocds-1", "Website redesign", 2, "r2")
newer.Status = "complete" newer.Status = "complete"
newer.Classifications = []types.ClassificationCode{ newer.Classifications = []types.ClassificationCode{
{IsCPV: true, Code: "72000000", Classification: "IT services (amended)"}, {
IsCPV: true,
Code: "72000000",
Classification: "IT services (amended)",
},
} }
if res := mustSave(t, s, newer); !res.OpportunityUpdated { if res := mustSave(t, s, newer); !res.OpportunityUpdated {
t.Fatalf("newer save not applied: %+v", res) t.Fatalf("newer save not applied: %+v", res)
@@ -246,7 +282,12 @@ func TestNewerReleaseUpdatesOpportunityAndSearchIndex(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
if len(ts) != want { if len(ts) != want {
t.Errorf("search %q after title change: %d results, want %d", title, len(ts), want) t.Errorf(
"search %q after title change: %d results, want %d",
title,
len(ts),
want,
)
} }
} }
} }
@@ -257,7 +298,8 @@ func TestClassificationRowsAreShared(t *testing.T) {
b := opportunity("ocds-b", "B", 1, "r1") b := opportunity("ocds-b", "B", 1, "r1")
mustSave(t, s, a) mustSave(t, s, a)
mustSave(t, s, b) mustSave(t, s, b)
if a.Classifications[0].ID == 0 || a.Classifications[0].ID != b.Classifications[0].ID { if a.Classifications[0].ID == 0 ||
a.Classifications[0].ID != b.Classifications[0].ID {
t.Errorf( t.Errorf(
"identical classifications should share an ID: %d vs %d", "identical classifications should share an ID: %d vs %d",
a.Classifications[0].ID, a.Classifications[0].ID,
@@ -267,7 +309,11 @@ func TestClassificationRowsAreShared(t *testing.T) {
c := opportunity("ocds-c", "C", 1, "r1") c := opportunity("ocds-c", "C", 1, "r1")
c.Classifications = []types.ClassificationCode{ c.Classifications = []types.ClassificationCode{
{IsCPV: false, Code: "72000000", Classification: "IT services"}, {
IsCPV: false,
Code: "72000000",
Classification: "IT services",
},
} }
mustSave(t, s, c) mustSave(t, s, c)
if c.Classifications[0].ID == a.Classifications[0].ID { if c.Classifications[0].ID == a.Classifications[0].ID {
@@ -279,13 +325,17 @@ func TestSameReleaseIDInDifferentOpportunities(t *testing.T) {
s := openStore(t) s := openStore(t)
a := opportunity("ocds-a", "A", 1, "001") a := opportunity("ocds-a", "A", 1, "001")
b := opportunity("ocds-b", "B", 1, "001") b := opportunity("ocds-b", "B", 1, "001")
b.Releases[0].Tags = []types.OpportunityReleaseTag{{Tag: "award"}} b.Releases[0].Tags = []types.OpportunityReleaseTag{{
Tag: "award",
}}
if res := mustSave(t, s, a); res.NewReleases != 1 { if res := mustSave(t, s, a); res.NewReleases != 1 {
t.Fatal(res) t.Fatal(res)
} }
if res := mustSave(t, s, b); res.NewReleases != 1 { if res := mustSave(t, s, b); res.NewReleases != 1 {
t.Fatalf("release id reused under another OCID must be stored: %+v", res) t.Fatalf("release id reused under another OCID must be stored: %+v", res)
} }
got := mustGet(t, s, "ocds-b").Releases got := mustGet(t, s, "ocds-b").Releases
if len(got) != 1 || len(got[0].Tags) != 1 || got[0].Tags[0].Tag != "award" { if len(got) != 1 || len(got[0].Tags) != 1 || got[0].Tags[0].Tag != "award" {
t.Errorf("tags leaked between opportunities: %+v", got) t.Errorf("tags leaked between opportunities: %+v", got)
@@ -300,7 +350,11 @@ func TestListOpportunitiesFilters(t *testing.T) {
roads := opportunity("ocds-2", "Road resurfacing works", 2, "r1") roads := opportunity("ocds-2", "Road resurfacing works", 2, "r1")
roads.Status = "complete" roads.Status = "complete"
roads.Classifications = []types.ClassificationCode{ roads.Classifications = []types.ClassificationCode{
{IsCPV: true, Code: "45233142", Classification: "Road-repair works"}, {
IsCPV: true,
Code: "45233142",
Classification: "Road-repair works",
},
} }
hosting := opportunity("ocds-3", "Managed cloud hosting", 3, "r1") hosting := opportunity("ocds-3", "Managed cloud hosting", 3, "r1")
for _, td := range []*types.Opportunity{cloud, roads, hosting} { for _, td := range []*types.Opportunity{cloud, roads, hosting} {
@@ -312,7 +366,11 @@ func TestListOpportunitiesFilters(t *testing.T) {
f store.OpportunityFilter f store.OpportunityFilter
want []string want []string
}{ }{
{"all, newest first", store.OpportunityFilter{}, []string{"ocds-3", "ocds-2", "ocds-1"}}, {
"all, newest first",
store.OpportunityFilter{},
[]string{"ocds-3", "ocds-2", "ocds-1"},
},
{"status", store.OpportunityFilter{ {"status", store.OpportunityFilter{
Status: "complete", Status: "complete",
}, []string{"ocds-2"}}, }, []string{"ocds-2"}},
@@ -385,9 +443,13 @@ func TestListOpportunitiesFilters(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if len(got) != 1 || len(got[0].Classifications) != 1 || got[0].Releases != nil || if len(got) != 1 || len(got[0].Classifications) != 1 ||
got[0].Marks.UUID != "ocds-2" || !got[0].Marks.Interested { got[0].Releases != nil || got[0].Marks.UUID != "ocds-2" ||
t.Errorf("list should include classifications and marks and omit releases: %+v", got) !got[0].Marks.Interested {
t.Errorf(
"list should include classifications and marks and omit releases: %+v",
got,
)
} }
} }
@@ -402,26 +464,28 @@ func TestReleaseExistsAndListReleases(t *testing.T) {
}{{"ocds-1", "r1", true}, {"ocds-1", "r3", false}, {"ocds-2", "r1", false}} { }{{"ocds-1", "r1", true}, {"ocds-1", "r3", false}, {"ocds-2", "r1", false}} {
got, err := s.ReleaseExists(ctx, tc.uuid, tc.id) got, err := s.ReleaseExists(ctx, tc.uuid, tc.id)
if err != nil || got != tc.want { if err != nil || got != tc.want {
t.Errorf("ReleaseExists(%s, %s) = %v, %v; want %v", tc.uuid, tc.id, got, err, tc.want) t.Errorf(
"ReleaseExists(%s, %s) = %v, %v; want %v",
tc.uuid,
tc.id,
got,
err,
tc.want,
)
} }
} }
rs, err := s.ListReleases(ctx, "ocds-1") rs, err := s.ListReleases(ctx, "ocds-1")
if err != nil || len(rs) != 2 || rs[0].ReleaseID != "r1" || rs[1].ReleaseID != "r2" || if err != nil || len(rs) != 2 || rs[0].ReleaseID != "r1" ||
len(rs[1].Tags) != 2 { rs[1].ReleaseID != "r2" || len(rs[1].Tags) != 2 {
t.Errorf("ListReleases: %+v, %v", rs, err) t.Errorf("ListReleases: %+v, %v", rs, err)
} }
} }
func TestGetOpportunityNotFound(t *testing.T) { func TestGetOpportunityNotFound(t *testing.T) {
s := openStore(t) s := openStore(t)
if _, err := s.GetOpportunity( _, err := s.GetOpportunity(context.Background(), "missing")
context.Background(), if !errors.Is(err, store.ErrNotFound) {
"missing",
); !errors.Is(
err,
store.ErrNotFound,
) {
t.Errorf("err = %v, want ErrNotFound", err) t.Errorf("err = %v, want ErrNotFound", err)
} }
} }
@@ -473,8 +537,12 @@ func TestResaveLoadedOpportunity(t *testing.T) {
ids := []int64{loaded.Classifications[0].ID, loaded.Classifications[1].ID} ids := []int64{loaded.Classifications[0].ID, loaded.Classifications[1].ID}
loaded.Title = "Amended" loaded.Title = "Amended"
loaded.Releases = append(loaded.Releases, types.OpportunityRelease{ loaded.Releases = append(loaded.Releases, types.OpportunityRelease{
ReleaseID: "r2", ReleaseDate: day(2), JSON: `{}`, ReleaseID: "r2",
Tags: []types.OpportunityReleaseTag{{Tag: "opportunityAmendment"}}, ReleaseDate: day(2),
JSON: `{}`,
Tags: []types.OpportunityReleaseTag{{
Tag: "opportunityAmendment",
}},
}) })
res := mustSave(t, s, loaded) res := mustSave(t, s, loaded)
if !res.OpportunityUpdated || res.NewReleases != 1 { if !res.OpportunityUpdated || res.NewReleases != 1 {
@@ -482,10 +550,12 @@ func TestResaveLoadedOpportunity(t *testing.T) {
} }
got := mustGet(t, s, "ocds-1") got := mustGet(t, s, "ocds-1")
if got.Title != "Amended" || len(got.Releases) != 2 || !got.LatestReleaseDate.Equal(day(2)) { if got.Title != "Amended" || len(got.Releases) != 2 ||
!got.LatestReleaseDate.Equal(day(2)) {
t.Errorf("resave not applied: %+v", got) t.Errorf("resave not applied: %+v", got)
} }
if got.Classifications[0].ID != ids[0] || got.Classifications[1].ID != ids[1] { if got.Classifications[0].ID != ids[0] ||
got.Classifications[1].ID != ids[1] {
t.Errorf( t.Errorf(
"classification IDs changed on resave: %v -> %d, %d", "classification IDs changed on resave: %v -> %d, %d",
ids, ids,
@@ -527,10 +597,7 @@ func TestSetMarksErrors(t *testing.T) {
UUID: "missing", UUID: "missing",
Hidden: true, Hidden: true,
}, },
); !errors.Is( ); !errors.Is(err, store.ErrNotFound) {
err,
store.ErrNotFound,
) {
t.Errorf("unknown opportunity: err = %v, want ErrNotFound", err) t.Errorf("unknown opportunity: err = %v, want ErrNotFound", err)
} }
if err := s.SetMarks(ctx, types.OpportunityMarks{ if err := s.SetMarks(ctx, types.OpportunityMarks{
@@ -565,7 +632,9 @@ func TestSaveKeepsExistingMarks(t *testing.T) {
newest := opportunity("ocds-1", "Amended again", 3, "r3") newest := opportunity("ocds-1", "Amended again", 3, "r3")
newest.Marks = types.OpportunityMarks{} newest.Marks = types.OpportunityMarks{}
if _, err := s.SaveOpportunities(ctx, []*types.Opportunity{newest}); err != nil { if _, err := s.SaveOpportunities(
ctx, []*types.Opportunity{newest},
); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if got := mustGet(t, s, "ocds-1").Marks; got != want { if got := mustGet(t, s, "ocds-1").Marks; got != want {
@@ -580,9 +649,13 @@ func TestSaveOpportunitiesStoresMarks(t *testing.T) {
b.Marks = types.OpportunityMarks{ b.Marks = types.OpportunityMarks{
Skipped: true, Skipped: true,
} }
if _, err := s.SaveOpportunities(context.Background(), []*types.Opportunity{a, b}); err != nil {
if _, err := s.SaveOpportunities(
context.Background(), []*types.Opportunity{a, b},
); err != nil {
t.Fatal(err) t.Fatal(err)
} }
for uuid, want := range map[string]types.OpportunityMarks{ for uuid, want := range map[string]types.OpportunityMarks{
"ocds-a": {UUID: "ocds-a", Interested: true, BidSubmitted: true}, "ocds-a": {UUID: "ocds-a", Interested: true, BidSubmitted: true},
"ocds-b": {UUID: "ocds-b", Skipped: true}, "ocds-b": {UUID: "ocds-b", Skipped: true},
@@ -610,7 +683,10 @@ func TestListOpportunitiesMarkFilters(t *testing.T) {
Lost: true, Lost: true,
}) })
mustSetMarks(t, s, types.OpportunityMarks{ mustSetMarks(t, s, types.OpportunityMarks{
UUID: "ocds-4", Interested: true, BidSubmitted: true, Won: true, UUID: "ocds-4",
Interested: true,
BidSubmitted: true,
Won: true,
}) })
set, unset := store.MarkSet, store.MarkUnset set, unset := store.MarkSet, store.MarkUnset
@@ -619,7 +695,11 @@ func TestListOpportunitiesMarkFilters(t *testing.T) {
f store.OpportunityFilter f store.OpportunityFilter
want []string want []string
}{ }{
{"any", store.OpportunityFilter{}, []string{"ocds-4", "ocds-3", "ocds-2", "ocds-1"}}, {
"any",
store.OpportunityFilter{},
[]string{"ocds-4", "ocds-3", "ocds-2", "ocds-1"},
},
{"hidden", store.OpportunityFilter{ {"hidden", store.OpportunityFilter{
Hidden: set, Hidden: set,
}, []string{"ocds-2"}}, }, []string{"ocds-2"}},
@@ -698,8 +778,8 @@ func TestListOpportunitiesMarkFilters(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if len(got) != 4 || !got[0].Marks.Won || !got[1].Marks.Lost || !got[2].Marks.Hidden || if len(got) != 4 || !got[0].Marks.Won || !got[1].Marks.Lost ||
got[3].Marks.Hidden || !got[3].Marks.Interested { !got[2].Marks.Hidden || got[3].Marks.Hidden || !got[3].Marks.Interested {
t.Errorf("list marks: %+v", got) t.Errorf("list marks: %+v", got)
} }
} }
@@ -727,19 +807,20 @@ func TestWonAndLostAreExclusive(t *testing.T) {
Won: true, Won: true,
Lost: true, Lost: true,
} }
if _, err := s.SaveOpportunity(ctx, in); !errors.Is(err, types.ErrWonAndLost) { _, err := s.SaveOpportunity(ctx, in)
if !errors.Is(err, types.ErrWonAndLost) {
t.Errorf("SaveOpportunity(won and lost): err = %v, want ErrWonAndLost", err) t.Errorf("SaveOpportunity(won and lost): err = %v, want ErrWonAndLost", err)
} }
if _, err := s.SaveOpportunities(
ctx, _, err = s.SaveOpportunities(ctx, []*types.Opportunity{in})
[]*types.Opportunity{in}, if !errors.Is(err, types.ErrWonAndLost) {
); !errors.Is( t.Errorf(
err, "SaveOpportunities(won and lost): err = %v, want ErrWonAndLost", err,
types.ErrWonAndLost, )
) {
t.Errorf("SaveOpportunities(won and lost): err = %v, want ErrWonAndLost", err)
} }
if _, err := s.GetOpportunity(ctx, "ocds-2"); !errors.Is(err, store.ErrNotFound) {
_, err = s.GetOpportunity(ctx, "ocds-2")
if !errors.Is(err, store.ErrNotFound) {
t.Errorf("rejected opportunity was stored: err = %v", err) t.Errorf("rejected opportunity was stored: err = %v", err)
} }
+2 -2
View File
@@ -17,7 +17,7 @@ type Opportunity struct {
Description string `db:"description" json:"description"` Description string `db:"description" json:"description"`
BuyerName string `db:"buyer_name" json:"buyer_name"` BuyerName string `db:"buyer_name" json:"buyer_name"`
Classifications []ClassificationCode `db:"-" json:"classifications"` Classifications []ClassificationCode `db:"-" json:"classifications"`
BuyerId string `db:"buyer_id" json:"buyer_id"` BuyerID string `db:"buyer_id" json:"buyer_id"`
MainCategory string `db:"main_category" json:"main_category"` MainCategory string `db:"main_category" json:"main_category"`
ProcurementMethod string `db:"procurement_method" json:"procurement_method"` ProcurementMethod string `db:"procurement_method" json:"procurement_method"`
MultiLot bool `db:"multi_lot" json:"multi_lot"` MultiLot bool `db:"multi_lot" json:"multi_lot"`
@@ -53,7 +53,7 @@ type OpportunityReleaseTag struct {
// ErrWonAndLost is returned by OpportunityMarks.Validate when both Won and // ErrWonAndLost is returned by OpportunityMarks.Validate when both Won and
// Lost are set. // Lost are set.
var ErrWonAndLost = errors.New("an opportunity cannot be both won and lost") var ErrWonAndLost = errors.New("types: an opportunity cannot be both won and lost")
// OpportunityMarks are the user's own marks on an opportunity, kept apart // OpportunityMarks are the user's own marks on an opportunity, kept apart
// from the fetched data so saving a newer release never changes them. The // from the fetched data so saving a newer release never changes them. The
+6 -2
View File
@@ -83,15 +83,19 @@ func ParseByteSize(text string) (ByteSize, error) {
return ByteSizeNoLimit, nil return ByteSizeNoLimit, nil
} }
if strings.HasPrefix(trimmed, "-") { if strings.HasPrefix(trimmed, "-") {
return 0, fmt.Errorf("byte size %q is negative; only -1 (no limit) is allowed", text) return 0, fmt.Errorf(
"util: byte size %q is negative; only -1 (no limit) is allowed", text,
)
} }
parsed, err := humanize.ParseBytes(text) parsed, err := humanize.ParseBytes(text)
if err != nil { if err != nil {
return 0, err return 0, err
} }
if parsed > math.MaxInt64 { if parsed > math.MaxInt64 {
return 0, fmt.Errorf("byte size %q is too large", text) return 0, fmt.Errorf("util: byte size %q is too large", text)
} }
return ByteSize(parsed), nil return ByteSize(parsed), nil
} }
+9 -2
View File
@@ -35,7 +35,9 @@ func TestByteSizeNoLimit(t *testing.T) {
for _, text := range []string{"-1", "-1B", "-1b", " -1 "} { for _, text := range []string{"-1", "-1B", "-1b", " -1 "} {
size, err := ParseByteSize(text) size, err := ParseByteSize(text)
if err != nil || size != ByteSizeNoLimit { if err != nil || size != ByteSizeNoLimit {
t.Errorf("ParseByteSize(%q) = %d, %v; want ByteSizeNoLimit", text, size, err) t.Errorf(
"ParseByteSize(%q) = %d, %v; want ByteSizeNoLimit", text, size, err,
)
} }
} }
if got := ByteSizeNoLimit.String(); got != "-1" { if got := ByteSizeNoLimit.String(); got != "-1" {
@@ -44,7 +46,12 @@ func TestByteSizeNoLimit(t *testing.T) {
for _, text := range []string{"-2", "-1KB", "-0.5", "-1KiB", "-100"} { for _, text := range []string{"-2", "-1KB", "-0.5", "-1KiB", "-100"} {
size, err := ParseByteSize(text) size, err := ParseByteSize(text)
if err == nil || !strings.Contains(err.Error(), "only -1 (no limit)") { if err == nil || !strings.Contains(err.Error(), "only -1 (no limit)") {
t.Errorf("ParseByteSize(%q) = %d, %v; want a negative size error", text, size, err) t.Errorf(
"ParseByteSize(%q) = %d, %v; want a negative size error",
text,
size,
err,
)
} }
} }
} }
+19
View File
@@ -0,0 +1,19 @@
package util
import (
"context"
"time"
)
// WithTimeout returns ctx with the timeout applied, or ctx itself if the
// timeout is 0 or less (no limit). Either way the returned cancel function
// must be called.
func WithTimeout(
ctx context.Context, timeout time.Duration,
) (context.Context, context.CancelFunc) {
if timeout <= 0 {
return ctx, func() {
}
}
return context.WithTimeout(ctx, timeout)
}
+20
View File
@@ -0,0 +1,20 @@
package util
import (
"context"
"testing"
"time"
)
func TestWithTimeout(t *testing.T) {
ctx, cancel := WithTimeout(context.Background(), 0)
defer cancel()
if _, ok := ctx.Deadline(); ok {
t.Error("timeout 0 set a deadline")
}
ctx, cancel = WithTimeout(context.Background(), time.Minute)
defer cancel()
if _, ok := ctx.Deadline(); !ok {
t.Error("timeout 1m set no deadline")
}
}
+26 -19
View File
@@ -11,18 +11,18 @@ import (
// DefaultWorkerShutdownTimeout is used when Worker.ShutdownTimeout is zero. // DefaultWorkerShutdownTimeout is used when Worker.ShutdownTimeout is zero.
const DefaultWorkerShutdownTimeout = 10 * time.Second const DefaultWorkerShutdownTimeout = 10 * time.Second
// ErrWorkerNoWork tells the Worker that a run succeeded but found nothing to do. // ErrWorkerNoWork tells the Worker that a run succeeded but found nothing to
// Tasks return it, or an error wrapping it, to have the run treated as a // do. Tasks return it, or an error wrapping it, to have the run treated as a
// success (checkpoint advances) while being logged distinctly. For example: // success (checkpoint advances) while being logged distinctly. For example:
// //
// var ErrNoNewReleases = fmt.Errorf("%w: no new releases", ErrWorkerNoWork) // var ErrNoNewReleases = fmt.Errorf("%w: no new releases", ErrWorkerNoWork)
var ErrWorkerNoWork = errors.New("no work") var ErrWorkerNoWork = errors.New("util: no work")
// Task processes everything that has changed since the given checkpoint. // Task processes everything that has changed since the given checkpoint.
// //
// Returning nil or an error matching ErrNoWork advances the checkpoint to the // Returning nil or an error matching ErrWorkerNoWork advances the checkpoint to
// time this run started. Any other error leaves the checkpoint unchanged, so // the time this run started. Any other error leaves the checkpoint unchanged,
// the next run retries from the same point. // so the next run retries from the same point.
type Task func(ctx context.Context, since time.Time) error type Task func(ctx context.Context, since time.Time) error
// Worker runs a Task periodically until its context is cancelled. // Worker runs a Task periodically until its context is cancelled.
@@ -32,7 +32,8 @@ type Worker struct {
// Interval is how often Task runs. Must be greater than zero. // Interval is how often Task runs. Must be greater than zero.
Interval time.Duration Interval time.Duration
// ShutdownTimeout is how long an in-flight Task may keep running after // ShutdownTimeout is how long an in-flight Task may keep running after
// the parent context is cancelled. Zero means DefaultShutdownTimeout. // the parent context is cancelled. 0 or less means
// DefaultWorkerShutdownTimeout.
ShutdownTimeout time.Duration ShutdownTimeout time.Duration
// Task is the work to perform on each tick. // Task is the work to perform on each tick.
Task Task Task Task
@@ -45,14 +46,16 @@ type Worker struct {
// cancelled. It blocks until shutdown has finished. // cancelled. It blocks until shutdown has finished.
func (w Worker) Run(ctx context.Context, since time.Time) error { func (w Worker) Run(ctx context.Context, since time.Time) error {
if w.Task == nil { if w.Task == nil {
return fmt.Errorf("worker %q: no task configured", w.Name) return fmt.Errorf("util: worker %q: no task configured", w.Name)
} }
if w.Interval <= 0 { if w.Interval <= 0 {
return fmt.Errorf( return fmt.Errorf(
"worker %q: interval must be greater than 0, got %s", "util: worker %q: interval must be greater than 0, got %s",
w.Name, w.Interval, w.Name,
w.Interval,
) )
} }
timeout := w.ShutdownTimeout timeout := w.ShutdownTimeout
if timeout <= 0 { if timeout <= 0 {
timeout = DefaultWorkerShutdownTimeout timeout = DefaultWorkerShutdownTimeout
@@ -124,7 +127,11 @@ func (w Worker) execute(
logger.Info("task completed", "elapsed", elapsed) logger.Info("task completed", "elapsed", elapsed)
return start return start
case errors.Is(err, ErrWorkerNoWork): case errors.Is(err, ErrWorkerNoWork):
logger.Info("task completed with nothing to do", "reason", err, "elapsed", elapsed) logger.Info(
"task completed with nothing to do",
"reason", err,
"elapsed", elapsed,
)
return start return start
case errors.Is(err, context.Canceled): case errors.Is(err, context.Canceled):
logger.Warn("task cancelled", "elapsed", elapsed) logger.Warn("task cancelled", "elapsed", elapsed)
@@ -142,22 +149,22 @@ func (w Worker) execute(
// The result channel is buffered, so if ctx wins the race the goroutine can // The result channel is buffered, so if ctx wins the race the goroutine can
// still deliver its result and exit instead of blocking forever. // still deliver its result and exit instead of blocking forever.
func AwaitTask[T any](ctx context.Context, task func() (T, error)) (T, error) { func AwaitTask[T any](ctx context.Context, task func() (T, error)) (T, error) {
type result struct { type outcome struct {
val T value T
err error err error
} }
results := make(chan result, 1) results := make(chan outcome, 1)
go func() { go func() {
value, err := task() value, err := task()
results <- result{ results <- outcome{
val: value, value: value,
err: err, err: err,
} }
}() }()
select { select {
case result := <-results: case result := <-results:
return result.val, result.err return result.value, result.err
case <-ctx.Done(): case <-ctx.Done():
var zero T var zero T
return zero, ctx.Err() return zero, ctx.Err()
+36 -19
View File
@@ -4,7 +4,6 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"io"
"log/slog" "log/slog"
"os" "os"
"sync" "sync"
@@ -14,7 +13,7 @@ import (
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
// Keep worker log lines out of the test output. // Keep worker log lines out of the test output.
slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, nil))) slog.SetDefault(slog.New(slog.DiscardHandler))
os.Exit(m.Run()) os.Exit(m.Run())
} }
@@ -53,10 +52,15 @@ func TestWorkerRunCancelledBeforeStart(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
cancel() cancel()
called := false called := false
w := Worker{Name: "w", Interval: time.Millisecond, Task: func(context.Context, time.Time) error { w := Worker{
called = true Name: "w",
return nil Interval: time.Millisecond,
}} Task: func(context.Context, time.Time) error {
called = true
return nil
},
}
if err := w.Run(ctx, time.Time{}); err != nil { if err := w.Run(ctx, time.Time{}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -91,12 +95,14 @@ func TestWorkerRunCheckpoints(t *testing.T) {
Task: func(_ context.Context, since time.Time) error { Task: func(_ context.Context, since time.Time) error {
mu.Lock() mu.Lock()
defer mu.Unlock() defer mu.Unlock()
i := len(seen) i := len(seen)
seen = append(seen, since) seen = append(seen, since)
starts = append(starts, time.Now()) starts = append(starts, time.Now())
if i == len(results)-1 { if i == len(results)-1 {
cancel() cancel()
} }
return results[i] return results[i]
}, },
} }
@@ -129,11 +135,20 @@ func TestWorkerRunCheckpoints(t *testing.T) {
} }
} }
if !seen[1].After(since0) || seen[1].After(starts[1]) { if !seen[1].After(since0) || seen[1].After(starts[1]) {
t.Fatalf("checkpoint after success = %v, want between %v and %v", seen[1], since0, starts[1]) t.Fatalf(
"checkpoint after success = %v, want between %v and %v",
seen[1],
since0,
starts[1],
)
} }
// Run 3 returned ErrWorkerNoWork, so run 4 sees a newer checkpoint. // Run 3 returned ErrWorkerNoWork, so run 4 sees a newer checkpoint.
if !seen[4].After(seen[3]) { if !seen[4].After(seen[3]) {
t.Fatalf("checkpoint did not advance after ErrWorkerNoWork: %v -> %v", seen[3], seen[4]) t.Fatalf(
"checkpoint did not advance after ErrWorkerNoWork: %v -> %v",
seen[3],
seen[4],
)
} }
} }
@@ -143,7 +158,9 @@ func TestWorkerRunGracefulShutdown(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
var workErr error var workErr error
w := Worker{ w := Worker{
Name: "w", Interval: time.Hour, ShutdownTimeout: time.Second, Name: "w",
Interval: time.Hour,
ShutdownTimeout: time.Second,
Task: func(taskCtx context.Context, _ time.Time) error { Task: func(taskCtx context.Context, _ time.Time) error {
cancel() cancel()
time.Sleep(10 * time.Millisecond) time.Sleep(10 * time.Millisecond)
@@ -151,6 +168,7 @@ func TestWorkerRunGracefulShutdown(t *testing.T) {
return nil return nil
}, },
} }
if err := w.Run(ctx, time.Time{}); err != nil { if err := w.Run(ctx, time.Time{}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -163,7 +181,9 @@ func TestWorkerRunGracefulShutdown(t *testing.T) {
func TestWorkerRunShutdownTimeout(t *testing.T) { func TestWorkerRunShutdownTimeout(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
w := Worker{ w := Worker{
Name: "w", Interval: time.Hour, ShutdownTimeout: 20 * time.Millisecond, Name: "w",
Interval: time.Hour,
ShutdownTimeout: 20 * time.Millisecond,
Task: func(taskCtx context.Context, _ time.Time) error { Task: func(taskCtx context.Context, _ time.Time) error {
cancel() cancel()
select { select {
@@ -174,6 +194,7 @@ func TestWorkerRunShutdownTimeout(t *testing.T) {
} }
}, },
} }
start := time.Now() start := time.Now()
if err := w.Run(ctx, time.Time{}); err != nil { if err := w.Run(ctx, time.Time{}); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -209,7 +230,9 @@ func TestWorkerExecute(t *testing.T) {
got := w.execute(context.Background(), logger, since) got := w.execute(context.Background(), logger, since)
if tc.advance { if tc.advance {
if got.Before(before) { if got.Before(before) {
t.Fatalf("checkpoint %v not advanced to run start (>= %v)", got, before) t.Fatalf(
"checkpoint %v not advanced to run start (>= %v)", got, before,
)
} }
} else if !got.Equal(since) { } else if !got.Equal(since) {
t.Fatalf("checkpoint = %v, want unchanged %v", got, since) t.Fatalf("checkpoint = %v, want unchanged %v", got, since)
@@ -261,10 +284,7 @@ func TestAwaitTaskE(t *testing.T) {
func() error { func() error {
return errFail return errFail
}, },
); !errors.Is( ); !errors.Is(err, errFail) {
err,
errFail,
) {
t.Fatalf("got %v, want %v", err, errFail) t.Fatalf("got %v, want %v", err, errFail)
} }
@@ -278,10 +298,7 @@ func TestAwaitTaskE(t *testing.T) {
<-release <-release
return nil return nil
}, },
); !errors.Is( ); !errors.Is(err, context.DeadlineExceeded) {
err,
context.DeadlineExceeded,
) {
t.Fatalf("got %v, want context.DeadlineExceeded", err) t.Fatalf("got %v, want context.DeadlineExceeded", err)
} }
} }
+18 -23
View File
@@ -12,6 +12,8 @@ import (
"github.com/coder/websocket" "github.com/coder/websocket"
"golang.org/x/time/rate" "golang.org/x/time/rate"
"git.avsdev.uk/Go/job-search/internal/util"
) )
// clientSendBuffer is how many messages are queued for a client before it // clientSendBuffer is how many messages are queued for a client before it
@@ -26,7 +28,9 @@ const logMessageLimit = 1 << 10
// *websocket.Conn, or a tcpConn for plain TCP clients (see tcp.go). // *websocket.Conn, or a tcpConn for plain TCP clients (see tcp.go).
type msgConn interface { type msgConn interface {
Read(ctx context.Context) (websocket.MessageType, []byte, error) Read(ctx context.Context) (websocket.MessageType, []byte, error)
Write(ctx context.Context, messageType websocket.MessageType, payload []byte) error Write(
ctx context.Context, messageType websocket.MessageType, payload []byte,
) error
Ping(ctx context.Context) error Ping(ctx context.Context) error
Close(code websocket.StatusCode, reason string) error Close(code websocket.StatusCode, reason string) error
CloseNow() error CloseNow() error
@@ -49,7 +53,7 @@ type client struct {
// errUnsupportedData is returned by readLoop (and a tcpConn's Write) for a // errUnsupportedData is returned by readLoop (and a tcpConn's Write) for a
// binary message. // binary message.
var errUnsupportedData = errors.New("binary frames are not supported") var errUnsupportedData = errors.New("websocket: binary frames are not supported")
// newClient creates a client for conn. cancel cancels the context the client // newClient creates a client for conn. cancel cancels the context the client
// runs with; the client calls it when it finishes or is closed. The client's // runs with; the client calls it when it finishes or is closed. The client's
@@ -59,10 +63,7 @@ func newClient(
) *client { ) *client {
limiter := rate.NewLimiter(rate.Inf, 0) limiter := rate.NewLimiter(rate.Inf, 0)
if rule := hub.conf.MessageRateLimit; rule.Enabled && rule.PerSecond > 0 { if rule := hub.conf.MessageRateLimit; rule.Enabled && rule.PerSecond > 0 {
limiter = rate.NewLimiter( limiter = rate.NewLimiter(rate.Limit(rule.PerSecond), max(rule.Burst, 1))
rate.Limit(rule.PerSecond),
max(rule.Burst, 1),
)
} }
client := &client{ client := &client{
@@ -95,7 +96,7 @@ func (c *client) run(ctx context.Context) {
c.sendEnvelope(Envelope{ c.sendEnvelope(Envelope{
Type: TypeWelcome, Type: TypeWelcome,
Payload: mustRaw(map[string]any{ Payload: rawJSON(map[string]any{
"clientId": c.id, "clientId": c.id,
}), }),
}) })
@@ -129,16 +130,20 @@ func (c *client) readLoop(ctx context.Context) error {
if err != nil { if err != nil {
return err return err
} }
c.logMessage(ctx, "message received", data) c.logMessage(ctx, "message received", data)
c.lastSeen.Store(time.Now().UnixNano()) c.lastSeen.Store(time.Now().UnixNano())
if messageType != websocket.MessageText { if messageType != websocket.MessageText {
return errUnsupportedData return errUnsupportedData
} }
// Check if the client is rate limited for early discard // Check if the client is rate limited for early discard
if !c.limiter.Allow() { if !c.limiter.Allow() {
c.sendError("", "rate limited") c.sendError("", "rate limited")
continue continue
} }
// Unpack the envelope and pass to handle to process if valid // Unpack the envelope and pass to handle to process if valid
var envelope Envelope var envelope Envelope
if err := json.Unmarshal(data, &envelope); err != nil { if err := json.Unmarshal(data, &envelope); err != nil {
@@ -157,7 +162,7 @@ func (c *client) handle(envelope Envelope) {
c.sendEnvelope(Envelope{ c.sendEnvelope(Envelope{
Type: TypePong, Type: TypePong,
ID: envelope.ID, ID: envelope.ID,
Payload: mustRaw(map[string]any{"time": time.Now().UTC()}), Payload: rawJSON(map[string]any{"time": time.Now().UTC()}),
}) })
case TypeEcho: case TypeEcho:
@@ -180,7 +185,7 @@ func (c *client) writeLoop(ctx context.Context) {
case <-ctx.Done(): case <-ctx.Done():
return return
case msg := <-c.send: case msg := <-c.send:
writeCtx, cancel := withTimeout(ctx, c.hub.conf.WriteTimeout) writeCtx, cancel := util.WithTimeout(ctx, c.hub.conf.WriteTimeout)
err := c.conn.Write(writeCtx, websocket.MessageText, msg) err := c.conn.Write(writeCtx, websocket.MessageText, msg)
cancel() cancel()
if err != nil { if err != nil {
@@ -217,7 +222,7 @@ func (c *client) pingLoop(ctx context.Context) {
if time.Now().Before(lastSeen.Add(pingInterval)) { if time.Now().Before(lastSeen.Add(pingInterval)) {
continue continue
} }
pingCtx, cancel := withTimeout(ctx, pongTimeout) pingCtx, cancel := util.WithTimeout(ctx, pongTimeout)
err := c.conn.Ping(pingCtx) err := c.conn.Ping(pingCtx)
cancel() cancel()
if err != nil { if err != nil {
@@ -283,29 +288,19 @@ func (c *client) logMessage(ctx context.Context, event string, data []byte) {
if !logger.Enabled(ctx, slog.LevelDebug) { if !logger.Enabled(ctx, slog.LevelDebug) {
return return
} }
logged, truncated := data, false logged, truncated := data, false
if len(logged) > logMessageLimit { if len(logged) > logMessageLimit {
logged, truncated = logged[:logMessageLimit], true logged, truncated = logged[:logMessageLimit], true
} }
logger.LogAttrs( logger.LogAttrs(
ctx, ctx,
slog.LevelDebug, slog.LevelDebug,
event, event, //nolint:sloglint // a constant message from the caller
slog.String("client", c.id), slog.String("client", c.id),
slog.Int("bytes", len(data)), slog.Int("bytes", len(data)),
slog.String("message", string(logged)), slog.String("message", string(logged)),
slog.Bool("truncated", truncated), slog.Bool("truncated", truncated),
) )
} }
// withTimeout returns ctx with the timeout applied, or ctx itself if the
// timeout is 0 or less (no limit).
func withTimeout(
ctx context.Context, timeout time.Duration,
) (context.Context, context.CancelFunc) {
if timeout <= 0 {
return ctx, func() {
}
}
return context.WithTimeout(ctx, timeout)
}
+19 -20
View File
@@ -34,7 +34,9 @@ func (f *fakeConn) Read(ctx context.Context) (ws.MessageType, []byte, error) {
return 0, nil, ctx.Err() return 0, nil, ctx.Err()
} }
func (f *fakeConn) Write(context.Context, ws.MessageType, []byte) error { return nil } func (f *fakeConn) Write(context.Context, ws.MessageType, []byte) error {
return nil
}
func (f *fakeConn) Ping(context.Context) error { return nil } func (f *fakeConn) Ping(context.Context) error { return nil }
@@ -102,8 +104,10 @@ func TestSendEnvelopeSlowConsumer(t *testing.T) {
func TestClientCloseOnce(t *testing.T) { func TestClientCloseOnce(t *testing.T) {
f := newFakeConn() f := newFakeConn()
c, _ := newTestClient(f) c, _ := newTestClient(f)
c.close(ws.StatusGoingAway, "bye") c.close(ws.StatusGoingAway, "bye")
c.close(ws.StatusNormalClosure, "again") c.close(ws.StatusNormalClosure, "again")
if code := <-f.closed; code != ws.StatusGoingAway { if code := <-f.closed; code != ws.StatusGoingAway {
t.Errorf("closed with %v; want going away", code) t.Errorf("closed with %v; want going away", code)
} }
@@ -147,12 +151,17 @@ func TestClientMessageRateLimit(t *testing.T) {
}) })
c.limiter.Allow() c.limiter.Allow()
if limited := !c.limiter.Allow(); limited != tt.limited { if limited := !c.limiter.Allow(); limited != tt.limited {
t.Errorf("rule %+v: second message limited = %v; want %v", tt.rule, limited, tt.limited) t.Errorf(
"rule %+v: second message limited = %v; want %v",
tt.rule,
limited,
tt.limited,
)
} }
} }
} }
func TestMustRaw(t *testing.T) { func TestRawJSON(t *testing.T) {
raw := json.RawMessage(`{"a":1}`) raw := json.RawMessage(`{"a":1}`)
tests := []struct { tests := []struct {
name string name string
@@ -166,25 +175,12 @@ func TestMustRaw(t *testing.T) {
}, ""}, }, ""},
} }
for _, tt := range tests { for _, tt := range tests {
if got := string(mustRaw(tt.v)); got != tt.want { if got := string(rawJSON(tt.v)); got != tt.want {
t.Errorf("%s: mustRaw = %q; want %q", tt.name, got, tt.want) t.Errorf("%s: rawJSON = %q; want %q", tt.name, got, tt.want)
} }
} }
} }
func TestWithTimeout(t *testing.T) {
ctx, cancel := withTimeout(context.Background(), 0)
defer cancel()
if _, ok := ctx.Deadline(); ok {
t.Error("timeout 0 set a deadline")
}
ctx, cancel = withTimeout(context.Background(), time.Minute)
defer cancel()
if _, ok := ctx.Deadline(); !ok {
t.Error("timeout 1m set no deadline")
}
}
// syncBuffer is a bytes.Buffer safe for the client's goroutines to log to. // syncBuffer is a bytes.Buffer safe for the client's goroutines to log to.
type syncBuffer struct { type syncBuffer struct {
mu sync.Mutex mu sync.Mutex
@@ -202,6 +198,7 @@ func (b *syncBuffer) entries(t *testing.T) []map[string]any {
t.Helper() t.Helper()
b.mu.Lock() b.mu.Lock()
defer b.mu.Unlock() defer b.mu.Unlock()
var entries []map[string]any var entries []map[string]any
for line := range strings.Lines(b.buf.String()) { for line := range strings.Lines(b.buf.String()) {
var e map[string]any var e map[string]any
@@ -210,6 +207,7 @@ func (b *syncBuffer) entries(t *testing.T) []map[string]any {
} }
entries = append(entries, e) entries = append(entries, e)
} }
return entries return entries
} }
@@ -283,8 +281,9 @@ func TestLogMessageLimit(t *testing.T) {
t.Fatalf("logged %d entries; want 1", len(entries)) t.Fatalf("logged %d entries; want 1", len(entries))
} }
e := entries[0] e := entries[0]
if e["message"] != long[:logMessageLimit] || e["bytes"] != float64(len(long)) || if e["message"] != long[:logMessageLimit] ||
e["truncated"] != true || e["client"] != c.id { e["bytes"] != float64(len(long)) || e["truncated"] != true ||
e["client"] != c.id {
t.Errorf("entry = %v", e) t.Errorf("entry = %v", e)
} }
+11 -2
View File
@@ -38,10 +38,12 @@ func dialWS(t *testing.T, ctx context.Context, url string) *ws.Conn {
t.Cleanup(func() { t.Cleanup(func() {
conn.CloseNow() conn.CloseNow()
}) })
var e Envelope var e Envelope
if err := wsjson.Read(ctx, conn, &e); err != nil || e.Type != TypeWelcome { if err := wsjson.Read(ctx, conn, &e); err != nil || e.Type != TypeWelcome {
t.Fatalf("want welcome, got %+v, %v", e, err) t.Fatalf("want welcome, got %+v, %v", e, err)
} }
return conn return conn
} }
@@ -66,7 +68,8 @@ func TestHTTPMessages(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
var e Envelope var e Envelope
if err := wsjson.Read(ctx, conn, &e); err != nil || e.Type != TypePong || e.ID != "1" { if err := wsjson.Read(ctx, conn, &e); err != nil || e.Type != TypePong ||
e.ID != "1" {
t.Fatalf("want pong 1, got %+v, %v", e, err) t.Fatalf("want pong 1, got %+v, %v", e, err)
} }
} }
@@ -75,6 +78,7 @@ func TestHTTPMessages(t *testing.T) {
func TestPushReachesBothTransports(t *testing.T) { func TestPushReachesBothTransports(t *testing.T) {
h, addr := tcpHub(t, nil) h, addr := tcpHub(t, nil)
url := wsHub(t, h) url := wsHub(t, h)
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel() defer cancel()
conn := dialWS(t, ctx, url) conn := dialWS(t, ctx, url)
@@ -155,7 +159,8 @@ func TestHTTPMessageRateLimit(t *testing.T) {
} }
} }
var e Envelope var e Envelope
if err := wsjson.Read(ctx, conn, &e); err != nil || e.Type != TypePong || e.ID != "1" { if err := wsjson.Read(ctx, conn, &e); err != nil || e.Type != TypePong ||
e.ID != "1" {
t.Fatalf("first: want pong 1, got %+v, %v", e, err) t.Fatalf("first: want pong 1, got %+v, %v", e, err)
} }
err := wsjson.Read(ctx, conn, &e) err := wsjson.Read(ctx, conn, &e)
@@ -173,11 +178,13 @@ func TestHTTPNotAnUpgrade(t *testing.T) {
h := NewHub(config.JSCServerWebSocket{}, WithLogger(discardLog)) h := NewHub(config.JSCServerWebSocket{}, WithLogger(discardLog))
srv := httptest.NewServer(h) srv := httptest.NewServer(h)
defer srv.Close() defer srv.Close()
res, err := http.Get(srv.URL) res, err := http.Get(srv.URL)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
res.Body.Close() res.Body.Close()
if res.StatusCode < 400 { if res.StatusCode < 400 {
t.Errorf("status = %d; want an error", res.StatusCode) t.Errorf("status = %d; want an error", res.StatusCode)
} }
@@ -194,6 +201,7 @@ func TestHTTPRefusedAfterShutdown(t *testing.T) {
if err := h.Shutdown(context.Background()); err != nil { if err := h.Shutdown(context.Background()); err != nil {
t.Fatal(err) t.Fatal(err)
} }
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel() defer cancel()
conn, _, err := ws.Dial(ctx, url, nil) conn, _, err := ws.Dial(ctx, url, nil)
@@ -201,6 +209,7 @@ func TestHTTPRefusedAfterShutdown(t *testing.T) {
t.Fatalf("dial: %v", err) t.Fatalf("dial: %v", err)
} }
defer conn.CloseNow() defer conn.CloseNow()
if _, _, err := conn.Read(ctx); ws.CloseStatus(err) != ws.StatusGoingAway { if _, _, err := conn.Read(ctx); ws.CloseStatus(err) != ws.StatusGoingAway {
t.Fatalf("read = %v; want close 1001", err) t.Fatalf("read = %v; want close 1001", err)
} }
+8 -1
View File
@@ -91,7 +91,11 @@ func (h *Hub) Push(msgType string, payload any) {
h.mu.RUnlock() h.mu.RUnlock()
for _, client := range slow { for _, client := range slow {
h.logger().Warn("dropping slow client", "client", client.id, "ip", client.ip) h.logger().Warn(
"dropping slow client",
"client", client.id,
"ip", client.ip,
)
go client.close(ws.StatusPolicyViolation, "slow consumer") go client.close(ws.StatusPolicyViolation, "slow consumer")
} }
} }
@@ -166,11 +170,14 @@ func (h *Hub) maxMessageSize() int {
func (h *Hub) register(client *client) bool { func (h *Hub) register(client *client) bool {
h.mu.Lock() h.mu.Lock()
defer h.mu.Unlock() defer h.mu.Unlock()
if h.closed { if h.closed {
return false return false
} }
h.clients[client] = struct{}{} h.clients[client] = struct{}{}
h.wg.Add(1) h.wg.Add(1)
return true return true
} }
+9 -1
View File
@@ -21,16 +21,19 @@ func pipeClient(t *testing.T, h *Hub) (net.Conn, chan struct{}) {
server.Close() server.Close()
peer.Close() peer.Close()
}) })
done := make(chan struct{}) done := make(chan struct{})
go func() { go func() {
h.ServeTCP(context.Background(), server) h.ServeTCP(context.Background(), server)
close(done) close(done)
}() }()
peer.SetReadDeadline(time.Now().Add(time.Second)) peer.SetReadDeadline(time.Now().Add(time.Second))
if _, err := bufio.NewReader(peer).ReadString('\n'); err != nil { if _, err := bufio.NewReader(peer).ReadString('\n'); err != nil {
t.Fatalf("welcome: %v", err) t.Fatalf("welcome: %v", err)
} }
waitCount(t, h, 1) waitCount(t, h, 1)
return peer, done return peer, done
} }
@@ -100,7 +103,12 @@ func TestMaxMessageSize(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
h := NewHub(tt.conf, WithLogger(discardLog)) h := NewHub(tt.conf, WithLogger(discardLog))
if got := h.maxMessageSize(); got != tt.want { if got := h.maxMessageSize(); got != tt.want {
t.Errorf("maxMessageSize(%d) = %d; want %d", tt.conf.MaxMessageSize, got, tt.want) t.Errorf(
"maxMessageSize(%d) = %d; want %d",
tt.conf.MaxMessageSize,
got,
tt.want,
)
} }
} }
} }
+3 -2
View File
@@ -55,15 +55,16 @@ const (
TypeError = "error" TypeError = "error"
) )
// mustRaw returns value as raw JSON: value itself if it is already // rawJSON returns value as raw JSON: value itself if it is already
// json.RawMessage, nil if value is nil or can't be marshalled. // json.RawMessage, nil if value is nil or can't be marshalled.
func mustRaw(value any) json.RawMessage { func rawJSON(value any) json.RawMessage {
if value == nil { if value == nil {
return nil return nil
} }
if raw, ok := value.(json.RawMessage); ok { if raw, ok := value.(json.RawMessage); ok {
return raw return raw
} }
encoded, err := json.Marshal(value) encoded, err := json.Marshal(value)
if err != nil { if err != nil {
return nil return nil
Loaded 100 of 104 files, more files were not shown because too many files have changed in this diff. Show more