Added unit tests, documentation and function descriptions across the board. Increased cyclop complexity threshold to 15 which is a bit more sane than 10 and excluded _test.go files from linters.

This commit is contained in:
bwilliams committed 2026-10-09 18:27:51 +01:00
1 parent 005b335371
commit 7fd321df97
31 files changed
+4106 -1

No files matched your search

+115
View File
@@ -0,0 +1,115 @@
# mock
Mock versions of the external services JobSearch fetches from, for local
development and testing without hitting the real APIs or their rate limits.
| Package | Mocks | Command |
|---|---|---|
| `findATender` | UK Government Find a Tender service (FTS) OCDS API | `cmd/mock-fts` |
## findATender
A mock of the FTS release package endpoint:
```
GET /api/1.0/ocdsReleasePackages?updatedFrom=2026-10-06T00:00:00&limit=5
```
It generates realistic, fictional OCDS releases from a catalog of buyers,
suppliers and tender subjects, keeps publishing new ones while it runs, and
behaves like the real service: paging, query validation and rate limiting.
The JobSearch `fts_mock` source points at it.
### Running
```bash
go run ./cmd/mock-fts --addr :8080
```
Then enable the `fts_mock` source in `config.yaml`:
```yaml
sources:
fts_mock:
enabled: true
interval: 1m
api_root: http://localhost:8080/api/1.0
```
Useful flags (`go run ./cmd/mock-fts --help` lists them all):
| Flag | Default | Meaning |
|---|---|---|
| `--addr` | `:8080` | Listen address |
| `--base-url` | `http://localhost:8080` | Public base URL used in `uri` and `links.next` |
| `--data-dir` | `./mock-data` | Where `releases.jsonl` is stored |
| `--reset` | | Delete stored releases and regenerate the backfill |
| `--backfill` | `72h` | History to generate on an empty data directory |
| `--min-gap`, `--max-gap` | `30s`, `5m` | Range of time between generated releases |
| `--rate-limit`, `--rate-window` | `12`, `5m` | Requests allowed per client per window |
| `--access-log` | `-` (stdout) | Access log file |
| `--log-format` | `combined` | Access log format: `combined` or `common` |
| `--tz` | `Europe/London` | Time zone for query times with no offset |
For quick testing, shorten the gaps and raise the rate limit:
```bash
go run ./cmd/mock-fts --min-gap 5s --max-gap 10s --rate-limit 1000
```
### API behaviour
Query parameters, as on the real service:
| Parameter | Format | Meaning |
|---|---|---|
| `updatedFrom` | `yyyy-MM-ddTHH:mm:ss` | Releases dated at or after this time |
| `updatedTo` | `yyyy-MM-ddTHH:mm:ss` | Releases dated at or before this time (optional) |
| `limit` | `1`–`100`, default `100` | Page size |
| `cursor` | opaque | Next page; taken from `links.next` |
- Like FTS, query times are UK local time (BST in summer) unless they carry an
explicit offset or `Z`. Release dates are stored and returned in UTC.
- Results are in date order. When there are more, `links.next` holds the URL
of the next page, with the original query and a `cursor`.
- Releases dated after the current time are never returned.
- Invalid parameters get `400 Bad Request` with a message, and methods other
than `GET` get `405 Method Not Allowed`.
- Each client (by remote IP) may make `--rate-limit` requests per
`--rate-window`. Windows are fixed and start at the client's first request
in the current clock hour. Over the limit the mock returns
`429 Too Many Requests` with a `Retry-After` header and the body
`Rate limit of N exceeded. Please retry after X seconds.`
### Generated data
- **Catalog** (`findATender/data/catalog.json`): the buyers, suppliers,
tender subjects, procurement methods and classifications that releases are
drawn from. It's embedded in the binary, so rebuild after changing it.
Subjects refer to classifications by `{scheme}-{id}` keys (e.g.
`CPV-90513000`). The mock refuses to start if the catalog has an empty pool
or a key that doesn't resolve.
- **Procedures**: a new release usually starts a new procedure, with a new
`ocds-mock-` ocid, as a `tender` (or sometimes `planning`). About a third of
the time it moves an open procedure on instead, from `planning` to `tender`,
`tender` to `tenderUpdate` (a deadline extension) or `award`. So the same
ocid appears in several releases, as it does on the real service.
- **Store** (`<data-dir>/releases.jsonl`): every release is appended as one
line of JSON, so releases survive restarts. On startup the mock reloads the
file, skips any corrupt lines, and generates releases to fill the time since
the last one (or `--backfill`, if that's shorter).
### Code
| File | Contents |
|---|---|
| `doc.go` | Package documentation and data file names |
| `catalog.go` | Loading and validating the embedded catalog |
| `generator.go` | Making releases, catching up on startup, and the background publisher |
| `store.go` | The in-memory, file-backed release store |
| `handler.go` | The HTTP handler: query parsing, paging and cursors |
| `ratelimit.go` | The per-client rate limiter and its middleware |
| `ocds/` | Generated OCDS types (`ocds_gen.go`, from `ocdsgen`); don't edit by hand |
| `testdata/` | A small catalog used by the tests |
The mock's `ocds` package is separate from `internal/findATender/ocds`, which
the JobSearch client uses.
+5
View File
@@ -73,6 +73,9 @@ func parseCatalog(b []byte, name string) (*Catalog, error) {
return &c, nil
}
// validate checks that every pool has at least one entry and that every
// classification key used by a subject or its items is in Classifications.
// It reports all the problems found, listing at most five missing keys.
func (c *Catalog) validate() error {
var errs []error
@@ -156,6 +159,8 @@ func (c *Catalog) subject(r *rand.Rand) ocds.Tender {
return t
}
// resolveOne returns a copy of the classification with key, or nil if there
// is none.
func (c *Catalog) resolveOne(key string) *ocds.Classification {
if cl, ok := c.Classifications[key]; ok {
return &cl
+134
View File
@@ -0,0 +1,134 @@
package mock
import (
"math/rand"
"os"
"strings"
"testing"
)
// loadTestCatalog parses the small fixture catalog used by the tests.
func loadTestCatalog(t *testing.T) *Catalog {
t.Helper()
b, err := os.ReadFile("testdata/catalog.json")
if err != nil {
t.Fatal(err)
}
cat, err := parseCatalog(b, "testdata/catalog.json")
if err != nil {
t.Fatal(err)
}
return cat
}
// The catalog compiled into the build must load: a bad data/catalog.json
// fails here rather than when the mock starts.
func TestEmbeddedCatalogIsValid(t *testing.T) {
if len(embeddedCatalog) == 0 {
t.Fatal("data/catalog.json was not embedded")
}
cat, err := LoadCatalog()
if err != nil {
t.Fatal(err)
}
if len(cat.Buyers) == 0 || len(cat.Suppliers) == 0 || len(cat.Subjects) == 0 ||
len(cat.Methods) == 0 {
t.Fatalf("embedded catalog has an empty pool: %d buyers, %d suppliers, %d subjects, %d methods",
len(cat.Buyers), len(cat.Suppliers), len(cat.Subjects), len(cat.Methods))
}
}
func TestParseCatalog(t *testing.T) {
cat := loadTestCatalog(t)
if len(cat.Buyers) != 2 || len(cat.Suppliers) != 1 || len(cat.Subjects) != 2 ||
len(cat.Methods) != 2 {
t.Fatalf("got %d buyers, %d suppliers, %d subjects, %d methods",
len(cat.Buyers), len(cat.Suppliers), len(cat.Subjects), len(cat.Methods))
}
if d := cat.Buyers[0].Details; d == nil || len(d.Classifications) != 1 ||
d.Classifications[0].Scheme != "UK_CA_TYPE" {
t.Fatalf("buyer details.classifications not read: %+v", d)
}
s := cat.Subjects[0]
if s.Classification != "CPV-77314000" || len(s.Items) != 1 {
t.Fatalf("subject classification key/items not read: %+v", s)
}
if ac := s.Items[0].AdditionalClassifications; len(ac) != 1 || ac[0] != "CPV-77310000" {
t.Fatalf("items.additionalClassifications keys not read: %+v", ac)
}
if v := s.Lots[0].Value; v == nil || v.AmountGross == nil || *v.AmountGross != 120000 {
t.Fatalf("lot value not read: %+v", v)
}
}
// Picking a subject resolves its classification keys to full classifications.
func TestSubjectResolvesClassifications(t *testing.T) {
cat := loadTestCatalog(t)
//nolint:gosec // G404: seeded math/rand keeps the test deterministic
r := rand.New(rand.NewSource(1))
for i := 0; i < 20; i++ {
tender := cat.subject(r)
if tender.Title != "Grounds maintenance services" {
continue
}
if cl := tender.Classification; cl == nil || cl.ID != "77314000" ||
cl.Description != "Grounds maintenance services" {
t.Fatalf("main classification = %+v", cl)
}
ac := tender.Items[0].AdditionalClassifications
if len(ac) != 1 || ac[0].Description != "Planting and maintenance services of green areas" {
t.Fatalf("item classifications = %+v", ac)
}
// The catalog itself still holds keys, untouched by picking.
if cat.Subjects[0].Classification != "CPV-77314000" {
t.Fatal("picking modified the catalog")
}
return
}
t.Fatal("never picked the grounds maintenance subject")
}
func TestParseCatalogRejectsInvalidCatalogs(t *testing.T) {
cases := map[string]struct {
doc string
want []string
}{
"empty pools": {
doc: `{"buyers": [], "subjects": [{"title": "T"}], "methods": [], "classifications": {}}`,
want: []string{"no buyers", "no suppliers", "no methods"},
},
"unknown classification keys": {
doc: `{"buyers": [{"name": "B"}], "suppliers": [{"name": "S"}], "methods": [{"procurementMethod": "open"}],
"subjects": [{"title": "T", "classification": "CPV-1", "items": [{"id": "1", "additionalClassifications": ["CPV-2"]}]}],
"classifications": {}}`,
want: []string{"2 classification key(s) not found", "CPV-1", "CPV-2"},
},
"not JSON": {doc: `{`, want: []string{"decoding"}},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
_, err := parseCatalog([]byte(tc.doc), "test catalog")
if err == nil {
t.Fatal("expected an error")
}
for _, w := range tc.want {
if !strings.Contains(err.Error(), w) {
t.Errorf("error %q does not mention %q", err, w)
}
}
})
}
}
func TestResolveOneUnknownKey(t *testing.T) {
cat := loadTestCatalog(t)
if cl := cat.resolveOne("CPV-00000000"); cl != nil {
t.Errorf("unknown key resolved to %+v", cl)
}
if cl := cat.resolveOne(""); cl != nil {
t.Errorf("empty key resolved to %+v", cl)
}
if got := cat.resolve([]string{"CPV-00000000"}); got != nil {
t.Errorf("resolve of unknown keys = %+v", got)
}
}
+10
View File
@@ -23,6 +23,8 @@ type Generator struct {
rnd *rand.Rand
}
// NewGenerator returns a generator drawing from cat and storing releases in
// store. The same seed and store contents give the same releases.
func NewGenerator(cat *Catalog, store *Store, seed int64) *Generator {
return &Generator{
cat: cat,
@@ -31,6 +33,9 @@ func NewGenerator(cat *Catalog, store *Store, seed int64) *Generator {
}
}
// Make returns a new release dated at (to the second). About a third of the
// time it moves on a procedure still open from the last 60 days; otherwise it
// starts a new one.
func (g *Generator) Make(at time.Time) ocds.Release {
at = at.UTC().Truncate(time.Second)
if g.rnd.Float64() < 0.35 {
@@ -105,6 +110,9 @@ func (g *Generator) newProcedure(at time.Time) ocds.Release {
}
}
// advance returns the next release in prev's procedure: planning moves to
// tender, and a tender either has its deadline extended (tenderUpdate) or is
// awarded to a supplier from the catalog.
func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release {
r := clone(prev)
r.ID = g.store.NextNoticeID(at.Year())
@@ -190,6 +198,8 @@ func (g *Generator) newOCID() string {
}
}
// gap returns a random duration from min to max inclusive, or min if max is
// not larger.
func (g *Generator) gap(min, max time.Duration) time.Duration {
if max <= min {
return min
+215
View File
@@ -0,0 +1,215 @@
package mock
import (
"context"
"io"
"log"
"math/rand"
"path/filepath"
"strings"
"testing"
"time"
"git.avsdev.uk/Go/job-search/internal/mock/findATender/ocds"
)
func quietLog(t *testing.T) {
t.Helper()
w := log.Writer()
log.SetOutput(io.Discard)
t.Cleanup(func() { log.SetOutput(w) })
}
func TestGap(t *testing.T) {
//nolint:gosec // G404: seeded math/rand keeps the test deterministic
g := &Generator{rnd: rand.New(rand.NewSource(1))}
if d := g.gap(time.Second, time.Second); d != time.Second {
t.Errorf("min == max: %s", d)
}
if d := g.gap(2*time.Second, time.Second); d != 2*time.Second {
t.Errorf("max < min: %s, want min", d)
}
for i := 0; i < 1000; i++ {
if d := g.gap(time.Second, 2*time.Second); d < time.Second || d > 2*time.Second {
t.Fatalf("gap out of range: %s", d)
}
}
}
func TestScaleValue(t *testing.T) {
scaleValue(nil, 2) // must not panic
v := &ocds.Value{Amount: ptr(12345.0), Currency: "GBP"}
scaleValue(v, 1.5)
if *v.Amount != 18500 { // 18517.5 rounded to the nearest 100
t.Errorf("amount = %v, want 18500", *v.Amount)
}
if v.AmountGross != nil {
t.Error("nil gross amount was set")
}
v = &ocds.Value{Amount: ptr(1000.0), AmountGross: ptr(1200.0)}
scaleValue(v, 0.5)
if *v.Amount != 500 || *v.AmountGross != 600 {
t.Errorf("amounts = %v / %v, want 500 / 600", *v.Amount, *v.AmountGross)
}
}
func TestEnsurePartyID(t *testing.T) {
p := ocds.Organization{ID: "GB-EXISTING", Name: "Council"}
ensurePartyID(&p)
if p.ID != "GB-EXISTING" {
t.Errorf("existing id replaced: %s", p.ID)
}
a := ocds.Organization{Name: "Somewhere Council"}
b := ocds.Organization{Name: "Somewhere Council"}
c := ocds.Organization{Name: "Elsewhere Council"}
ensurePartyID(&a)
ensurePartyID(&b)
ensurePartyID(&c)
if !strings.HasPrefix(a.ID, "GB-FTS-") {
t.Errorf("id = %q", a.ID)
}
if a.ID != b.ID {
t.Errorf("same name gave %s and %s", a.ID, b.ID)
}
if a.ID == c.ID {
t.Errorf("different names gave the same id %s", a.ID)
}
}
func TestAdvance(t *testing.T) {
cat := loadTestCatalog(t)
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
g := NewGenerator(cat, store, 1)
at := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
// A planning notice without a tender becomes an active tender.
r := g.advance(ocds.Release{OCID: "ocds-x", Tag: []string{"planning"}}, at)
if r.Tag[0] != "tender" || r.Tender == nil || r.Tender.Status != "active" ||
r.Tender.TenderPeriod == nil || r.Tender.TenderPeriod.EndDate == nil {
t.Fatalf("planning -> %v, tender %+v", r.Tag, r.Tender)
}
if string(r.Tender.ID) != "ocds-x" || !r.Date.Equal(at) {
t.Errorf("tender id %s, date %s", r.Tender.ID, r.Date)
}
// A tender is either updated (deadline + 14 days) or awarded.
end := at.AddDate(0, 0, 20)
prev := ocds.Release{
OCID: "ocds-y",
Tag: []string{"tender"},
Tender: &ocds.Tender{
ID: "ocds-y",
Value: &ocds.Value{Amount: ptr(10000.0), Currency: "GBP"},
Lots: []ocds.Lot{{ID: "1"}, {ID: ""}},
TenderPeriod: &ocds.Period{EndDate: &end},
},
}
seen := map[string]bool{}
for i := 0; i < 50; i++ {
r := g.advance(prev, at)
seen[r.Tag[0]] = true
switch r.Tag[0] {
case "tenderUpdate":
if got := *r.Tender.TenderPeriod.EndDate; !got.Equal(end.AddDate(0, 0, 14)) {
t.Fatalf("updated deadline %s", got)
}
case "award":
if r.Tender.Status != "complete" || len(r.Awards) != 1 {
t.Fatalf("award: status %s, %d awards", r.Tender.Status, len(r.Awards))
}
aw := r.Awards[0]
if aw.Value == nil || *aw.Value.Amount < 8000 || *aw.Value.Amount > 10000 {
t.Fatalf("award value %+v", aw.Value)
}
if len(aw.RelatedLots) != 1 || aw.RelatedLots[0] != "1" {
t.Fatalf("related lots %v", aw.RelatedLots)
}
if len(aw.Suppliers) != 1 || len(r.Parties) != 1 || r.Parties[0].Roles[0] != "supplier" {
t.Fatalf("suppliers %v, parties %v", aw.Suppliers, r.Parties)
}
default:
t.Fatalf("tender -> %s", r.Tag[0])
}
}
if !seen["tenderUpdate"] || !seen["award"] {
t.Errorf("expected both updates and awards, saw %v", seen)
}
if *prev.Tender.TenderPeriod.EndDate != end || prev.Awards != nil {
t.Error("advance modified the previous release")
}
// An update without a deadline gets 30 + 14 days; a tender with no value
// is awarded with no value.
prev = ocds.Release{OCID: "ocds-z", Tag: []string{"tender"}, Tender: &ocds.Tender{ID: "ocds-z"}}
for i := 0; i < 50; i++ {
r := g.advance(prev, at)
switch r.Tag[0] {
case "tenderUpdate":
if got := *r.Tender.TenderPeriod.EndDate; !got.Equal(at.AddDate(0, 0, 44)) {
t.Fatalf("default deadline %s", got)
}
case "award":
if r.Awards[0].Value != nil {
t.Fatalf("award value %+v, want nil", r.Awards[0].Value)
}
}
}
}
func TestRunPublishesUntilCancelled(t *testing.T) {
quietLog(t)
cat := loadTestCatalog(t)
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
NewGenerator(cat, store, 1).Run(ctx, time.Millisecond, 2*time.Millisecond)
close(done)
}()
deadline := time.After(5 * time.Second)
for store.Len() < 3 {
select {
case <-deadline:
cancel()
t.Fatalf("only %d releases published", store.Len())
case <-time.After(time.Millisecond):
}
}
cancel()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("Run did not return after cancel")
}
}
// A failing store is logged and Run carries on until cancelled.
func TestRunSurvivesStoreErrors(t *testing.T) {
quietLog(t)
cat := loadTestCatalog(t)
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
if err != nil {
t.Fatal(err)
}
store.Close()
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
NewGenerator(cat, store, 1).Run(ctx, time.Millisecond, time.Millisecond)
if store.Len() != 0 {
t.Errorf("Len = %d, want 0", store.Len())
}
}
+2
View File
@@ -43,6 +43,8 @@ func encodeCursor(i int) string {
return base64.RawURLEncoding.EncodeToString([]byte("idx:" + strconv.Itoa(i)))
}
// decodeCursor returns the release index in a cursor made by encodeCursor.
// Anything else, including a negative index, is an error.
func decodeCursor(c string) (int, error) {
b, err := base64.RawURLEncoding.DecodeString(c)
if err != nil || !strings.HasPrefix(string(b), "idx:") {
+209
View File
@@ -0,0 +1,209 @@
package mock
import (
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"git.avsdev.uk/Go/job-search/internal/mock/findATender/ocds"
)
func TestUpdatedFromIsLocalTime(t *testing.T) {
london, err := time.LoadLocation("Europe/London")
if err != nil {
t.Fatal(err)
}
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
// Two releases at 11:45Z and 12:15Z, i.e. 12:45 and 13:15 BST.
for i, at := range []time.Time{
time.Date(2026, 10, 8, 11, 45, 0, 0, time.UTC),
time.Date(2026, 10, 8, 12, 15, 0, 0, time.UTC),
} {
if err := store.Append(
ocds.Release{
OCID: "ocds-test",
ID: []string{"000001-2026", "000002-2026"}[i],
Date: at,
Tag: []string{"tender"},
},
); err != nil {
t.Fatal(err)
}
}
h := NewHandler(store, "", london)
count := func(query string) int {
rec := httptest.NewRecorder()
h(rec, httptest.NewRequest("GET", BasePath+"?"+query, nil))
var pkg ocds.ReleasePackage
if err := json.Unmarshal(rec.Body.Bytes(), &pkg); err != nil {
t.Fatalf("%s: %v (%s)", query, err, rec.Body.String())
}
return len(pkg.Releases)
}
cases := map[string]int{
"updatedFrom=2026-10-08T13:00:00": 1, // 13:00 BST = 12:00Z
"updatedFrom=2026-10-08T12:30:00": 2, // 12:30 BST = 11:30Z
"updatedFrom=2026-10-08T12:00:00Z": 1, // explicit UTC still honoured
"updatedFrom=2026-10-08T12:00:00%2B01:00": 2, // explicit offset still honoured
"updatedFrom=2026-10-08T12:00:00&updatedTo=2026-10-08T13:00:00": 1, // window in BST
}
for q, want := range cases {
if got := count(q); got != want {
t.Errorf("%s: got %d releases, want %d", q, got, want)
}
}
// In winter (GMT) London local time equals UTC.
winter := time.Date(2026, 12, 1, 9, 0, 0, 0, time.UTC)
if got, _ := parseTime("2026-12-01T09:00:00", london); !got.Equal(winter) {
t.Errorf("GMT: got %s, want %s", got, winter)
}
}
func openTestStore(t *testing.T) *Store {
t.Helper()
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { store.Close() })
return store
}
func TestHandlerRejectsBadRequests(t *testing.T) {
h := NewHandler(openTestStore(t), "", nil)
rec := httptest.NewRecorder()
h(rec, httptest.NewRequest("POST", BasePath, nil))
if rec.Code != http.StatusMethodNotAllowed {
t.Errorf("POST: status %d, want 405", rec.Code)
}
cursor := func(s string) string { return base64.RawURLEncoding.EncodeToString([]byte(s)) }
cases := map[string]string{
"limit=0": "limit must be an integer between 1 and 100",
"limit=101": "limit must be an integer between 1 and 100",
"limit=ten": "limit must be an integer between 1 and 100",
"updatedFrom=yesterday": "updatedFrom must be in format yyyy-MM-ddTHH:mm:ss",
"updatedTo=2026-13-01": "updatedTo must be in format yyyy-MM-ddTHH:mm:ss",
"updatedFrom=2026-10-08T12:00:00&updatedTo=2026-10-08T11:00:00": "updatedTo must not be before updatedFrom",
"cursor=!!!": "invalid cursor",
"cursor=" + cursor("pos:1"): "invalid cursor",
"cursor=" + cursor("idx:one"): "invalid cursor",
"cursor=" + cursor("idx:-1"): "invalid cursor",
}
for q, want := range cases {
rec := httptest.NewRecorder()
h(rec, httptest.NewRequest("GET", BasePath+"?"+q, nil))
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status %d, want 400", q, rec.Code)
continue
}
if ct := rec.Header().Get("Content-Type"); ct != "application/json" {
t.Errorf("%s: Content-Type %q", q, ct)
}
var body map[string]string
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil || body["error"] != want {
t.Errorf("%s: body %s, want error %q", q, rec.Body.String(), want)
}
}
}
func TestCursorRoundTrip(t *testing.T) {
for _, i := range []int{0, 1, 99, 123456} {
if got, err := decodeCursor(encodeCursor(i)); err != nil || got != i {
t.Errorf("decodeCursor(encodeCursor(%d)) = %d, %v", i, got, err)
}
}
}
func TestParseTimeFormats(t *testing.T) {
cases := map[string]time.Time{
"2026-10-08T12:00:00Z": time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC),
"2026-10-08T12:00:00+02:00": time.Date(2026, 10, 8, 10, 0, 0, 0, time.UTC),
"2026-10-08T12:00:00": time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC),
"2026-10-08": time.Date(2026, 10, 8, 0, 0, 0, 0, time.UTC),
}
for v, want := range cases {
got, err := parseTime(v, nil) // nil location means UTC
if err != nil || !got.Equal(want) {
t.Errorf("parseTime(%q) = %s, %v; want %s", v, got, err, want)
}
}
for _, v := range []string{"", "08/10/2026", "2026-10-08 12:00:00", "2026-10-08T25:00:00"} {
if _, err := parseTime(v, nil); err == nil {
t.Errorf("parseTime(%q): expected error", v)
}
}
}
// Following links.next walks every release exactly once, keeping the other
// query parameters, and the last page has no next link.
func TestHandlerPagination(t *testing.T) {
store := openTestStore(t)
base := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
for i := 0; i < 5; i++ {
r := ocds.Release{
OCID: fmt.Sprintf("ocds-test-%d", i),
ID: fmt.Sprintf("%06d-2026", i+1),
Date: base.Add(time.Duration(i) * time.Minute),
Tag: []string{"tender"},
}
if err := store.Append(r); err != nil {
t.Fatal(err)
}
}
const baseURL = "http://mock.test"
h := NewHandler(store, baseURL, nil)
next := BasePath + "?limit=2&updatedFrom=2026-10-08T09:01:00"
var ids []string
for pages := 0; next != ""; pages++ {
if pages > 5 {
t.Fatal("too many pages")
}
rec := httptest.NewRecorder()
h(rec, httptest.NewRequest("GET", next, nil))
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d: %s", next, rec.Code, rec.Body.String())
}
var pkg ocds.ReleasePackage
if err := json.Unmarshal(rec.Body.Bytes(), &pkg); err != nil {
t.Fatal(err)
}
if pkg.URI != baseURL+next {
t.Errorf("uri = %q, want %q", pkg.URI, baseURL+next)
}
for _, r := range pkg.Releases {
ids = append(ids, r.ID)
}
next = ""
if pkg.Links != nil && pkg.Links.Next != "" {
if !strings.HasPrefix(pkg.Links.Next, baseURL+BasePath+"?") {
t.Fatalf("next = %q", pkg.Links.Next)
}
if !strings.Contains(pkg.Links.Next, "limit=2") ||
!strings.Contains(pkg.Links.Next, "updatedFrom=") {
t.Errorf("next lost query parameters: %q", pkg.Links.Next)
}
next = strings.TrimPrefix(pkg.Links.Next, baseURL)
}
}
want := "000002-2026,000003-2026,000004-2026,000005-2026"
if got := strings.Join(ids, ","); got != want {
t.Errorf("walked %s, want %s", got, want)
}
}
+134
View File
@@ -0,0 +1,134 @@
package mock
import (
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"time"
)
func TestRateLimitWindows(t *testing.T) {
now := time.Date(2026, 10, 6, 10, 3, 20, 0, time.UTC) // first request of the hour
rl := NewRateLimiter(12, 5*time.Minute)
rl.now = func() time.Time { return now }
for i := 0; i < 12; i++ {
if ok, _ := rl.allow("a"); !ok {
t.Fatalf("request %d rejected", i+1)
}
}
now = now.Add(100 * time.Second) // 10:05:00, window ends 10:08:20
if ok, secs := rl.allow("a"); ok || secs != 200 {
t.Fatalf("want reject with 200s, got ok=%v secs=%d", ok, secs)
}
if ok, _ := rl.allow("b"); !ok {
t.Fatal("other client should be unaffected")
}
now = time.Date(2026, 10, 6, 10, 8, 20, 0, time.UTC) // next window opens
if ok, _ := rl.allow("a"); !ok {
t.Fatal("new window should allow")
}
// Window 10:58:20-11:03:20 is cut short by the new hour at 11:00:00.
now = time.Date(2026, 10, 6, 10, 58, 30, 0, time.UTC)
for i := 0; i < 12; i++ {
rl.allow("a")
}
if ok, secs := rl.allow("a"); ok || secs != 90 {
t.Fatalf("want reject with 90s, got ok=%v secs=%d", ok, secs)
}
now = time.Date(2026, 10, 6, 11, 0, 5, 0, time.UTC) // re-anchors here
if ok, _ := rl.allow("a"); !ok {
t.Fatal("new hour should allow")
}
}
func TestClientKey(t *testing.T) {
cases := map[string]string{
"192.0.2.1:1234": "192.0.2.1",
"[::1]:8080": "::1",
"no-port": "no-port", // not host:port, used as-is
}
for addr, want := range cases {
req := httptest.NewRequest("GET", "/", nil)
req.RemoteAddr = addr
if got := clientKey(req); got != want {
t.Errorf("clientKey(%q) = %q, want %q", addr, got, want)
}
}
}
func TestRateLimitMiddleware(t *testing.T) {
now := time.Date(2026, 10, 6, 10, 0, 0, 0, time.UTC)
rl := NewRateLimiter(2, 5*time.Minute)
rl.now = func() time.Time { return now }
calls := 0
h := rl.Middleware(func(w http.ResponseWriter, req *http.Request) {
calls++
w.WriteHeader(http.StatusOK)
})
do := func(addr string) *httptest.ResponseRecorder {
req := httptest.NewRequest("GET", "/", nil)
req.RemoteAddr = addr
rec := httptest.NewRecorder()
h(rec, req)
return rec
}
for i := 0; i < 2; i++ {
if rec := do("192.0.2.1:1000"); rec.Code != http.StatusOK {
t.Fatalf("request %d: status %d", i+1, rec.Code)
}
}
// Same host, different port: still the same client.
rec := do("192.0.2.1:2000")
if rec.Code != http.StatusTooManyRequests {
t.Fatalf("status %d, want 429", rec.Code)
}
if got := rec.Header().Get("Retry-After"); got != "300" {
t.Errorf("Retry-After = %q, want 300", got)
}
if !strings.Contains(rec.Body.String(), "Rate limit of 2 exceeded") ||
!strings.Contains(rec.Body.String(), "300 seconds") {
t.Errorf("body = %q", rec.Body.String())
}
if calls != 2 {
t.Errorf("next called %d times, want 2", calls)
}
if rec := do("192.0.2.2:1000"); rec.Code != http.StatusOK {
t.Errorf("other client: status %d", rec.Code)
}
}
// Once more than 10000 clients are tracked, a new client in a new hour drops
// every client left over from earlier hours.
func TestRateLimitPrunesStaleClients(t *testing.T) {
now := time.Date(2026, 10, 6, 10, 30, 0, 0, time.UTC)
rl := NewRateLimiter(1, time.Minute)
rl.now = func() time.Time { return now }
for i := 0; i <= 10000; i++ {
rl.allow(strconv.Itoa(i))
}
if len(rl.clients) != 10001 {
t.Fatalf("tracking %d clients", len(rl.clients))
}
// Same hour: nothing is stale, so nothing is dropped.
rl.allow("same-hour")
if len(rl.clients) != 10002 {
t.Fatalf("same hour: tracking %d clients, want 10002", len(rl.clients))
}
now = now.Add(time.Hour)
rl.allow("new")
if len(rl.clients) != 1 {
t.Fatalf("after prune: tracking %d clients, want 1", len(rl.clients))
}
if _, ok := rl.clients["new"]; !ok {
t.Fatal("new client was pruned")
}
}
+11
View File
@@ -34,6 +34,10 @@ type Store struct {
openPos map[string]int
}
// OpenStore loads the releases stored at path, creating the file and its
// directory if needed, and opens it for appending. Lines that can't be read
// are skipped, releases are put back in date order if needed, and a record
// left incomplete by a crash is ended so new ones start on a fresh line.
func OpenStore(path string) (*Store, error) {
s := &Store{latest: map[string]int{}, lastNum: map[int]int{}, openPos: map[string]int{}}
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
@@ -115,6 +119,9 @@ func openForAppend(path string) (*os.File, error) {
return f, nil
}
// index records release r, at position i, in the lookups: the ocid's latest
// release, whether its procedure is still open (an award closes it), and the
// highest notice number used each year.
func (s *Store) index(i int, r ocds.Release) {
s.latest[r.OCID] = i
if len(r.Tag) > 0 && r.Tag[0] == "award" {
@@ -132,6 +139,7 @@ func (s *Store) index(i int, r ocds.Release) {
}
}
// closeProcedure removes ocid from the open procedures, if it is there.
func (s *Store) closeProcedure(ocid string) {
pos, ok := s.openPos[ocid]
if !ok {
@@ -151,6 +159,7 @@ func (s *Store) Append(r ocds.Release) error { return s.append(r, true) }
// Used for bulk catch-up, where a sync per record would be very slow.
func (s *Store) AppendNoSync(r ocds.Release) error { return s.append(r, false) }
// Sync flushes appended releases to disk. Call it after AppendNoSync.
func (s *Store) Sync() error {
s.mu.Lock()
defer s.mu.Unlock()
@@ -204,6 +213,8 @@ func (s *Store) Page(from, upper time.Time, start, limit int) ([]ocds.Release, i
return out, i, more
}
// Last returns the date of the newest release, or false if the store is
// empty.
func (s *Store) Last() (time.Time, bool) {
s.mu.RLock()
defer s.mu.RUnlock()
+269
View File
@@ -0,0 +1,269 @@
package mock
import (
"bytes"
"encoding/json"
"math/rand"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"git.avsdev.uk/Go/job-search/internal/mock/findATender/ocds"
)
func TestStorePersistsAndGeneratorProgresses(t *testing.T) {
cat := loadTestCatalog(t)
path := filepath.Join(t.TempDir(), "releases.jsonl")
store, err := OpenStore(path)
if err != nil {
t.Fatal(err)
}
gen := NewGenerator(cat, store, 42)
now := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
n, err := gen.CatchUp(now, 48*time.Hour, 3*time.Minute, 15*time.Minute)
if err != nil || n < 48*60/15 {
t.Fatalf("catch-up generated %d (err %v)", n, err)
}
tags := map[string]int{}
ocids := map[string]int{}
all, _, _ := store.Page(time.Time{}, now, 0, 100000)
for _, r := range all {
tags[r.Tag[0]]++
ocids[r.OCID]++
if r.Tender == nil || r.Tender.Classification == nil {
t.Fatalf("release %s has no tender.classification", r.ID)
}
}
if tags["award"] == 0 || tags["tender"] == 0 {
t.Fatalf("expected tenders and awards, got %v", tags)
}
if len(ocids) == len(all) {
t.Fatal("expected some procedures with more than one release")
}
store.Close()
// Reopen: everything is still there, and catch-up only fills the new gap.
store2, err := OpenStore(path)
if err != nil {
t.Fatal(err)
}
defer store2.Close()
if store2.Len() != len(all) {
t.Fatalf("after reopen: %d releases, want %d", store2.Len(), len(all))
}
gen2 := NewGenerator(cat, store2, 7)
m, _ := gen2.CatchUp(now.Add(time.Hour), 48*time.Hour, 3*time.Minute, 15*time.Minute)
if m < 4 || m > 20 {
t.Fatalf("one hour gap produced %d releases", m)
}
if last, _ := store2.Last(); last.After(now.Add(time.Hour)) {
t.Fatal("release dated in the future")
}
// The HTTP handler pages through the store without gaps or repeats.
srv := &server{store: store2, baseURL: "", loc: time.UTC}
seen := map[string]bool{}
next := BasePath + "?updatedFrom=2026-10-07T00:00:00&limit=7"
for pages := 0; next != ""; pages++ {
rec := httptest.NewRecorder()
srv.handle(rec, httptest.NewRequest("GET", next, nil))
// Strict decode: the output must contain only fields the ocds types define.
var pkg ocds.ReleasePackage
dec := json.NewDecoder(bytes.NewReader(rec.Body.Bytes()))
dec.DisallowUnknownFields()
if err := dec.Decode(&pkg); err != nil {
t.Fatalf("response doesn't match ocds types: %v", err)
}
for _, r := range pkg.Releases {
if seen[r.ID] {
t.Fatalf("release %s returned twice", r.ID)
}
seen[r.ID] = true
}
next = ""
if pkg.Links != nil {
next = pkg.Links.Next
}
}
from := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
want, _, _ := store2.Page(from, time.Now(), 0, 100000)
if len(seen) != len(want) {
t.Fatalf("paged %d releases, want %d", len(seen), len(want))
}
}
func writeLines(t *testing.T, path string, lines ...string) {
t.Helper()
if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")), 0o600); err != nil {
t.Fatal(err)
}
}
func releaseLine(t *testing.T, ocid, id, tag string, at time.Time) string {
t.Helper()
b, err := json.Marshal(ocds.Release{OCID: ocid, ID: id, Date: at, Tag: []string{tag}})
if err != nil {
t.Fatal(err)
}
return string(b)
}
// Unreadable lines are skipped, blank lines ignored, a hand-edited file is put
// back in date order, and the index (open procedures, notice numbers) is rebuilt.
func TestOpenStoreRecoversFromMessyFile(t *testing.T) {
path := filepath.Join(t.TempDir(), "r.jsonl")
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
writeLines(t, path,
releaseLine(t, "ocds-b", "000007-2026", "tender", t0.Add(2*time.Hour)),
"{not json",
"",
releaseLine(t, "ocds-a", "000003-2026", "tender", t0),
releaseLine(t, "ocds-a", "000005-2026", "award", t0.Add(time.Hour)),
"",
)
store, err := OpenStore(path)
if err != nil {
t.Fatal(err)
}
defer store.Close()
all, _, _ := store.Page(time.Time{}, t0.Add(24*time.Hour), 0, 100)
var ids []string
for _, r := range all {
ids = append(ids, r.ID)
}
if got, want := strings.Join(ids, ","), "000003-2026,000005-2026,000007-2026"; got != want {
t.Fatalf("releases %s, want %s", got, want)
}
if got := store.NextNoticeID(2026); got != "000008-2026" {
t.Errorf("NextNoticeID = %s, want 000008-2026", got)
}
if got := store.NextNoticeID(2027); got != "000001-2027" {
t.Errorf("NextNoticeID(2027) = %s", got)
}
// ocds-a was awarded, so only ocds-b is still open.
//nolint:gosec // G404: seeded math/rand keeps the test deterministic
r, ok := store.RandomOpen(rand.New(rand.NewSource(1)), t0)
if !ok || r.OCID != "ocds-b" {
t.Errorf("RandomOpen = %s, %v; want ocds-b", r.OCID, ok)
}
}
// A record cut off mid-write is skipped, and the next record starts on a fresh
// line so it is still readable after a restart.
func TestOpenStoreAfterTruncatedWrite(t *testing.T) {
path := filepath.Join(t.TempDir(), "r.jsonl")
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
full := releaseLine(t, "ocds-a", "000001-2026", "tender", t0)
cut := releaseLine(t, "ocds-b", "000002-2026", "tender", t0.Add(time.Minute))
writeLines(t, path, full, cut[:len(cut)/2]) // no trailing newline
store, err := OpenStore(path)
if err != nil {
t.Fatal(err)
}
if store.Len() != 1 {
t.Fatalf("Len = %d, want 1", store.Len())
}
if err := store.Append(
ocds.Release{
OCID: "ocds-c",
ID: "000002-2026",
Date: t0.Add(2 * time.Minute),
Tag: []string{"tender"},
},
); err != nil {
t.Fatal(err)
}
store.Close()
store2, err := OpenStore(path)
if err != nil {
t.Fatal(err)
}
defer store2.Close()
if store2.Len() != 2 || !store2.HasOCID("ocds-c") {
t.Fatalf("after reopen: Len = %d, has ocds-c = %v", store2.Len(), store2.HasOCID("ocds-c"))
}
}
func TestOpenStoreErrors(t *testing.T) {
dir := t.TempDir()
// The parent "directory" is a regular file.
file := filepath.Join(dir, "file")
if err := os.WriteFile(file, nil, 0o600); err != nil {
t.Fatal(err)
}
if _, err := OpenStore(filepath.Join(file, "r.jsonl")); err == nil {
t.Error("parent is a file: expected error")
}
// The store path is a directory, which opens but can't be read.
if _, err := OpenStore(dir); err == nil {
t.Error("path is a directory: expected error")
}
// A missing parent directory is created.
store, err := OpenStore(filepath.Join(dir, "a", "b", "r.jsonl"))
if err != nil {
t.Fatalf("nested path: %v", err)
}
store.Close()
}
// A release dated before the newest one is moved up to keep the order valid.
func TestAppendKeepsDateOrder(t *testing.T) {
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
store.Append(ocds.Release{OCID: "a", ID: "000001-2026", Date: t0, Tag: []string{"tender"}})
store.Append(
ocds.Release{OCID: "b", ID: "000002-2026", Date: t0.Add(-time.Hour), Tag: []string{"tender"}},
)
if last, _ := store.Last(); !last.Equal(t0) {
t.Errorf("Last = %s, want %s", last, t0)
}
// Appending to a closed store fails and leaves the store unchanged.
store.Close()
if err := store.Append(ocds.Release{OCID: "c", Date: t0, Tag: []string{"tender"}}); err == nil {
t.Error("append after close: expected error")
}
if store.Len() != 2 || store.HasOCID("c") {
t.Errorf("failed append changed the store: Len = %d", store.Len())
}
}
// Procedures idle since before the cut-off are dropped from the open set.
func TestRandomOpenDropsStaleProcedures(t *testing.T) {
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
store.Append(ocds.Release{OCID: "old", ID: "000001-2026", Date: t0, Tag: []string{"tender"}})
store.Append(
ocds.Release{OCID: "new", ID: "000002-2026", Date: t0.Add(time.Hour), Tag: []string{"tender"}},
)
//nolint:gosec // G404: seeded math/rand keeps the test deterministic
rnd := rand.New(rand.NewSource(1))
for i := 0; i < 10; i++ {
if r, ok := store.RandomOpen(rnd, t0.Add(time.Minute)); !ok || r.OCID != "new" {
t.Fatalf("RandomOpen = %s, %v; want new", r.OCID, ok)
}
}
if _, ok := store.RandomOpen(rnd, t0.Add(2*time.Hour)); ok {
t.Error("everything is stale: expected none")
}
}