Added unit tests, documentation and function descriptions across the board. Increased cyclop complexity threshold to 15 which is a bit more sane than 10 and excluded _test.go files from linters.
This commit is contained in:
1 parent
005b335371
commit
7fd321df97
31 files changed
+4106
-1
No files matched your search
@@ -0,0 +1,115 @@
|
||||
# mock
|
||||
|
||||
Mock versions of the external services JobSearch fetches from, for local
|
||||
development and testing without hitting the real APIs or their rate limits.
|
||||
|
||||
| Package | Mocks | Command |
|
||||
|---|---|---|
|
||||
| `findATender` | UK Government Find a Tender service (FTS) OCDS API | `cmd/mock-fts` |
|
||||
|
||||
|
||||
## findATender
|
||||
A mock of the FTS release package endpoint:
|
||||
|
||||
```
|
||||
GET /api/1.0/ocdsReleasePackages?updatedFrom=2026-10-06T00:00:00&limit=5
|
||||
```
|
||||
|
||||
It generates realistic, fictional OCDS releases from a catalog of buyers,
|
||||
suppliers and tender subjects, keeps publishing new ones while it runs, and
|
||||
behaves like the real service: paging, query validation and rate limiting.
|
||||
The JobSearch `fts_mock` source points at it.
|
||||
|
||||
|
||||
### Running
|
||||
```bash
|
||||
go run ./cmd/mock-fts --addr :8080
|
||||
```
|
||||
|
||||
Then enable the `fts_mock` source in `config.yaml`:
|
||||
```yaml
|
||||
sources:
|
||||
fts_mock:
|
||||
enabled: true
|
||||
interval: 1m
|
||||
api_root: http://localhost:8080/api/1.0
|
||||
```
|
||||
|
||||
Useful flags (`go run ./cmd/mock-fts --help` lists them all):
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `--addr` | `:8080` | Listen address |
|
||||
| `--base-url` | `http://localhost:8080` | Public base URL used in `uri` and `links.next` |
|
||||
| `--data-dir` | `./mock-data` | Where `releases.jsonl` is stored |
|
||||
| `--reset` | | Delete stored releases and regenerate the backfill |
|
||||
| `--backfill` | `72h` | History to generate on an empty data directory |
|
||||
| `--min-gap`, `--max-gap` | `30s`, `5m` | Range of time between generated releases |
|
||||
| `--rate-limit`, `--rate-window` | `12`, `5m` | Requests allowed per client per window |
|
||||
| `--access-log` | `-` (stdout) | Access log file |
|
||||
| `--log-format` | `combined` | Access log format: `combined` or `common` |
|
||||
| `--tz` | `Europe/London` | Time zone for query times with no offset |
|
||||
|
||||
For quick testing, shorten the gaps and raise the rate limit:
|
||||
```bash
|
||||
go run ./cmd/mock-fts --min-gap 5s --max-gap 10s --rate-limit 1000
|
||||
```
|
||||
|
||||
|
||||
### API behaviour
|
||||
Query parameters, as on the real service:
|
||||
|
||||
| Parameter | Format | Meaning |
|
||||
|---|---|---|
|
||||
| `updatedFrom` | `yyyy-MM-ddTHH:mm:ss` | Releases dated at or after this time |
|
||||
| `updatedTo` | `yyyy-MM-ddTHH:mm:ss` | Releases dated at or before this time (optional) |
|
||||
| `limit` | `1`–`100`, default `100` | Page size |
|
||||
| `cursor` | opaque | Next page; taken from `links.next` |
|
||||
|
||||
- Like FTS, query times are UK local time (BST in summer) unless they carry an
|
||||
explicit offset or `Z`. Release dates are stored and returned in UTC.
|
||||
- Results are in date order. When there are more, `links.next` holds the URL
|
||||
of the next page, with the original query and a `cursor`.
|
||||
- Releases dated after the current time are never returned.
|
||||
- Invalid parameters get `400 Bad Request` with a message, and methods other
|
||||
than `GET` get `405 Method Not Allowed`.
|
||||
- Each client (by remote IP) may make `--rate-limit` requests per
|
||||
`--rate-window`. Windows are fixed and start at the client's first request
|
||||
in the current clock hour. Over the limit the mock returns
|
||||
`429 Too Many Requests` with a `Retry-After` header and the body
|
||||
`Rate limit of N exceeded. Please retry after X seconds.`
|
||||
|
||||
|
||||
### Generated data
|
||||
- **Catalog** (`findATender/data/catalog.json`): the buyers, suppliers,
|
||||
tender subjects, procurement methods and classifications that releases are
|
||||
drawn from. It's embedded in the binary, so rebuild after changing it.
|
||||
Subjects refer to classifications by `{scheme}-{id}` keys (e.g.
|
||||
`CPV-90513000`). The mock refuses to start if the catalog has an empty pool
|
||||
or a key that doesn't resolve.
|
||||
- **Procedures**: a new release usually starts a new procedure, with a new
|
||||
`ocds-mock-` ocid, as a `tender` (or sometimes `planning`). About a third of
|
||||
the time it moves an open procedure on instead, from `planning` to `tender`,
|
||||
`tender` to `tenderUpdate` (a deadline extension) or `award`. So the same
|
||||
ocid appears in several releases, as it does on the real service.
|
||||
- **Store** (`<data-dir>/releases.jsonl`): every release is appended as one
|
||||
line of JSON, so releases survive restarts. On startup the mock reloads the
|
||||
file, skips any corrupt lines, and generates releases to fill the time since
|
||||
the last one (or `--backfill`, if that's shorter).
|
||||
|
||||
|
||||
### Code
|
||||
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| `doc.go` | Package documentation and data file names |
|
||||
| `catalog.go` | Loading and validating the embedded catalog |
|
||||
| `generator.go` | Making releases, catching up on startup, and the background publisher |
|
||||
| `store.go` | The in-memory, file-backed release store |
|
||||
| `handler.go` | The HTTP handler: query parsing, paging and cursors |
|
||||
| `ratelimit.go` | The per-client rate limiter and its middleware |
|
||||
| `ocds/` | Generated OCDS types (`ocds_gen.go`, from `ocdsgen`); don't edit by hand |
|
||||
| `testdata/` | A small catalog used by the tests |
|
||||
|
||||
The mock's `ocds` package is separate from `internal/findATender/ocds`, which
|
||||
the JobSearch client uses.
|
||||
@@ -73,6 +73,9 @@ func parseCatalog(b []byte, name string) (*Catalog, error) {
|
||||
return &c, nil
|
||||
}
|
||||
|
||||
// validate checks that every pool has at least one entry and that every
|
||||
// classification key used by a subject or its items is in Classifications.
|
||||
// It reports all the problems found, listing at most five missing keys.
|
||||
func (c *Catalog) validate() error {
|
||||
var errs []error
|
||||
|
||||
@@ -156,6 +159,8 @@ func (c *Catalog) subject(r *rand.Rand) ocds.Tender {
|
||||
return t
|
||||
}
|
||||
|
||||
// resolveOne returns a copy of the classification with key, or nil if there
|
||||
// is none.
|
||||
func (c *Catalog) resolveOne(key string) *ocds.Classification {
|
||||
if cl, ok := c.Classifications[key]; ok {
|
||||
return &cl
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
package mock
|
||||
|
||||
import (
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// loadTestCatalog parses the small fixture catalog used by the tests.
|
||||
func loadTestCatalog(t *testing.T) *Catalog {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile("testdata/catalog.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cat, err := parseCatalog(b, "testdata/catalog.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return cat
|
||||
}
|
||||
|
||||
// The catalog compiled into the build must load: a bad data/catalog.json
|
||||
// fails here rather than when the mock starts.
|
||||
func TestEmbeddedCatalogIsValid(t *testing.T) {
|
||||
if len(embeddedCatalog) == 0 {
|
||||
t.Fatal("data/catalog.json was not embedded")
|
||||
}
|
||||
cat, err := LoadCatalog()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(cat.Buyers) == 0 || len(cat.Suppliers) == 0 || len(cat.Subjects) == 0 ||
|
||||
len(cat.Methods) == 0 {
|
||||
t.Fatalf("embedded catalog has an empty pool: %d buyers, %d suppliers, %d subjects, %d methods",
|
||||
len(cat.Buyers), len(cat.Suppliers), len(cat.Subjects), len(cat.Methods))
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCatalog(t *testing.T) {
|
||||
cat := loadTestCatalog(t)
|
||||
if len(cat.Buyers) != 2 || len(cat.Suppliers) != 1 || len(cat.Subjects) != 2 ||
|
||||
len(cat.Methods) != 2 {
|
||||
t.Fatalf("got %d buyers, %d suppliers, %d subjects, %d methods",
|
||||
len(cat.Buyers), len(cat.Suppliers), len(cat.Subjects), len(cat.Methods))
|
||||
}
|
||||
if d := cat.Buyers[0].Details; d == nil || len(d.Classifications) != 1 ||
|
||||
d.Classifications[0].Scheme != "UK_CA_TYPE" {
|
||||
t.Fatalf("buyer details.classifications not read: %+v", d)
|
||||
}
|
||||
s := cat.Subjects[0]
|
||||
if s.Classification != "CPV-77314000" || len(s.Items) != 1 {
|
||||
t.Fatalf("subject classification key/items not read: %+v", s)
|
||||
}
|
||||
if ac := s.Items[0].AdditionalClassifications; len(ac) != 1 || ac[0] != "CPV-77310000" {
|
||||
t.Fatalf("items.additionalClassifications keys not read: %+v", ac)
|
||||
}
|
||||
if v := s.Lots[0].Value; v == nil || v.AmountGross == nil || *v.AmountGross != 120000 {
|
||||
t.Fatalf("lot value not read: %+v", v)
|
||||
}
|
||||
}
|
||||
|
||||
// Picking a subject resolves its classification keys to full classifications.
|
||||
func TestSubjectResolvesClassifications(t *testing.T) {
|
||||
cat := loadTestCatalog(t)
|
||||
//nolint:gosec // G404: seeded math/rand keeps the test deterministic
|
||||
r := rand.New(rand.NewSource(1))
|
||||
for i := 0; i < 20; i++ {
|
||||
tender := cat.subject(r)
|
||||
if tender.Title != "Grounds maintenance services" {
|
||||
continue
|
||||
}
|
||||
if cl := tender.Classification; cl == nil || cl.ID != "77314000" ||
|
||||
cl.Description != "Grounds maintenance services" {
|
||||
t.Fatalf("main classification = %+v", cl)
|
||||
}
|
||||
ac := tender.Items[0].AdditionalClassifications
|
||||
if len(ac) != 1 || ac[0].Description != "Planting and maintenance services of green areas" {
|
||||
t.Fatalf("item classifications = %+v", ac)
|
||||
}
|
||||
// The catalog itself still holds keys, untouched by picking.
|
||||
if cat.Subjects[0].Classification != "CPV-77314000" {
|
||||
t.Fatal("picking modified the catalog")
|
||||
}
|
||||
return
|
||||
}
|
||||
t.Fatal("never picked the grounds maintenance subject")
|
||||
}
|
||||
|
||||
func TestParseCatalogRejectsInvalidCatalogs(t *testing.T) {
|
||||
cases := map[string]struct {
|
||||
doc string
|
||||
want []string
|
||||
}{
|
||||
"empty pools": {
|
||||
doc: `{"buyers": [], "subjects": [{"title": "T"}], "methods": [], "classifications": {}}`,
|
||||
want: []string{"no buyers", "no suppliers", "no methods"},
|
||||
},
|
||||
"unknown classification keys": {
|
||||
doc: `{"buyers": [{"name": "B"}], "suppliers": [{"name": "S"}], "methods": [{"procurementMethod": "open"}],
|
||||
"subjects": [{"title": "T", "classification": "CPV-1", "items": [{"id": "1", "additionalClassifications": ["CPV-2"]}]}],
|
||||
"classifications": {}}`,
|
||||
want: []string{"2 classification key(s) not found", "CPV-1", "CPV-2"},
|
||||
},
|
||||
"not JSON": {doc: `{`, want: []string{"decoding"}},
|
||||
}
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
_, err := parseCatalog([]byte(tc.doc), "test catalog")
|
||||
if err == nil {
|
||||
t.Fatal("expected an error")
|
||||
}
|
||||
for _, w := range tc.want {
|
||||
if !strings.Contains(err.Error(), w) {
|
||||
t.Errorf("error %q does not mention %q", err, w)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveOneUnknownKey(t *testing.T) {
|
||||
cat := loadTestCatalog(t)
|
||||
if cl := cat.resolveOne("CPV-00000000"); cl != nil {
|
||||
t.Errorf("unknown key resolved to %+v", cl)
|
||||
}
|
||||
if cl := cat.resolveOne(""); cl != nil {
|
||||
t.Errorf("empty key resolved to %+v", cl)
|
||||
}
|
||||
if got := cat.resolve([]string{"CPV-00000000"}); got != nil {
|
||||
t.Errorf("resolve of unknown keys = %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -23,6 +23,8 @@ type Generator struct {
|
||||
rnd *rand.Rand
|
||||
}
|
||||
|
||||
// NewGenerator returns a generator drawing from cat and storing releases in
|
||||
// store. The same seed and store contents give the same releases.
|
||||
func NewGenerator(cat *Catalog, store *Store, seed int64) *Generator {
|
||||
return &Generator{
|
||||
cat: cat,
|
||||
@@ -31,6 +33,9 @@ func NewGenerator(cat *Catalog, store *Store, seed int64) *Generator {
|
||||
}
|
||||
}
|
||||
|
||||
// Make returns a new release dated at (to the second). About a third of the
|
||||
// time it moves on a procedure still open from the last 60 days; otherwise it
|
||||
// starts a new one.
|
||||
func (g *Generator) Make(at time.Time) ocds.Release {
|
||||
at = at.UTC().Truncate(time.Second)
|
||||
if g.rnd.Float64() < 0.35 {
|
||||
@@ -105,6 +110,9 @@ func (g *Generator) newProcedure(at time.Time) ocds.Release {
|
||||
}
|
||||
}
|
||||
|
||||
// advance returns the next release in prev's procedure: planning moves to
|
||||
// tender, and a tender either has its deadline extended (tenderUpdate) or is
|
||||
// awarded to a supplier from the catalog.
|
||||
func (g *Generator) advance(prev ocds.Release, at time.Time) ocds.Release {
|
||||
r := clone(prev)
|
||||
r.ID = g.store.NextNoticeID(at.Year())
|
||||
@@ -190,6 +198,8 @@ func (g *Generator) newOCID() string {
|
||||
}
|
||||
}
|
||||
|
||||
// gap returns a random duration from min to max inclusive, or min if max is
|
||||
// not larger.
|
||||
func (g *Generator) gap(min, max time.Duration) time.Duration {
|
||||
if max <= min {
|
||||
return min
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
package mock
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log"
|
||||
"math/rand"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.avsdev.uk/Go/job-search/internal/mock/findATender/ocds"
|
||||
)
|
||||
|
||||
func quietLog(t *testing.T) {
|
||||
t.Helper()
|
||||
w := log.Writer()
|
||||
log.SetOutput(io.Discard)
|
||||
t.Cleanup(func() { log.SetOutput(w) })
|
||||
}
|
||||
|
||||
func TestGap(t *testing.T) {
|
||||
//nolint:gosec // G404: seeded math/rand keeps the test deterministic
|
||||
g := &Generator{rnd: rand.New(rand.NewSource(1))}
|
||||
if d := g.gap(time.Second, time.Second); d != time.Second {
|
||||
t.Errorf("min == max: %s", d)
|
||||
}
|
||||
if d := g.gap(2*time.Second, time.Second); d != 2*time.Second {
|
||||
t.Errorf("max < min: %s, want min", d)
|
||||
}
|
||||
for i := 0; i < 1000; i++ {
|
||||
if d := g.gap(time.Second, 2*time.Second); d < time.Second || d > 2*time.Second {
|
||||
t.Fatalf("gap out of range: %s", d)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestScaleValue(t *testing.T) {
|
||||
scaleValue(nil, 2) // must not panic
|
||||
|
||||
v := &ocds.Value{Amount: ptr(12345.0), Currency: "GBP"}
|
||||
scaleValue(v, 1.5)
|
||||
if *v.Amount != 18500 { // 18517.5 rounded to the nearest 100
|
||||
t.Errorf("amount = %v, want 18500", *v.Amount)
|
||||
}
|
||||
if v.AmountGross != nil {
|
||||
t.Error("nil gross amount was set")
|
||||
}
|
||||
|
||||
v = &ocds.Value{Amount: ptr(1000.0), AmountGross: ptr(1200.0)}
|
||||
scaleValue(v, 0.5)
|
||||
if *v.Amount != 500 || *v.AmountGross != 600 {
|
||||
t.Errorf("amounts = %v / %v, want 500 / 600", *v.Amount, *v.AmountGross)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsurePartyID(t *testing.T) {
|
||||
p := ocds.Organization{ID: "GB-EXISTING", Name: "Council"}
|
||||
ensurePartyID(&p)
|
||||
if p.ID != "GB-EXISTING" {
|
||||
t.Errorf("existing id replaced: %s", p.ID)
|
||||
}
|
||||
|
||||
a := ocds.Organization{Name: "Somewhere Council"}
|
||||
b := ocds.Organization{Name: "Somewhere Council"}
|
||||
c := ocds.Organization{Name: "Elsewhere Council"}
|
||||
ensurePartyID(&a)
|
||||
ensurePartyID(&b)
|
||||
ensurePartyID(&c)
|
||||
if !strings.HasPrefix(a.ID, "GB-FTS-") {
|
||||
t.Errorf("id = %q", a.ID)
|
||||
}
|
||||
if a.ID != b.ID {
|
||||
t.Errorf("same name gave %s and %s", a.ID, b.ID)
|
||||
}
|
||||
if a.ID == c.ID {
|
||||
t.Errorf("different names gave the same id %s", a.ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdvance(t *testing.T) {
|
||||
cat := loadTestCatalog(t)
|
||||
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
g := NewGenerator(cat, store, 1)
|
||||
at := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
// A planning notice without a tender becomes an active tender.
|
||||
r := g.advance(ocds.Release{OCID: "ocds-x", Tag: []string{"planning"}}, at)
|
||||
if r.Tag[0] != "tender" || r.Tender == nil || r.Tender.Status != "active" ||
|
||||
r.Tender.TenderPeriod == nil || r.Tender.TenderPeriod.EndDate == nil {
|
||||
t.Fatalf("planning -> %v, tender %+v", r.Tag, r.Tender)
|
||||
}
|
||||
if string(r.Tender.ID) != "ocds-x" || !r.Date.Equal(at) {
|
||||
t.Errorf("tender id %s, date %s", r.Tender.ID, r.Date)
|
||||
}
|
||||
|
||||
// A tender is either updated (deadline + 14 days) or awarded.
|
||||
end := at.AddDate(0, 0, 20)
|
||||
prev := ocds.Release{
|
||||
OCID: "ocds-y",
|
||||
Tag: []string{"tender"},
|
||||
Tender: &ocds.Tender{
|
||||
ID: "ocds-y",
|
||||
Value: &ocds.Value{Amount: ptr(10000.0), Currency: "GBP"},
|
||||
Lots: []ocds.Lot{{ID: "1"}, {ID: ""}},
|
||||
TenderPeriod: &ocds.Period{EndDate: &end},
|
||||
},
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i := 0; i < 50; i++ {
|
||||
r := g.advance(prev, at)
|
||||
seen[r.Tag[0]] = true
|
||||
switch r.Tag[0] {
|
||||
case "tenderUpdate":
|
||||
if got := *r.Tender.TenderPeriod.EndDate; !got.Equal(end.AddDate(0, 0, 14)) {
|
||||
t.Fatalf("updated deadline %s", got)
|
||||
}
|
||||
case "award":
|
||||
if r.Tender.Status != "complete" || len(r.Awards) != 1 {
|
||||
t.Fatalf("award: status %s, %d awards", r.Tender.Status, len(r.Awards))
|
||||
}
|
||||
aw := r.Awards[0]
|
||||
if aw.Value == nil || *aw.Value.Amount < 8000 || *aw.Value.Amount > 10000 {
|
||||
t.Fatalf("award value %+v", aw.Value)
|
||||
}
|
||||
if len(aw.RelatedLots) != 1 || aw.RelatedLots[0] != "1" {
|
||||
t.Fatalf("related lots %v", aw.RelatedLots)
|
||||
}
|
||||
if len(aw.Suppliers) != 1 || len(r.Parties) != 1 || r.Parties[0].Roles[0] != "supplier" {
|
||||
t.Fatalf("suppliers %v, parties %v", aw.Suppliers, r.Parties)
|
||||
}
|
||||
default:
|
||||
t.Fatalf("tender -> %s", r.Tag[0])
|
||||
}
|
||||
}
|
||||
if !seen["tenderUpdate"] || !seen["award"] {
|
||||
t.Errorf("expected both updates and awards, saw %v", seen)
|
||||
}
|
||||
if *prev.Tender.TenderPeriod.EndDate != end || prev.Awards != nil {
|
||||
t.Error("advance modified the previous release")
|
||||
}
|
||||
|
||||
// An update without a deadline gets 30 + 14 days; a tender with no value
|
||||
// is awarded with no value.
|
||||
prev = ocds.Release{OCID: "ocds-z", Tag: []string{"tender"}, Tender: &ocds.Tender{ID: "ocds-z"}}
|
||||
for i := 0; i < 50; i++ {
|
||||
r := g.advance(prev, at)
|
||||
switch r.Tag[0] {
|
||||
case "tenderUpdate":
|
||||
if got := *r.Tender.TenderPeriod.EndDate; !got.Equal(at.AddDate(0, 0, 44)) {
|
||||
t.Fatalf("default deadline %s", got)
|
||||
}
|
||||
case "award":
|
||||
if r.Awards[0].Value != nil {
|
||||
t.Fatalf("award value %+v, want nil", r.Awards[0].Value)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunPublishesUntilCancelled(t *testing.T) {
|
||||
quietLog(t)
|
||||
cat := loadTestCatalog(t)
|
||||
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
NewGenerator(cat, store, 1).Run(ctx, time.Millisecond, 2*time.Millisecond)
|
||||
close(done)
|
||||
}()
|
||||
|
||||
deadline := time.After(5 * time.Second)
|
||||
for store.Len() < 3 {
|
||||
select {
|
||||
case <-deadline:
|
||||
cancel()
|
||||
t.Fatalf("only %d releases published", store.Len())
|
||||
case <-time.After(time.Millisecond):
|
||||
}
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("Run did not return after cancel")
|
||||
}
|
||||
}
|
||||
|
||||
// A failing store is logged and Run carries on until cancelled.
|
||||
func TestRunSurvivesStoreErrors(t *testing.T) {
|
||||
quietLog(t)
|
||||
cat := loadTestCatalog(t)
|
||||
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store.Close()
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
||||
defer cancel()
|
||||
NewGenerator(cat, store, 1).Run(ctx, time.Millisecond, time.Millisecond)
|
||||
if store.Len() != 0 {
|
||||
t.Errorf("Len = %d, want 0", store.Len())
|
||||
}
|
||||
}
|
||||
@@ -43,6 +43,8 @@ func encodeCursor(i int) string {
|
||||
return base64.RawURLEncoding.EncodeToString([]byte("idx:" + strconv.Itoa(i)))
|
||||
}
|
||||
|
||||
// decodeCursor returns the release index in a cursor made by encodeCursor.
|
||||
// Anything else, including a negative index, is an error.
|
||||
func decodeCursor(c string) (int, error) {
|
||||
b, err := base64.RawURLEncoding.DecodeString(c)
|
||||
if err != nil || !strings.HasPrefix(string(b), "idx:") {
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
package mock
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.avsdev.uk/Go/job-search/internal/mock/findATender/ocds"
|
||||
)
|
||||
|
||||
func TestUpdatedFromIsLocalTime(t *testing.T) {
|
||||
london, err := time.LoadLocation("Europe/London")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
// Two releases at 11:45Z and 12:15Z, i.e. 12:45 and 13:15 BST.
|
||||
for i, at := range []time.Time{
|
||||
time.Date(2026, 10, 8, 11, 45, 0, 0, time.UTC),
|
||||
time.Date(2026, 10, 8, 12, 15, 0, 0, time.UTC),
|
||||
} {
|
||||
if err := store.Append(
|
||||
ocds.Release{
|
||||
OCID: "ocds-test",
|
||||
ID: []string{"000001-2026", "000002-2026"}[i],
|
||||
Date: at,
|
||||
Tag: []string{"tender"},
|
||||
},
|
||||
); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
h := NewHandler(store, "", london)
|
||||
count := func(query string) int {
|
||||
rec := httptest.NewRecorder()
|
||||
h(rec, httptest.NewRequest("GET", BasePath+"?"+query, nil))
|
||||
var pkg ocds.ReleasePackage
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &pkg); err != nil {
|
||||
t.Fatalf("%s: %v (%s)", query, err, rec.Body.String())
|
||||
}
|
||||
return len(pkg.Releases)
|
||||
}
|
||||
|
||||
cases := map[string]int{
|
||||
"updatedFrom=2026-10-08T13:00:00": 1, // 13:00 BST = 12:00Z
|
||||
"updatedFrom=2026-10-08T12:30:00": 2, // 12:30 BST = 11:30Z
|
||||
"updatedFrom=2026-10-08T12:00:00Z": 1, // explicit UTC still honoured
|
||||
"updatedFrom=2026-10-08T12:00:00%2B01:00": 2, // explicit offset still honoured
|
||||
"updatedFrom=2026-10-08T12:00:00&updatedTo=2026-10-08T13:00:00": 1, // window in BST
|
||||
}
|
||||
for q, want := range cases {
|
||||
if got := count(q); got != want {
|
||||
t.Errorf("%s: got %d releases, want %d", q, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// In winter (GMT) London local time equals UTC.
|
||||
winter := time.Date(2026, 12, 1, 9, 0, 0, 0, time.UTC)
|
||||
if got, _ := parseTime("2026-12-01T09:00:00", london); !got.Equal(winter) {
|
||||
t.Errorf("GMT: got %s, want %s", got, winter)
|
||||
}
|
||||
}
|
||||
|
||||
func openTestStore(t *testing.T) *Store {
|
||||
t.Helper()
|
||||
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { store.Close() })
|
||||
return store
|
||||
}
|
||||
|
||||
func TestHandlerRejectsBadRequests(t *testing.T) {
|
||||
h := NewHandler(openTestStore(t), "", nil)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h(rec, httptest.NewRequest("POST", BasePath, nil))
|
||||
if rec.Code != http.StatusMethodNotAllowed {
|
||||
t.Errorf("POST: status %d, want 405", rec.Code)
|
||||
}
|
||||
|
||||
cursor := func(s string) string { return base64.RawURLEncoding.EncodeToString([]byte(s)) }
|
||||
cases := map[string]string{
|
||||
"limit=0": "limit must be an integer between 1 and 100",
|
||||
"limit=101": "limit must be an integer between 1 and 100",
|
||||
"limit=ten": "limit must be an integer between 1 and 100",
|
||||
"updatedFrom=yesterday": "updatedFrom must be in format yyyy-MM-ddTHH:mm:ss",
|
||||
"updatedTo=2026-13-01": "updatedTo must be in format yyyy-MM-ddTHH:mm:ss",
|
||||
"updatedFrom=2026-10-08T12:00:00&updatedTo=2026-10-08T11:00:00": "updatedTo must not be before updatedFrom",
|
||||
"cursor=!!!": "invalid cursor",
|
||||
"cursor=" + cursor("pos:1"): "invalid cursor",
|
||||
"cursor=" + cursor("idx:one"): "invalid cursor",
|
||||
"cursor=" + cursor("idx:-1"): "invalid cursor",
|
||||
}
|
||||
for q, want := range cases {
|
||||
rec := httptest.NewRecorder()
|
||||
h(rec, httptest.NewRequest("GET", BasePath+"?"+q, nil))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("%s: status %d, want 400", q, rec.Code)
|
||||
continue
|
||||
}
|
||||
if ct := rec.Header().Get("Content-Type"); ct != "application/json" {
|
||||
t.Errorf("%s: Content-Type %q", q, ct)
|
||||
}
|
||||
var body map[string]string
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil || body["error"] != want {
|
||||
t.Errorf("%s: body %s, want error %q", q, rec.Body.String(), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCursorRoundTrip(t *testing.T) {
|
||||
for _, i := range []int{0, 1, 99, 123456} {
|
||||
if got, err := decodeCursor(encodeCursor(i)); err != nil || got != i {
|
||||
t.Errorf("decodeCursor(encodeCursor(%d)) = %d, %v", i, got, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseTimeFormats(t *testing.T) {
|
||||
cases := map[string]time.Time{
|
||||
"2026-10-08T12:00:00Z": time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC),
|
||||
"2026-10-08T12:00:00+02:00": time.Date(2026, 10, 8, 10, 0, 0, 0, time.UTC),
|
||||
"2026-10-08T12:00:00": time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC),
|
||||
"2026-10-08": time.Date(2026, 10, 8, 0, 0, 0, 0, time.UTC),
|
||||
}
|
||||
for v, want := range cases {
|
||||
got, err := parseTime(v, nil) // nil location means UTC
|
||||
if err != nil || !got.Equal(want) {
|
||||
t.Errorf("parseTime(%q) = %s, %v; want %s", v, got, err, want)
|
||||
}
|
||||
}
|
||||
for _, v := range []string{"", "08/10/2026", "2026-10-08 12:00:00", "2026-10-08T25:00:00"} {
|
||||
if _, err := parseTime(v, nil); err == nil {
|
||||
t.Errorf("parseTime(%q): expected error", v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Following links.next walks every release exactly once, keeping the other
|
||||
// query parameters, and the last page has no next link.
|
||||
func TestHandlerPagination(t *testing.T) {
|
||||
store := openTestStore(t)
|
||||
base := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
|
||||
for i := 0; i < 5; i++ {
|
||||
r := ocds.Release{
|
||||
OCID: fmt.Sprintf("ocds-test-%d", i),
|
||||
ID: fmt.Sprintf("%06d-2026", i+1),
|
||||
Date: base.Add(time.Duration(i) * time.Minute),
|
||||
Tag: []string{"tender"},
|
||||
}
|
||||
if err := store.Append(r); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
const baseURL = "http://mock.test"
|
||||
h := NewHandler(store, baseURL, nil)
|
||||
next := BasePath + "?limit=2&updatedFrom=2026-10-08T09:01:00"
|
||||
var ids []string
|
||||
for pages := 0; next != ""; pages++ {
|
||||
if pages > 5 {
|
||||
t.Fatal("too many pages")
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
h(rec, httptest.NewRequest("GET", next, nil))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("%s: status %d: %s", next, rec.Code, rec.Body.String())
|
||||
}
|
||||
var pkg ocds.ReleasePackage
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &pkg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pkg.URI != baseURL+next {
|
||||
t.Errorf("uri = %q, want %q", pkg.URI, baseURL+next)
|
||||
}
|
||||
for _, r := range pkg.Releases {
|
||||
ids = append(ids, r.ID)
|
||||
}
|
||||
next = ""
|
||||
if pkg.Links != nil && pkg.Links.Next != "" {
|
||||
if !strings.HasPrefix(pkg.Links.Next, baseURL+BasePath+"?") {
|
||||
t.Fatalf("next = %q", pkg.Links.Next)
|
||||
}
|
||||
if !strings.Contains(pkg.Links.Next, "limit=2") ||
|
||||
!strings.Contains(pkg.Links.Next, "updatedFrom=") {
|
||||
t.Errorf("next lost query parameters: %q", pkg.Links.Next)
|
||||
}
|
||||
next = strings.TrimPrefix(pkg.Links.Next, baseURL)
|
||||
}
|
||||
}
|
||||
want := "000002-2026,000003-2026,000004-2026,000005-2026"
|
||||
if got := strings.Join(ids, ","); got != want {
|
||||
t.Errorf("walked %s, want %s", got, want)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
package mock
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRateLimitWindows(t *testing.T) {
|
||||
now := time.Date(2026, 10, 6, 10, 3, 20, 0, time.UTC) // first request of the hour
|
||||
rl := NewRateLimiter(12, 5*time.Minute)
|
||||
rl.now = func() time.Time { return now }
|
||||
|
||||
for i := 0; i < 12; i++ {
|
||||
if ok, _ := rl.allow("a"); !ok {
|
||||
t.Fatalf("request %d rejected", i+1)
|
||||
}
|
||||
}
|
||||
now = now.Add(100 * time.Second) // 10:05:00, window ends 10:08:20
|
||||
if ok, secs := rl.allow("a"); ok || secs != 200 {
|
||||
t.Fatalf("want reject with 200s, got ok=%v secs=%d", ok, secs)
|
||||
}
|
||||
if ok, _ := rl.allow("b"); !ok {
|
||||
t.Fatal("other client should be unaffected")
|
||||
}
|
||||
now = time.Date(2026, 10, 6, 10, 8, 20, 0, time.UTC) // next window opens
|
||||
if ok, _ := rl.allow("a"); !ok {
|
||||
t.Fatal("new window should allow")
|
||||
}
|
||||
|
||||
// Window 10:58:20-11:03:20 is cut short by the new hour at 11:00:00.
|
||||
now = time.Date(2026, 10, 6, 10, 58, 30, 0, time.UTC)
|
||||
for i := 0; i < 12; i++ {
|
||||
rl.allow("a")
|
||||
}
|
||||
if ok, secs := rl.allow("a"); ok || secs != 90 {
|
||||
t.Fatalf("want reject with 90s, got ok=%v secs=%d", ok, secs)
|
||||
}
|
||||
now = time.Date(2026, 10, 6, 11, 0, 5, 0, time.UTC) // re-anchors here
|
||||
if ok, _ := rl.allow("a"); !ok {
|
||||
t.Fatal("new hour should allow")
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientKey(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"192.0.2.1:1234": "192.0.2.1",
|
||||
"[::1]:8080": "::1",
|
||||
"no-port": "no-port", // not host:port, used as-is
|
||||
}
|
||||
for addr, want := range cases {
|
||||
req := httptest.NewRequest("GET", "/", nil)
|
||||
req.RemoteAddr = addr
|
||||
if got := clientKey(req); got != want {
|
||||
t.Errorf("clientKey(%q) = %q, want %q", addr, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimitMiddleware(t *testing.T) {
|
||||
now := time.Date(2026, 10, 6, 10, 0, 0, 0, time.UTC)
|
||||
rl := NewRateLimiter(2, 5*time.Minute)
|
||||
rl.now = func() time.Time { return now }
|
||||
|
||||
calls := 0
|
||||
h := rl.Middleware(func(w http.ResponseWriter, req *http.Request) {
|
||||
calls++
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
do := func(addr string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest("GET", "/", nil)
|
||||
req.RemoteAddr = addr
|
||||
rec := httptest.NewRecorder()
|
||||
h(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
for i := 0; i < 2; i++ {
|
||||
if rec := do("192.0.2.1:1000"); rec.Code != http.StatusOK {
|
||||
t.Fatalf("request %d: status %d", i+1, rec.Code)
|
||||
}
|
||||
}
|
||||
// Same host, different port: still the same client.
|
||||
rec := do("192.0.2.1:2000")
|
||||
if rec.Code != http.StatusTooManyRequests {
|
||||
t.Fatalf("status %d, want 429", rec.Code)
|
||||
}
|
||||
if got := rec.Header().Get("Retry-After"); got != "300" {
|
||||
t.Errorf("Retry-After = %q, want 300", got)
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), "Rate limit of 2 exceeded") ||
|
||||
!strings.Contains(rec.Body.String(), "300 seconds") {
|
||||
t.Errorf("body = %q", rec.Body.String())
|
||||
}
|
||||
if calls != 2 {
|
||||
t.Errorf("next called %d times, want 2", calls)
|
||||
}
|
||||
if rec := do("192.0.2.2:1000"); rec.Code != http.StatusOK {
|
||||
t.Errorf("other client: status %d", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// Once more than 10000 clients are tracked, a new client in a new hour drops
|
||||
// every client left over from earlier hours.
|
||||
func TestRateLimitPrunesStaleClients(t *testing.T) {
|
||||
now := time.Date(2026, 10, 6, 10, 30, 0, 0, time.UTC)
|
||||
rl := NewRateLimiter(1, time.Minute)
|
||||
rl.now = func() time.Time { return now }
|
||||
|
||||
for i := 0; i <= 10000; i++ {
|
||||
rl.allow(strconv.Itoa(i))
|
||||
}
|
||||
if len(rl.clients) != 10001 {
|
||||
t.Fatalf("tracking %d clients", len(rl.clients))
|
||||
}
|
||||
|
||||
// Same hour: nothing is stale, so nothing is dropped.
|
||||
rl.allow("same-hour")
|
||||
if len(rl.clients) != 10002 {
|
||||
t.Fatalf("same hour: tracking %d clients, want 10002", len(rl.clients))
|
||||
}
|
||||
|
||||
now = now.Add(time.Hour)
|
||||
rl.allow("new")
|
||||
if len(rl.clients) != 1 {
|
||||
t.Fatalf("after prune: tracking %d clients, want 1", len(rl.clients))
|
||||
}
|
||||
if _, ok := rl.clients["new"]; !ok {
|
||||
t.Fatal("new client was pruned")
|
||||
}
|
||||
}
|
||||
@@ -34,6 +34,10 @@ type Store struct {
|
||||
openPos map[string]int
|
||||
}
|
||||
|
||||
// OpenStore loads the releases stored at path, creating the file and its
|
||||
// directory if needed, and opens it for appending. Lines that can't be read
|
||||
// are skipped, releases are put back in date order if needed, and a record
|
||||
// left incomplete by a crash is ended so new ones start on a fresh line.
|
||||
func OpenStore(path string) (*Store, error) {
|
||||
s := &Store{latest: map[string]int{}, lastNum: map[int]int{}, openPos: map[string]int{}}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
|
||||
@@ -115,6 +119,9 @@ func openForAppend(path string) (*os.File, error) {
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// index records release r, at position i, in the lookups: the ocid's latest
|
||||
// release, whether its procedure is still open (an award closes it), and the
|
||||
// highest notice number used each year.
|
||||
func (s *Store) index(i int, r ocds.Release) {
|
||||
s.latest[r.OCID] = i
|
||||
if len(r.Tag) > 0 && r.Tag[0] == "award" {
|
||||
@@ -132,6 +139,7 @@ func (s *Store) index(i int, r ocds.Release) {
|
||||
}
|
||||
}
|
||||
|
||||
// closeProcedure removes ocid from the open procedures, if it is there.
|
||||
func (s *Store) closeProcedure(ocid string) {
|
||||
pos, ok := s.openPos[ocid]
|
||||
if !ok {
|
||||
@@ -151,6 +159,7 @@ func (s *Store) Append(r ocds.Release) error { return s.append(r, true) }
|
||||
// Used for bulk catch-up, where a sync per record would be very slow.
|
||||
func (s *Store) AppendNoSync(r ocds.Release) error { return s.append(r, false) }
|
||||
|
||||
// Sync flushes appended releases to disk. Call it after AppendNoSync.
|
||||
func (s *Store) Sync() error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
@@ -204,6 +213,8 @@ func (s *Store) Page(from, upper time.Time, start, limit int) ([]ocds.Release, i
|
||||
return out, i, more
|
||||
}
|
||||
|
||||
// Last returns the date of the newest release, or false if the store is
|
||||
// empty.
|
||||
func (s *Store) Last() (time.Time, bool) {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
|
||||
@@ -0,0 +1,269 @@
|
||||
package mock
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"math/rand"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.avsdev.uk/Go/job-search/internal/mock/findATender/ocds"
|
||||
)
|
||||
|
||||
func TestStorePersistsAndGeneratorProgresses(t *testing.T) {
|
||||
cat := loadTestCatalog(t)
|
||||
path := filepath.Join(t.TempDir(), "releases.jsonl")
|
||||
store, err := OpenStore(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
gen := NewGenerator(cat, store, 42)
|
||||
now := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
|
||||
n, err := gen.CatchUp(now, 48*time.Hour, 3*time.Minute, 15*time.Minute)
|
||||
if err != nil || n < 48*60/15 {
|
||||
t.Fatalf("catch-up generated %d (err %v)", n, err)
|
||||
}
|
||||
|
||||
tags := map[string]int{}
|
||||
ocids := map[string]int{}
|
||||
all, _, _ := store.Page(time.Time{}, now, 0, 100000)
|
||||
for _, r := range all {
|
||||
tags[r.Tag[0]]++
|
||||
ocids[r.OCID]++
|
||||
if r.Tender == nil || r.Tender.Classification == nil {
|
||||
t.Fatalf("release %s has no tender.classification", r.ID)
|
||||
}
|
||||
}
|
||||
if tags["award"] == 0 || tags["tender"] == 0 {
|
||||
t.Fatalf("expected tenders and awards, got %v", tags)
|
||||
}
|
||||
if len(ocids) == len(all) {
|
||||
t.Fatal("expected some procedures with more than one release")
|
||||
}
|
||||
store.Close()
|
||||
|
||||
// Reopen: everything is still there, and catch-up only fills the new gap.
|
||||
store2, err := OpenStore(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store2.Close()
|
||||
if store2.Len() != len(all) {
|
||||
t.Fatalf("after reopen: %d releases, want %d", store2.Len(), len(all))
|
||||
}
|
||||
gen2 := NewGenerator(cat, store2, 7)
|
||||
m, _ := gen2.CatchUp(now.Add(time.Hour), 48*time.Hour, 3*time.Minute, 15*time.Minute)
|
||||
if m < 4 || m > 20 {
|
||||
t.Fatalf("one hour gap produced %d releases", m)
|
||||
}
|
||||
if last, _ := store2.Last(); last.After(now.Add(time.Hour)) {
|
||||
t.Fatal("release dated in the future")
|
||||
}
|
||||
|
||||
// The HTTP handler pages through the store without gaps or repeats.
|
||||
srv := &server{store: store2, baseURL: "", loc: time.UTC}
|
||||
seen := map[string]bool{}
|
||||
next := BasePath + "?updatedFrom=2026-10-07T00:00:00&limit=7"
|
||||
for pages := 0; next != ""; pages++ {
|
||||
rec := httptest.NewRecorder()
|
||||
srv.handle(rec, httptest.NewRequest("GET", next, nil))
|
||||
// Strict decode: the output must contain only fields the ocds types define.
|
||||
var pkg ocds.ReleasePackage
|
||||
dec := json.NewDecoder(bytes.NewReader(rec.Body.Bytes()))
|
||||
dec.DisallowUnknownFields()
|
||||
if err := dec.Decode(&pkg); err != nil {
|
||||
t.Fatalf("response doesn't match ocds types: %v", err)
|
||||
}
|
||||
for _, r := range pkg.Releases {
|
||||
if seen[r.ID] {
|
||||
t.Fatalf("release %s returned twice", r.ID)
|
||||
}
|
||||
seen[r.ID] = true
|
||||
}
|
||||
next = ""
|
||||
if pkg.Links != nil {
|
||||
next = pkg.Links.Next
|
||||
}
|
||||
}
|
||||
from := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
|
||||
want, _, _ := store2.Page(from, time.Now(), 0, 100000)
|
||||
if len(seen) != len(want) {
|
||||
t.Fatalf("paged %d releases, want %d", len(seen), len(want))
|
||||
}
|
||||
}
|
||||
|
||||
func writeLines(t *testing.T, path string, lines ...string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func releaseLine(t *testing.T, ocid, id, tag string, at time.Time) string {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(ocds.Release{OCID: ocid, ID: id, Date: at, Tag: []string{tag}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// Unreadable lines are skipped, blank lines ignored, a hand-edited file is put
|
||||
// back in date order, and the index (open procedures, notice numbers) is rebuilt.
|
||||
func TestOpenStoreRecoversFromMessyFile(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "r.jsonl")
|
||||
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
|
||||
writeLines(t, path,
|
||||
releaseLine(t, "ocds-b", "000007-2026", "tender", t0.Add(2*time.Hour)),
|
||||
"{not json",
|
||||
"",
|
||||
releaseLine(t, "ocds-a", "000003-2026", "tender", t0),
|
||||
releaseLine(t, "ocds-a", "000005-2026", "award", t0.Add(time.Hour)),
|
||||
"",
|
||||
)
|
||||
|
||||
store, err := OpenStore(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
all, _, _ := store.Page(time.Time{}, t0.Add(24*time.Hour), 0, 100)
|
||||
var ids []string
|
||||
for _, r := range all {
|
||||
ids = append(ids, r.ID)
|
||||
}
|
||||
if got, want := strings.Join(ids, ","), "000003-2026,000005-2026,000007-2026"; got != want {
|
||||
t.Fatalf("releases %s, want %s", got, want)
|
||||
}
|
||||
if got := store.NextNoticeID(2026); got != "000008-2026" {
|
||||
t.Errorf("NextNoticeID = %s, want 000008-2026", got)
|
||||
}
|
||||
if got := store.NextNoticeID(2027); got != "000001-2027" {
|
||||
t.Errorf("NextNoticeID(2027) = %s", got)
|
||||
}
|
||||
// ocds-a was awarded, so only ocds-b is still open.
|
||||
//nolint:gosec // G404: seeded math/rand keeps the test deterministic
|
||||
r, ok := store.RandomOpen(rand.New(rand.NewSource(1)), t0)
|
||||
if !ok || r.OCID != "ocds-b" {
|
||||
t.Errorf("RandomOpen = %s, %v; want ocds-b", r.OCID, ok)
|
||||
}
|
||||
}
|
||||
|
||||
// A record cut off mid-write is skipped, and the next record starts on a fresh
|
||||
// line so it is still readable after a restart.
|
||||
func TestOpenStoreAfterTruncatedWrite(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "r.jsonl")
|
||||
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
|
||||
full := releaseLine(t, "ocds-a", "000001-2026", "tender", t0)
|
||||
cut := releaseLine(t, "ocds-b", "000002-2026", "tender", t0.Add(time.Minute))
|
||||
writeLines(t, path, full, cut[:len(cut)/2]) // no trailing newline
|
||||
|
||||
store, err := OpenStore(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if store.Len() != 1 {
|
||||
t.Fatalf("Len = %d, want 1", store.Len())
|
||||
}
|
||||
if err := store.Append(
|
||||
ocds.Release{
|
||||
OCID: "ocds-c",
|
||||
ID: "000002-2026",
|
||||
Date: t0.Add(2 * time.Minute),
|
||||
Tag: []string{"tender"},
|
||||
},
|
||||
); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store.Close()
|
||||
|
||||
store2, err := OpenStore(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store2.Close()
|
||||
if store2.Len() != 2 || !store2.HasOCID("ocds-c") {
|
||||
t.Fatalf("after reopen: Len = %d, has ocds-c = %v", store2.Len(), store2.HasOCID("ocds-c"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenStoreErrors(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
// The parent "directory" is a regular file.
|
||||
file := filepath.Join(dir, "file")
|
||||
if err := os.WriteFile(file, nil, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := OpenStore(filepath.Join(file, "r.jsonl")); err == nil {
|
||||
t.Error("parent is a file: expected error")
|
||||
}
|
||||
|
||||
// The store path is a directory, which opens but can't be read.
|
||||
if _, err := OpenStore(dir); err == nil {
|
||||
t.Error("path is a directory: expected error")
|
||||
}
|
||||
|
||||
// A missing parent directory is created.
|
||||
store, err := OpenStore(filepath.Join(dir, "a", "b", "r.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatalf("nested path: %v", err)
|
||||
}
|
||||
store.Close()
|
||||
}
|
||||
|
||||
// A release dated before the newest one is moved up to keep the order valid.
|
||||
func TestAppendKeepsDateOrder(t *testing.T) {
|
||||
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
|
||||
store.Append(ocds.Release{OCID: "a", ID: "000001-2026", Date: t0, Tag: []string{"tender"}})
|
||||
store.Append(
|
||||
ocds.Release{OCID: "b", ID: "000002-2026", Date: t0.Add(-time.Hour), Tag: []string{"tender"}},
|
||||
)
|
||||
if last, _ := store.Last(); !last.Equal(t0) {
|
||||
t.Errorf("Last = %s, want %s", last, t0)
|
||||
}
|
||||
|
||||
// Appending to a closed store fails and leaves the store unchanged.
|
||||
store.Close()
|
||||
if err := store.Append(ocds.Release{OCID: "c", Date: t0, Tag: []string{"tender"}}); err == nil {
|
||||
t.Error("append after close: expected error")
|
||||
}
|
||||
if store.Len() != 2 || store.HasOCID("c") {
|
||||
t.Errorf("failed append changed the store: Len = %d", store.Len())
|
||||
}
|
||||
}
|
||||
|
||||
// Procedures idle since before the cut-off are dropped from the open set.
|
||||
func TestRandomOpenDropsStaleProcedures(t *testing.T) {
|
||||
store, err := OpenStore(filepath.Join(t.TempDir(), "r.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
t0 := time.Date(2026, 10, 8, 9, 0, 0, 0, time.UTC)
|
||||
store.Append(ocds.Release{OCID: "old", ID: "000001-2026", Date: t0, Tag: []string{"tender"}})
|
||||
store.Append(
|
||||
ocds.Release{OCID: "new", ID: "000002-2026", Date: t0.Add(time.Hour), Tag: []string{"tender"}},
|
||||
)
|
||||
|
||||
//nolint:gosec // G404: seeded math/rand keeps the test deterministic
|
||||
rnd := rand.New(rand.NewSource(1))
|
||||
for i := 0; i < 10; i++ {
|
||||
if r, ok := store.RandomOpen(rnd, t0.Add(time.Minute)); !ok || r.OCID != "new" {
|
||||
t.Fatalf("RandomOpen = %s, %v; want new", r.OCID, ok)
|
||||
}
|
||||
}
|
||||
if _, ok := store.RandomOpen(rnd, t0.Add(2*time.Hour)); ok {
|
||||
t.Error("everything is stale: expected none")
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user