Backfilling a long window (an early --since, or a long time offline)
hit the API's rate limit, which could still be exceeded after waiting
once, and then everything fetched so far was lost.
Before each Find a Tender source's worker starts, a start point more
than 7 days ago is now backfilled in 7-day windows bounded on both
sides, oldest first, each saved before the next is fetched. A window
that fails is saved as far as it got and fetched again from its start
after the source's interval. The worker then carries on from the end
of the last window.
A 429 is now waited out up to sources.<source>.rate_limit_retries
times (default 5) rather than once; fts-once --ignore-rate-limit uses
0. FetchAll returns what it fetched before an error along with it, and
the workers save it, so a failed run no longer loses its data.
findatender's FetchAll and FetchOnce take RequestOptions, which gains
Until (updatedTo), and RateLimitRetries replaces AllowRateLimit.
sources.skip_rules_file names a JSON file of filters, each with a
comment, an active flag, a logic mode ("or" by default, or "and") and
rules by buyer, title words or CPV codes. The fetcher applies them
before saving, so only new opportunities are marked; the marks of
stored ones are never touched. CPV rules match when the opportunity's
codes are all in the list, or any of them with exclusive false, and
by prefix with exact false.
The file is read again when its time or size changes: an invalid edit
is logged and the last good rules kept, and a deleted file means no
rules until it's back. fetcher service and serve refuse to start if it
is define but missing or invalid.
job-search skip-rules check validates a rules file, and skip-rules
apply marks the stored opportunities that match Skipped, or only lists
them with --dry-run.
go generate ran ocdsgen with go run, from go.mod, but go-ocds had been
dropped from go.mod (nothing imports it, so go mod tidy removes it), which
broke go generate and so scripts/build.sh. It now runs the installed
ocdsgen from PATH, as it does swag.
scripts/install-dev-tools.sh installs the Go tools (ocdsgen v1.0.0 and swag
v2.0.0-rc6 pinned, as other versions could generate different code;
golangci-lint, goose and cobra-cli at their latest), checks for Node.js 24
or later and runs npm ci in web/. The README's dev tools section now points
to it and lists what each tool is for.
`npm run sync` in web/ copies the build output (web/dist, from npm run build)
to internal/frontend/dist, which the binary embeds. Go can't embed from
outside the package's directory (web/ is also outside the Go module), hence
the copy. Only dist/.gitkeep is committed, so a fresh clone still compiles,
and a binary built without syncing first serves no frontend.
job-search serve serves server.frontend.dir if it is set, or else the
embedded build, or warns that there is no frontend. scripts/build.sh builds
and syncs the frontend before building, running npm ci first for a release
or when web/node_modules is missing.
GET /api/opportunities lists the stored opportunities, newest release first,
a page at a time (offset, and limit up to 100, default 50); has_more says
whether another page follows. GET /api/opportunities/{uuid} returns one with
its releases. PUT and DELETE /api/opportunities/{uuid}/marks/{mark} set and
clear one mark (hidden, skipped, interested, bid_submitted, won, lost) and
return them all; won and lost exclude each other (409). The API uses the
types in internal/types, and changing a mark is made atomic, as the store
replaces all of an opportunity's marks at once. The marks are found by
reflection: every bool field of types.OpportunityMarks, by its JSON name, so
a new mark needs no API change.
The API has one entry point, api.New, with its dependencies as options
(api.WithStore), and the server only mounts it under /api, keeping the
middleware. New resources are added inside internal/api. job-search serve
passes server.WithStore when database.file is set, opening the store before
the server; without one there are no opportunity routes.
The API is described by swag v2 annotations on the handlers. go generate
writes the OpenAPI 3.1 description (git-ignored, like the OCDS types), which
the server embeds and serves at /api/openapi.json, with Swagger UI at
/api/docs. A test checks the description and the routes match. swag is in
the README's dev tools, pinned to v2.0.0-rc6, and the build now needs it.
Without --since, fetcher service and job-search serve start each source
from the newest release stored for it (the new store.LatestReleaseDate,
by Source), so a restart carries on where it left off instead of fetching
the last day again; with nothing stored, the client's default still
applies. --since overrides it for every source. The Find a Tender source's
name is now a constant, as the Source its opportunities are stored with.
The README's guide to adding a source is brought up to date with the
current worker, start and config type names.
Without server.csrf.tokens.key, each server process signs CSRF tokens with
a random key of its own, so tokens are lost on a restart and rejected by
other instances. "job-search config generate-csrf-key" stores a new random
32-byte hex key in the configuration file, without printing it. --print
only prints a key, for JOBSEARCH_SERVER_CSRF_TOKENS_KEY or another
instance. An existing key is only replaced with --force, as that
invalidates every issued token, and no key is stored while
JOBSEARCH_SERVER_CSRF_TOKENS_KEY is set, as it would override the file.
One of the two is required. --http-upgrade upgrades an HTTP connection to
a WebSocket on /ws, as before, for job-search serve; --url and --origin go
with it. --direct connects straight to ws-test serve over plain TCP, one
JSON message per line, at --addr (default from server.host and
server.port). --ping, --message, --listen and stdin work the same in both
modes. The README's ws-test section explains the two modes.
job-search no longer has a fetch command. Its subcommands are now in
cmd/fetcher, which reads the same configuration: "fetcher service" runs
every enabled source's fetcher on its interval until stopped, and
"fetcher fts-once" (was "fetch once") fetches from Find a Tender once and
prints the releases as JSON. The new namea say what the commands do,
because service runs every source while fts-once fetches only from Find a
Tender.
The code stays in internal/cli/fetch, where NewCommands replaces the fetch
parent command. once.go is now fts_once.go and commandOnce is
commandFTSOnce. The README's usage, the config README and the find_a_tender
api_root comment now give the new command names.
bwilliams