Made each source start from its newest stored release, with --since as an override

Without --since, fetcher service and job-search serve start each source
from the newest release stored for it (the new store.LatestReleaseDate,
by Source), so a restart carries on where it left off instead of fetching
the last day again; with nothing stored, the client's default still
applies. --since overrides it for every source. The Find a Tender source's
name is now a constant, as the Source its opportunities are stored with.

The README's guide to adding a source is brought up to date with the
current worker, start and config type names.
This commit is contained in:
bwilliams committed 2026-10-16 11:37:09 +01:00
1 parent 3c4fde8dfd
commit 284a589627
10 files changed
+256 -22

No files matched your search

+24 -11
View File
@@ -84,6 +84,10 @@ refuses to open a database that isn't at this build's schema version, so run
`service` logs to `logging.file` if it's set, or to the terminal with
`--foreground`. Both commands take `--since "YYYY-MM-DD[ hh:mm:ss]"`.
Without it, `service` (and `job-search serve`) starts each source from the
newest release stored for it, so a restart carries on where it left off; with
nothing stored, Find a Tender's client fetches the last day. `--since`
overrides that for every source.
`fts-once` can be limited with `--planning`, `--opportunities` or
`--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
of waiting.
@@ -169,7 +173,7 @@ Using a source called `example` as an illustration:
1. **Configuration section:** create `internal/config/conf_source_example.go`,
copying an existing source such as `conf_source_ftsmock.go`:
- Add a `JSCKSourceExample = "example"` constant.
- Define `JobSearchConfigSourceExample`, embedding `JobSearchConfigSource`
- Define `JSCSourceExample`, embedding `JSCSourcesSource`
(for `enabled` and `interval`), plus any fields of its own with `cfg` and
optional `default` tags, e.g. ``APIRoot string `cfg:"api_root"` ``.
- Implement the `Section` interface (see `internal/config/section.go`):
@@ -178,8 +182,8 @@ Using a source called `example` as an illustration:
`// ---------- Section methods (see Section) ----------` banner, renaming
the receiver type and using the new constant in `Flatten`.
2. **Register the section:** add
``Example JobSearchConfigSourceExample `cfg:"example"` `` to
`JobSearchConfigSources` in `internal/config/conf_sources.go`. The `cfg` tag
``Example JSCSourceExample `cfg:"example"` `` to
`JSCSources` in `internal/config/conf_sources.go`. The `cfg` tag
must match the constant. The source's keys (`sources.example.*`) are then
loaded, validated and available to the `config` commands with no further
code.
@@ -189,18 +193,27 @@ Using a source called `example` as an illustration:
for `fts_mock`: create it with the source's own API root.
4. **Worker:** create `internal/fetcher/worker_example.go`, following
`worker_ftsmock.go`:
- `runExample(ctx, conf, since)` builds a `util.Worker` with the source's
name, `conf.Sources.Example.Interval` and a task, then calls `Run`.
- The task fetches everything since its `since` argument. Wrap blocking
calls in `util.AwaitTask` so shutdown isn't held up, and return
`ErrNoNewReleases` when nothing is found, so the checkpoint still moves
forward. Any other error makes the next run retry from the same point.
- Add an `exampleSource = "Example"` constant: the worker's name, and the
`Source` its opportunities are stored with (set it on each one if the
client doesn't, as `fetchFTSMock` does).
- `runExample(ctx, logger, conf, save, since)` builds a `util.Worker`
with `exampleSource`, `conf.Interval` and a task, then calls `Run`.
- The task fetches everything since its `since` argument, then returns
`save(ctx, logger, opportunities)`, which stores them when the fetcher
has a store. Wrap blocking calls in `util.AwaitTask` so shutdown isn't
held up, and return `ErrNoNewReleases` when nothing is found, so the
checkpoint still moves forward. Any other error makes the next run
retry from the same point.
5. **Start it:** in `Run` in `internal/fetcher/fetcher.go`, add
```go
if conf.Sources.Example.Enabled {
start("Example", runExample)
if f.conf.Example.Enabled {
start(exampleSource, func(ctx context.Context, since time.Time) error {
return runExample(ctx, logger, &f.conf.Example, f.save, since)
})
}
```
`start` passes the worker its starting point: `--since`, or the newest
release stored with `exampleSource`.
6. **Tests:** `TestSections` checks the new section implements `Section`
and that its keys match its `cfg` tag, with no changes needed. In
`internal/config/conf_sections_test.go`, add the section to