Made each source start from its newest stored release, with --since as an override
Without --since, fetcher service and job-search serve start each source from the newest release stored for it (the new store.LatestReleaseDate, by Source), so a restart carries on where it left off instead of fetching the last day again; with nothing stored, the client's default still applies. --since overrides it for every source. The Find a Tender source's name is now a constant, as the Source its opportunities are stored with. The README's guide to adding a source is brought up to date with the current worker, start and config type names.
This commit is contained in:
1 parent
3c4fde8dfd
commit
284a589627
10 files changed
+256
-22
No files matched your search
@@ -84,6 +84,10 @@ refuses to open a database that isn't at this build's schema version, so run
|
||||
|
||||
`service` logs to `logging.file` if it's set, or to the terminal with
|
||||
`--foreground`. Both commands take `--since "YYYY-MM-DD[ hh:mm:ss]"`.
|
||||
Without it, `service` (and `job-search serve`) starts each source from the
|
||||
newest release stored for it, so a restart carries on where it left off; with
|
||||
nothing stored, Find a Tender's client fetches the last day. `--since`
|
||||
overrides that for every source.
|
||||
`fts-once` can be limited with `--planning`, `--opportunities` or
|
||||
`--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
|
||||
of waiting.
|
||||
@@ -169,7 +173,7 @@ Using a source called `example` as an illustration:
|
||||
1. **Configuration section:** create `internal/config/conf_source_example.go`,
|
||||
copying an existing source such as `conf_source_ftsmock.go`:
|
||||
- Add a `JSCKSourceExample = "example"` constant.
|
||||
- Define `JobSearchConfigSourceExample`, embedding `JobSearchConfigSource`
|
||||
- Define `JSCSourceExample`, embedding `JSCSourcesSource`
|
||||
(for `enabled` and `interval`), plus any fields of its own with `cfg` and
|
||||
optional `default` tags, e.g. ``APIRoot string `cfg:"api_root"` ``.
|
||||
- Implement the `Section` interface (see `internal/config/section.go`):
|
||||
@@ -178,8 +182,8 @@ Using a source called `example` as an illustration:
|
||||
`// ---------- Section methods (see Section) ----------` banner, renaming
|
||||
the receiver type and using the new constant in `Flatten`.
|
||||
2. **Register the section:** add
|
||||
``Example JobSearchConfigSourceExample `cfg:"example"` `` to
|
||||
`JobSearchConfigSources` in `internal/config/conf_sources.go`. The `cfg` tag
|
||||
``Example JSCSourceExample `cfg:"example"` `` to
|
||||
`JSCSources` in `internal/config/conf_sources.go`. The `cfg` tag
|
||||
must match the constant. The source's keys (`sources.example.*`) are then
|
||||
loaded, validated and available to the `config` commands with no further
|
||||
code.
|
||||
@@ -189,18 +193,27 @@ Using a source called `example` as an illustration:
|
||||
for `fts_mock`: create it with the source's own API root.
|
||||
4. **Worker:** create `internal/fetcher/worker_example.go`, following
|
||||
`worker_ftsmock.go`:
|
||||
- `runExample(ctx, conf, since)` builds a `util.Worker` with the source's
|
||||
name, `conf.Sources.Example.Interval` and a task, then calls `Run`.
|
||||
- The task fetches everything since its `since` argument. Wrap blocking
|
||||
calls in `util.AwaitTask` so shutdown isn't held up, and return
|
||||
`ErrNoNewReleases` when nothing is found, so the checkpoint still moves
|
||||
forward. Any other error makes the next run retry from the same point.
|
||||
- Add an `exampleSource = "Example"` constant: the worker's name, and the
|
||||
`Source` its opportunities are stored with (set it on each one if the
|
||||
client doesn't, as `fetchFTSMock` does).
|
||||
- `runExample(ctx, logger, conf, save, since)` builds a `util.Worker`
|
||||
with `exampleSource`, `conf.Interval` and a task, then calls `Run`.
|
||||
- The task fetches everything since its `since` argument, then returns
|
||||
`save(ctx, logger, opportunities)`, which stores them when the fetcher
|
||||
has a store. Wrap blocking calls in `util.AwaitTask` so shutdown isn't
|
||||
held up, and return `ErrNoNewReleases` when nothing is found, so the
|
||||
checkpoint still moves forward. Any other error makes the next run
|
||||
retry from the same point.
|
||||
5. **Start it:** in `Run` in `internal/fetcher/fetcher.go`, add
|
||||
```go
|
||||
if conf.Sources.Example.Enabled {
|
||||
start("Example", runExample)
|
||||
if f.conf.Example.Enabled {
|
||||
start(exampleSource, func(ctx context.Context, since time.Time) error {
|
||||
return runExample(ctx, logger, &f.conf.Example, f.save, since)
|
||||
})
|
||||
}
|
||||
```
|
||||
`start` passes the worker its starting point: `--since`, or the newest
|
||||
release stored with `exampleSource`.
|
||||
6. **Tests:** `TestSections` checks the new section implements `Section`
|
||||
and that its keys match its `cfg` tag, with no changes needed. In
|
||||
`internal/config/conf_sections_test.go`, add the section to
|
||||
|
||||
Reference in new issue
Block a user