Made each source start from its newest stored release, with --since as an override
Without --since, fetcher service and job-search serve start each source from the newest release stored for it (the new store.LatestReleaseDate, by Source), so a restart carries on where it left off instead of fetching the last day again; with nothing stored, the client's default still applies. --since overrides it for every source. The Find a Tender source's name is now a constant, as the Source its opportunities are stored with. The README's guide to adding a source is brought up to date with the current worker, start and config type names.
This commit is contained in:
1 parent
3c4fde8dfd
commit
284a589627
10 files changed
+256
-22
No files matched your search
@@ -12,6 +12,10 @@ import (
|
||||
"git.avsdev.uk/Go/job-search/internal/util"
|
||||
)
|
||||
|
||||
// findATenderSource is the Source of the opportunities Find a Tender's
|
||||
// converter makes, and so of the ones the Find a Tender source fetches.
|
||||
const findATenderSource = "FindATender"
|
||||
|
||||
// runFindATender polls Find a Tender every configured interval until ctx
|
||||
// is cancelled, passing what each run fetches to save.
|
||||
func runFindATender(
|
||||
@@ -27,7 +31,7 @@ func runFindATender(
|
||||
|
||||
worker := util.Worker{
|
||||
Logger: tagged(logger),
|
||||
Name: "FindATender",
|
||||
Name: findATenderSource,
|
||||
Interval: conf.Interval,
|
||||
Task: func(ctx context.Context, since time.Time) error {
|
||||
opportunities, err := fetchFindATender(ctx, logger, conf, since)
|
||||
|
||||
Reference in new issue
Block a user