Made each source start from its newest stored release, with --since as an override
Without --since, fetcher service and job-search serve start each source from the newest release stored for it (the new store.LatestReleaseDate, by Source), so a restart carries on where it left off instead of fetching the last day again; with nothing stored, the client's default still applies. --since overrides it for every source. The Find a Tender source's name is now a constant, as the Source its opportunities are stored with. The README's guide to adding a source is brought up to date with the current worker, start and config type names.
This commit is contained in:
1 parent
3c4fde8dfd
commit
284a589627
10 files changed
+256
-22
No files matched your search
@@ -716,6 +716,26 @@ func (s *Store) ListReleases(
|
||||
return releases, nil
|
||||
}
|
||||
|
||||
// LatestReleaseDate returns the date of the newest release stored for the
|
||||
// opportunities from source, or the zero time if none are stored, so a
|
||||
// fetcher can carry on from where it left off.
|
||||
func (s *Store) LatestReleaseDate(
|
||||
ctx context.Context, source string,
|
||||
) (time.Time, error) {
|
||||
var latest time.Time
|
||||
err := s.db.GetContext(ctx, &latest, selectSourceLatestReleaseDate, source)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return time.Time{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf(
|
||||
"reading the latest release date of %s: %w", source, err,
|
||||
)
|
||||
}
|
||||
|
||||
return latest, nil
|
||||
}
|
||||
|
||||
// ReleaseExists reports whether a release is already stored, so a fetcher
|
||||
// can skip converting it.
|
||||
func (s *Store) ReleaseExists(
|
||||
|
||||
Reference in new issue
Block a user