Added a backfill in 7-day windows, and fetches now keep their data on a failure
Backfilling a long window (an early --since, or a long time offline) hit the API's rate limit, which could still be exceeded after waiting once, and then everything fetched so far was lost. Before each Find a Tender source's worker starts, a start point more than 7 days ago is now backfilled in 7-day windows bounded on both sides, oldest first, each saved before the next is fetched. A window that fails is saved as far as it got and fetched again from its start after the source's interval. The worker then carries on from the end of the last window. A 429 is now waited out up to sources.<source>.rate_limit_retries times (default 5) rather than once; fts-once --ignore-rate-limit uses 0. FetchAll returns what it fetched before an error along with it, and the workers save it, so a failed run no longer loses its data. findatender's FetchAll and FetchOnce take RequestOptions, which gains Until (updatedTo), and RateLimitRetries replaces AllowRateLimit.
This commit is contained in:
1 parent
ff2aa316b5
commit
c09c30ae71
16 files changed
+683
-149
No files matched your search
@@ -76,34 +76,25 @@ func (ft *FindATender) Shutdown() {
|
||||
}
|
||||
}
|
||||
|
||||
// FetchOnce fetches the first page of releases updated since since, and
|
||||
// converts them to opportunities. If allowRateLimit is true, a 429 response
|
||||
// is waited out and retried; otherwise it fails with ErrRateLimitExceeded.
|
||||
// FetchOnce fetches the first page of the releases opts selects, and
|
||||
// converts them to opportunities.
|
||||
func (ft *FindATender) FetchOnce(
|
||||
since time.Time, allowRateLimit bool,
|
||||
opts RequestOptions,
|
||||
) ([]*types.Opportunity, error) {
|
||||
releases, err := ft.fetchOnce(RequestOptions{
|
||||
Since: since,
|
||||
AllowRateLimit: allowRateLimit,
|
||||
})
|
||||
releases, err := ft.fetchOnce(opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return Convert(releases), nil
|
||||
}
|
||||
|
||||
// FetchAll fetches every page of releases updated since since, following
|
||||
// each page's next link, and converts them to opportunities. allowRateLimit
|
||||
// behaves as for FetchOnce.
|
||||
// FetchAll fetches every page of the releases opts selects, following each
|
||||
// page's next link, and converts them to opportunities. If a page fails, the
|
||||
// opportunities from the pages before it are returned with the error, so the
|
||||
// caller can keep them.
|
||||
func (ft *FindATender) FetchAll(
|
||||
since time.Time, allowRateLimit bool,
|
||||
opts RequestOptions,
|
||||
) ([]*types.Opportunity, error) {
|
||||
releases, err := ft.fetchAll(RequestOptions{
|
||||
Since: since,
|
||||
AllowRateLimit: allowRateLimit,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return Convert(releases), nil
|
||||
releases, err := ft.fetchAll(opts)
|
||||
return Convert(releases), err
|
||||
}
|
||||
Reference in new issue
Block a user