Added a backfill in 7-day windows, and fetches now keep their data on a failure

Backfilling a long window (an early --since, or a long time offline)
hit the API's rate limit, which could still be exceeded after waiting
once, and then everything fetched so far was lost.

Before each Find a Tender source's worker starts, a start point more
than 7 days ago is now backfilled in 7-day windows bounded on both
sides, oldest first, each saved before the next is fetched. A window
that fails is saved as far as it got and fetched again from its start
after the source's interval. The worker then carries on from the end
of the last window.

A 429 is now waited out up to sources.<source>.rate_limit_retries
times (default 5) rather than once; fts-once --ignore-rate-limit uses
0. FetchAll returns what it fetched before an error along with it, and
the workers save it, so a failed run no longer loses its data.
findatender's FetchAll and FetchOnce take RequestOptions, which gains
Until (updatedTo), and RateLimitRetries replaces AllowRateLimit.
This commit is contained in:
bwilliams committed 2026-10-21 12:40:39 +01:00
1 parent ff2aa316b5
commit c09c30ae71
16 files changed
+683 -149

No files matched your search

+14
View File
@@ -118,6 +118,20 @@ Without it, `service` (and `job-search serve`) starts each source from the
newest release stored for it, so a restart carries on where it left off; with
nothing stored, Find a Tender's client fetches the last day. `--since`
overrides that for every source.
When the start is more than 7 days ago (an early `--since`, or a long time
offline), the Find a Tender sources first backfill it in 7-day windows,
oldest first, saving each before fetching the next. A window that fails is
saved as far as it got and fetched again after the source's interval. The
API returns the newest releases first, so if a backfill is stopped partway
through a window, restarting without `--since` skips the rest of that
window: restart with `--since` set to the window's start (logged as
`backfilling`) to fill it.
A rate limit is waited out, as the API asks, up to
`sources.<source>.rate_limit_retries` times per request. A failed fetch
keeps what it fetched before the failure.
`fts-once` can be limited with `--planning`, `--opportunities` or
`--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
of waiting.