Made each source start from its newest stored release, with --since as an override
Without --since, fetcher service and job-search serve start each source from the newest release stored for it (the new store.LatestReleaseDate, by Source), so a restart carries on where it left off instead of fetching the last day again; with nothing stored, the client's default still applies. --since overrides it for every source. The Find a Tender source's name is now a constant, as the Source its opportunities are stored with. The README's guide to adding a source is brought up to date with the current worker, start and config type names.
This commit is contained in:
1 parent
3c4fde8dfd
commit
284a589627
10 files changed
+256
-22
No files matched your search
@@ -23,6 +23,16 @@ package store
|
||||
const selectOpportunityLatestReleaseDate = `
|
||||
SELECT latest_release_date FROM opportunities WHERE uuid = ?`
|
||||
|
||||
// selectSourceLatestReleaseDate reads the newest latest_release_date of one
|
||||
// source's opportunities. ORDER BY rather than MAX keeps the column's DATETIME
|
||||
// type, so the driver reads it as a time; it walks the latest_release_date
|
||||
// index.
|
||||
const selectSourceLatestReleaseDate = `
|
||||
SELECT latest_release_date FROM opportunities
|
||||
WHERE source = ?
|
||||
ORDER BY latest_release_date DESC
|
||||
LIMIT 1`
|
||||
|
||||
// upsertOpportunity inserts or updates a opportunity. Named parameters
|
||||
// (types.Opportunity).
|
||||
//
|
||||
|
||||
Reference in new issue
Block a user