Added an EU Supply source, scraping the UK portal's public tenders
internal/eusupply replaces the old Node scraper (eu-supply.js) for uk.eu-supply.com. The site has no API, so it posts the public search form, which now needs the form's current defaults and reads the HTML with golang.org/x/net/html. Each tender not stored yet has its own page read too, for its description, contract type, CPV codes and buyer address. Tenders are stored with the source "EU Supply - UK" as eusupply-uk-<tender id>, with one release holding the tender as read. The site publishes no rate limit, so the client sets its own: one request at a time, with a random wait between sources.eu_supply.request_min_gap and request_max_gap (2s to 10s). The source has the same 7-day backfill as Find a Tender's. Most of its tenders are also on Find a Tender, so a new one is stored Hidden and Skipped when another source's opportunity released within 30 days has a title at least 90% alike, and its buyer or description matches too (80% of the shorter one's significant words).
This commit is contained in:
1 parent
5c62376b6e
commit
3a06132071
25 files changed
+2520
-8
No files matched your search
@@ -149,6 +149,13 @@ LIMIT ? OFFSET ?`
|
||||
)`
|
||||
)
|
||||
|
||||
// selectOtherSourcesTitles lists the uuid, title and latest_release_date of
|
||||
// the opportunities not from one source whose latest_release_date is in
|
||||
// [from, before); it walks the latest_release_date index.
|
||||
const selectOtherSourcesTitles = `
|
||||
SELECT uuid, title, latest_release_date FROM opportunities
|
||||
WHERE source <> ? AND latest_release_date >= ? AND latest_release_date < ?`
|
||||
|
||||
// ---------- opportunity_releases ----------
|
||||
|
||||
// insertRelease stores a release; 0 rows affected means it was already
|
||||
|
||||
Reference in new issue
Block a user