Find a Tender never sends tender.tenderPeriod.startDate, so an
opportunity's start date was always empty, and its end date was the
bid deadline. Start and end are now the contract period: from the
contracts' periods, else the awards' contract periods, else the lots'
contract periods, each date on its own, taking the earliest start and
the latest end of several.
The bid deadline (tenderPeriod.endDate) moves to a new closing_date
column (migration 00009), in the API as closing_date. EU Supply's
pages give no contract period, so its start and end are left empty
and its response deadline is the closing date.
internal/eusupply replaces the old Node scraper (eu-supply.js) for
uk.eu-supply.com. The site has no API, so it posts the public search
form, which now needs the form's current defaults and reads the HTML
with golang.org/x/net/html. Each tender not stored yet has its own
page read too, for its description, contract type, CPV codes and
buyer address. Tenders are stored with the source "EU Supply - UK"
as eusupply-uk-<tender id>, with one release holding the tender as
read.
The site publishes no rate limit, so the client sets its own: one
request at a time, with a random wait between
sources.eu_supply.request_min_gap and request_max_gap (2s to 10s).
The source has the same 7-day backfill as Find a Tender's.
Most of its tenders are also on Find a Tender, so a new one is stored
Hidden and Skipped when another source's opportunity released within
30 days has a title at least 90% alike, and its buyer or description
matches too (80% of the shorter one's significant words).
bwilliams