bwilliams
50ee510520
Find a Tender never sends tender.tenderPeriod.startDate, so an opportunity's start date was always empty, and its end date was the bid deadline. Start and end are now the contract period: from the contracts' periods, else the awards' contract periods, else the lots' contract periods, each date on its own, taking the earliest start and the latest end of several. The bid deadline (tenderPeriod.endDate) moves to a new closing_date column (migration 00009), in the API as closing_date. EU Supply's pages give no contract period, so its start and end are left empty and its response deadline is the closing date.
EU Supply
A scraper for the public tenders on EU Supply's UK portal, uk.eu-supply.com. The site (Mercell CTM) has no API, so the client posts its public search form and reads the HTML. It replaces an old Node scraper, which also covered the Sellafield and Shared Systems portals; only the UK portal is still used.
Requests
| Request | Path | Gives |
|---|---|---|
| Search | POST /ctm/Supplier/PublicTenders |
Up to 100 tenders a page: id, reference, title, publication date, response deadline, process, buyers, countries |
| Tender page | GET /ctm/Supplier/PublicPurchase/<PID>/0/0?returnUrl=ctm/Supplier/PublicTenders&b=UK |
Description, type of contract, CPV codes, buyer address, lots |
- The search filters on the publication date, by whole UK days
(
SearchFilter.FromDate/ToDateasDD/MM/YYYY, both included). Only issued tenders are searched (SearchFilter.PublishType1), not awarded ones. - The form must be posted whole, with the portal's own defaults
(
searchForm). With the old scraper's values (OperatorId1 and an emptySavedCategoryId), the portal answers 500. - Times are UK time. Each date cell's tooltip
titleholds the full time (21/10/2026 12:45:00); the cell's text has only the date. - A tender's PID (its pages' id) is only in its search result's link
(
rwlentrance_s.asp?PID=…), which redirects to its page. - A tender page's fields are each a
div.ctm-content-label(CPV codes:) followed by a<p>with the value. - The system certificate authorities are enough: the old scraper's own CA file isn't needed.
Rate limit
The site publishes no rate limit, so the client sets its own: it makes one
request at a time, waiting a random time from the minimum to the maximum gap
(WithRequestGap; the source's request_min_gap and request_max_gap,
2s to 10s by default) after each before making the next.
Opportunities
Convert makes one opportunity per tender, with the source EU Supply - UK
and the UUID eusupply-uk-<tender id>. It has one release (id: the tender
id), dated its publication, tagged tender, whose JSON is the Tender as
read. CPV codes are stored without their check digit, as Find a Tender's
are. The package sets no marks: the fetcher marks the tenders it takes as
repeats of another source's opportunities Hidden and Skipped (see
internal/fetcher/match.go).