bwilliams
3a06132071
internal/eusupply replaces the old Node scraper (eu-supply.js) for uk.eu-supply.com. The site has no API, so it posts the public search form, which now needs the form's current defaults and reads the HTML with golang.org/x/net/html. Each tender not stored yet has its own page read too, for its description, contract type, CPV codes and buyer address. Tenders are stored with the source "EU Supply - UK" as eusupply-uk-<tender id>, with one release holding the tender as read. The site publishes no rate limit, so the client sets its own: one request at a time, with a random wait between sources.eu_supply.request_min_gap and request_max_gap (2s to 10s). The source has the same 7-day backfill as Find a Tender's. Most of its tenders are also on Find a Tender, so a new one is stored Hidden and Skipped when another source's opportunity released within 30 days has a title at least 90% alike, and its buyer or description matches too (80% of the shorter one's significant words).
2.3 KiB
EU Supply
A scraper for the public tenders on EU Supply's UK portal, uk.eu-supply.com. The site (Mercell CTM) has no API, so the client posts its public search form and reads the HTML. It replaces an old Node scraper, which also covered the Sellafield and Shared Systems portals; only the UK portal is still used.
Requests
| Request | Path | Gives |
|---|---|---|
| Search | POST /ctm/Supplier/PublicTenders |
Up to 100 tenders a page: id, reference, title, publication date, response deadline, process, buyers, countries |
| Tender page | GET /ctm/Supplier/PublicPurchase/<PID>/0/0?returnUrl=ctm/Supplier/PublicTenders&b=UK |
Description, type of contract, CPV codes, buyer address, lots |
- The search filters on the publication date, by whole UK days
(
SearchFilter.FromDate/ToDateasDD/MM/YYYY, both included). Only issued tenders are searched (SearchFilter.PublishType1), not awarded ones. - The form must be posted whole, with the portal's own defaults
(
searchForm). With the old scraper's values (OperatorId1 and an emptySavedCategoryId), the portal answers 500. - Times are UK time. Each date cell's tooltip
titleholds the full time (21/10/2026 12:45:00); the cell's text has only the date. - A tender's PID (its pages' id) is only in its search result's link
(
rwlentrance_s.asp?PID=…), which redirects to its page. - A tender page's fields are each a
div.ctm-content-label(CPV codes:) followed by a<p>with the value. - The system certificate authorities are enough: the old scraper's own CA file isn't needed.
Rate limit
The site publishes no rate limit, so the client sets its own: it makes one
request at a time, waiting a random time from the minimum to the maximum gap
(WithRequestGap; the source's request_min_gap and request_max_gap,
2s to 10s by default) after each before making the next.
Opportunities
Convert makes one opportunity per tender, with the source EU Supply - UK
and the UUID eusupply-uk-<tender id>. It has one release (id: the tender
id), dated its publication, tagged tender, whose JSON is the Tender as
read. CPV codes are stored without their check digit, as Find a Tender's
are. The package sets no marks: the fetcher marks the tenders it takes as
repeats of another source's opportunities Hidden and Skipped (see
internal/fetcher/match.go).