bwilliams
3a06132071
internal/eusupply replaces the old Node scraper (eu-supply.js) for uk.eu-supply.com. The site has no API, so it posts the public search form, which now needs the form's current defaults and reads the HTML with golang.org/x/net/html. Each tender not stored yet has its own page read too, for its description, contract type, CPV codes and buyer address. Tenders are stored with the source "EU Supply - UK" as eusupply-uk-<tender id>, with one release holding the tender as read. The site publishes no rate limit, so the client sets its own: one request at a time, with a random wait between sources.eu_supply.request_min_gap and request_max_gap (2s to 10s). The source has the same 7-day backfill as Find a Tender's. Most of its tenders are also on Find a Tender, so a new one is stored Hidden and Skipped when another source's opportunity released within 30 days has a title at least 90% alike, and its buyer or description matches too (80% of the shorter one's significant words).
48 lines
2.3 KiB
Markdown
48 lines
2.3 KiB
Markdown
# EU Supply
|
|
|
|
A scraper for the public tenders on EU Supply's UK portal,
|
|
[uk.eu-supply.com](https://uk.eu-supply.com). The site (Mercell CTM) has no
|
|
API, so the client posts its public search form and reads the HTML. It
|
|
replaces an old Node scraper, which also covered the Sellafield and Shared
|
|
Systems portals; only the UK portal is still used.
|
|
|
|
## Requests
|
|
|
|
| Request | Path | Gives |
|
|
|---|---|---|
|
|
| Search | `POST /ctm/Supplier/PublicTenders` | Up to 100 tenders a page: id, reference, title, publication date, response deadline, process, buyers, countries |
|
|
| Tender page | `GET /ctm/Supplier/PublicPurchase/<PID>/0/0?returnUrl=ctm/Supplier/PublicTenders&b=UK` | Description, type of contract, CPV codes, buyer address, lots |
|
|
|
|
- The search filters on the **publication date**, by whole UK days
|
|
(`SearchFilter.FromDate`/`ToDate` as `DD/MM/YYYY`, both included). Only
|
|
issued tenders are searched (`SearchFilter.PublishType` 1), not awarded
|
|
ones.
|
|
- The form must be posted whole, with the portal's own defaults
|
|
(`searchForm`). With the old scraper's values (`OperatorId` 1 and an
|
|
empty `SavedCategoryId`), the portal answers 500.
|
|
- Times are UK time. Each date cell's tooltip `title` holds the full time
|
|
(`21/10/2026 12:45:00`); the cell's text has only the date.
|
|
- A tender's PID (its pages' id) is only in its search result's link
|
|
(`rwlentrance_s.asp?PID=…`), which redirects to its page.
|
|
- A tender page's fields are each a `div.ctm-content-label` (`CPV codes:`)
|
|
followed by a `<p>` with the value.
|
|
- The system certificate authorities are enough: the old scraper's own CA
|
|
file isn't needed.
|
|
|
|
## Rate limit
|
|
|
|
The site publishes no rate limit, so the client sets its own: it makes one
|
|
request at a time, waiting a random time from the minimum to the maximum gap
|
|
(`WithRequestGap`; the source's `request_min_gap` and `request_max_gap`,
|
|
2s to 10s by default) after each before making the next.
|
|
|
|
## Opportunities
|
|
|
|
`Convert` makes one opportunity per tender, with the source `EU Supply - UK`
|
|
and the UUID `eusupply-uk-<tender id>`. It has one release (id: the tender
|
|
id), dated its publication, tagged `tender`, whose JSON is the `Tender` as
|
|
read. CPV codes are stored without their check digit, as Find a Tender's
|
|
are. The package sets no marks: the fetcher marks the tenders it takes as
|
|
repeats of another source's opportunities Hidden and Skipped (see
|
|
`internal/fetcher/match.go`).
|