Added an EU Supply source, scraping the UK portal's public tenders
internal/eusupply replaces the old Node scraper (eu-supply.js) for uk.eu-supply.com. The site has no API, so it posts the public search form, which now needs the form's current defaults and reads the HTML with golang.org/x/net/html. Each tender not stored yet has its own page read too, for its description, contract type, CPV codes and buyer address. Tenders are stored with the source "EU Supply - UK" as eusupply-uk-<tender id>, with one release holding the tender as read. The site publishes no rate limit, so the client sets its own: one request at a time, with a random wait between sources.eu_supply.request_min_gap and request_max_gap (2s to 10s). The source has the same 7-day backfill as Find a Tender's. Most of its tenders are also on Find a Tender, so a new one is stored Hidden and Skipped when another source's opportunity released within 30 days has a title at least 90% alike, and its buyer or description matches too (80% of the shorter one's significant words).
This commit is contained in:
1 parent
5c62376b6e
commit
3a06132071
25 files changed
+2520
-8
No files matched your search
@@ -0,0 +1,47 @@
|
||||
# EU Supply
|
||||
|
||||
A scraper for the public tenders on EU Supply's UK portal,
|
||||
[uk.eu-supply.com](https://uk.eu-supply.com). The site (Mercell CTM) has no
|
||||
API, so the client posts its public search form and reads the HTML. It
|
||||
replaces an old Node scraper, which also covered the Sellafield and Shared
|
||||
Systems portals; only the UK portal is still used.
|
||||
|
||||
## Requests
|
||||
|
||||
| Request | Path | Gives |
|
||||
|---|---|---|
|
||||
| Search | `POST /ctm/Supplier/PublicTenders` | Up to 100 tenders a page: id, reference, title, publication date, response deadline, process, buyers, countries |
|
||||
| Tender page | `GET /ctm/Supplier/PublicPurchase/<PID>/0/0?returnUrl=ctm/Supplier/PublicTenders&b=UK` | Description, type of contract, CPV codes, buyer address, lots |
|
||||
|
||||
- The search filters on the **publication date**, by whole UK days
|
||||
(`SearchFilter.FromDate`/`ToDate` as `DD/MM/YYYY`, both included). Only
|
||||
issued tenders are searched (`SearchFilter.PublishType` 1), not awarded
|
||||
ones.
|
||||
- The form must be posted whole, with the portal's own defaults
|
||||
(`searchForm`). With the old scraper's values (`OperatorId` 1 and an
|
||||
empty `SavedCategoryId`), the portal answers 500.
|
||||
- Times are UK time. Each date cell's tooltip `title` holds the full time
|
||||
(`21/10/2026 12:45:00`); the cell's text has only the date.
|
||||
- A tender's PID (its pages' id) is only in its search result's link
|
||||
(`rwlentrance_s.asp?PID=…`), which redirects to its page.
|
||||
- A tender page's fields are each a `div.ctm-content-label` (`CPV codes:`)
|
||||
followed by a `<p>` with the value.
|
||||
- The system certificate authorities are enough: the old scraper's own CA
|
||||
file isn't needed.
|
||||
|
||||
## Rate limit
|
||||
|
||||
The site publishes no rate limit, so the client sets its own: it makes one
|
||||
request at a time, waiting a random time from the minimum to the maximum gap
|
||||
(`WithRequestGap`; the source's `request_min_gap` and `request_max_gap`,
|
||||
2s to 10s by default) after each before making the next.
|
||||
|
||||
## Opportunities
|
||||
|
||||
`Convert` makes one opportunity per tender, with the source `EU Supply - UK`
|
||||
and the UUID `eusupply-uk-<tender id>`. It has one release (id: the tender
|
||||
id), dated its publication, tagged `tender`, whose JSON is the `Tender` as
|
||||
read. CPV codes are stored without their check digit, as Find a Tender's
|
||||
are. The package sets no marks: the fetcher marks the tenders it takes as
|
||||
repeats of another source's opportunities Hidden and Skipped (see
|
||||
`internal/fetcher/match.go`).
|
||||
Reference in new issue
Block a user