Added an EU Supply source, scraping the UK portal's public tenders

internal/eusupply replaces the old Node scraper (eu-supply.js) for
uk.eu-supply.com. The site has no API, so it posts the public search
form, which now needs the form's current defaults and reads the HTML
with golang.org/x/net/html. Each tender not stored yet has its own
page read too, for its description, contract type, CPV codes and
buyer address. Tenders are stored with the source "EU Supply - UK"
as eusupply-uk-<tender id>, with one release holding the tender as
read.

The site publishes no rate limit, so the client sets its own: one
request at a time, with a random wait between
sources.eu_supply.request_min_gap and request_max_gap (2s to 10s).
The source has the same 7-day backfill as Find a Tender's.

Most of its tenders are also on Find a Tender, so a new one is stored
Hidden and Skipped when another source's opportunity released within
30 days has a title at least 90% alike, and its buyer or description
matches too (80% of the shorter one's significant words).
This commit is contained in:
bwilliams committed 2026-10-21 13:57:54 +01:00
1 parent 5c62376b6e
commit 3a06132071
25 files changed
+2520 -8

No files matched your search

+5 -1
View File
@@ -75,9 +75,13 @@ supported.
| `sources.find_a_tender.rate_limit_retries` | int | `5` | How many times a request refused for the API's rate limit is waited out (for its `Retry-After`) and retried. 0 fails at the first; can't be negative. `fetcher fts-once --ignore-rate-limit` uses 0 |
| `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`). Empty: the source fails to start if enabled |
| `sources.fts_mock.rate_limit_retries` | int | `5` | As for `find_a_tender` |
| `sources.eu_supply.base_url` | string | `https://uk.eu-supply.com` | Root of EU Supply's UK portal (see `internal/eusupply`) |
| `sources.eu_supply.request_min_gap` | duration | `2s` | The shortest wait between two requests to the portal. Must be greater than 0: the site publishes no rate limit |
| `sources.eu_supply.request_max_gap` | duration | `10s` | The longest wait between two requests; each wait is random, from `request_min_gap`. Must be greater than 0; less than `request_min_gap` is read as `request_min_gap` |
| `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown |
The current sources are `find_a_tender`, `fts_mock` and `sleeper`. See
The current sources are `find_a_tender`, `fts_mock`, `eu_supply` and
`sleeper`. See
"Addition of a new opportunities source" in the top level README for how to add
a new source.