Added an EU Supply source, scraping the UK portal's public tenders

internal/eusupply replaces the old Node scraper (eu-supply.js) for
uk.eu-supply.com. The site has no API, so it posts the public search
form, which now needs the form's current defaults and reads the HTML
with golang.org/x/net/html. Each tender not stored yet has its own
page read too, for its description, contract type, CPV codes and
buyer address. Tenders are stored with the source "EU Supply - UK"
as eusupply-uk-<tender id>, with one release holding the tender as
read.

The site publishes no rate limit, so the client sets its own: one
request at a time, with a random wait between
sources.eu_supply.request_min_gap and request_max_gap (2s to 10s).
The source has the same 7-day backfill as Find a Tender's.

Most of its tenders are also on Find a Tender, so a new one is stored
Hidden and Skipped when another source's opportunity released within
30 days has a title at least 90% alike, and its buyer or description
matches too (80% of the shorter one's significant words).
This commit is contained in:
bwilliams committed 2026-10-21 13:57:54 +01:00
1 parent 5c62376b6e
commit 3a06132071
25 files changed
+2520 -8

No files matched your search

+15 -1
View File
@@ -120,7 +120,8 @@ nothing stored, Find a Tender's client fetches the last day. `--since`
overrides that for every source.
When the start is more than 7 days ago (an early `--since`, or a long time
offline), the Find a Tender sources first backfill it in 7-day windows,
offline), the Find a Tender and EU Supply sources first backfill it in 7-day
windows,
oldest first, saving each before fetching the next. A window that fails is
saved as far as it got and fetched again after the source's interval. The
API returns the newest releases first, so if a backfill is stopped partway
@@ -132,6 +133,18 @@ A rate limit is waited out, as the API asks, up to
`sources.<source>.rate_limit_retries` times per request. A failed fetch
keeps what it fetched before the failure.
### EU Supply
The `eu_supply` source scrapes the public tenders on EU Supply's UK portal
(`uk.eu-supply.com`), stored with the source `EU Supply - UK`. The site has
no API and publishes no rate limit, so the client waits a random time from
`request_min_gap` to `request_max_gap` (2s to 10s) between its requests. Each
tender not stored yet has its own page read too, for its description and
CPV codes. Most of its tenders are also published on Find a Tender: a new
one is stored Hidden and Skipped when another source's opportunity released
within 30 days of it has a title at least 90% alike, and its buyer or its
description matches too (80% of the shorter one's words). See
[internal/eusupply](internal/eusupply/README.md).
`fts-once` can be limited with `--planning`, `--opportunities` or
`--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
of waiting.
@@ -191,6 +204,7 @@ second one fails with "address already in use".
| `internal/skiprules` | The rules that mark new opportunities Skipped (see [internal/skiprules](internal/skiprules/README.md)) |
| `internal/store` | SQLite storage of opportunities, and its migrations (see [internal/store](internal/store/README.md)) |
| `internal/findatender` | FTS API client and conversion from OCDS to JobSearch types |
| `internal/eusupply` | EU Supply UK portal scraper and conversion to JobSearch types (see [internal/eusupply](internal/eusupply/README.md)) |
| `internal/types` | JobSearch's own opportunity types |
| `internal/util` | The periodic `Worker` used by the fetcher |
| `internal/logging` | slog setup and the HTTP access log |