Added an EU Supply source, scraping the UK portal's public tenders

internal/eusupply replaces the old Node scraper (eu-supply.js) for
uk.eu-supply.com. The site has no API, so it posts the public search
form, which now needs the form's current defaults and reads the HTML
with golang.org/x/net/html. Each tender not stored yet has its own
page read too, for its description, contract type, CPV codes and
buyer address. Tenders are stored with the source "EU Supply - UK"
as eusupply-uk-<tender id>, with one release holding the tender as
read.

The site publishes no rate limit, so the client sets its own: one
request at a time, with a random wait between
sources.eu_supply.request_min_gap and request_max_gap (2s to 10s).
The source has the same 7-day backfill as Find a Tender's.

Most of its tenders are also on Find a Tender, so a new one is stored
Hidden and Skipped when another source's opportunity released within
30 days has a title at least 90% alike, and its buyer or description
matches too (80% of the shorter one's significant words).
This commit is contained in:
bwilliams committed 2026-10-21 13:57:54 +01:00
1 parent 5c62376b6e
commit 3a06132071
25 files changed
+2520 -8

No files matched your search

+26
View File
@@ -866,6 +866,32 @@ func TestLatestReleaseDate(t *testing.T) {
}
}
// ListOtherSourcesTitles lists only other sources' opportunities, with
// their latest release in [from, before).
func TestListOtherSourcesTitles(t *testing.T) {
s := openStore(t)
ctx := context.Background()
mustSave(t, s, opportunity("ocds-1", "Too early", 1, "r1"))
mustSave(t, s, opportunity("ocds-2", "In the window", 3, "r1"))
mustSave(t, s, opportunity("ocds-3", "At the end", 5, "r1"))
own := opportunity("eusupply-uk-1", "Its own source", 3, "r1")
own.Source = "EU Supply - UK"
mustSave(t, s, own)
titles, err := s.ListOtherSourcesTitles(
ctx, "EU Supply - UK", day(2), day(5),
)
if err != nil {
t.Fatal(err)
}
if len(titles) != 1 || titles[0].UUID != "ocds-2" ||
titles[0].Title != "In the window" ||
!titles[0].LatestReleaseDate.Equal(day(3)) {
t.Errorf("titles = %+v; want only ocds-2", titles)
}
}
// NewOpportunity is set only for an opportunity that wasn't already stored,
// by both SaveOpportunity and SaveOpportunities.
func TestSaveResultNewOpportunity(t *testing.T) {