Added an EU Supply source, scraping the UK portal's public tenders

internal/eusupply replaces the old Node scraper (eu-supply.js) for
uk.eu-supply.com. The site has no API, so it posts the public search
form, which now needs the form's current defaults and reads the HTML
with golang.org/x/net/html. Each tender not stored yet has its own
page read too, for its description, contract type, CPV codes and
buyer address. Tenders are stored with the source "EU Supply - UK"
as eusupply-uk-<tender id>, with one release holding the tender as
read.

The site publishes no rate limit, so the client sets its own: one
request at a time, with a random wait between
sources.eu_supply.request_min_gap and request_max_gap (2s to 10s).
The source has the same 7-day backfill as Find a Tender's.

Most of its tenders are also on Find a Tender, so a new one is stored
Hidden and Skipped when another source's opportunity released within
30 days has a title at least 90% alike, and its buyer or description
matches too (80% of the shorter one's significant words).
This commit is contained in:
bwilliams committed 2026-10-21 13:57:54 +01:00
1 parent 5c62376b6e
commit 3a06132071
25 files changed
+2520 -8

No files matched your search

+31
View File
@@ -1041,6 +1041,37 @@ func (s *Store) LatestReleaseDate(
return latest, nil
}
// OpportunityTitle is an opportunity's UUID, title and latest release date,
// as ListOtherSourcesTitles lists them.
type OpportunityTitle struct {
UUID string `db:"uuid"`
Title string `db:"title"`
LatestReleaseDate time.Time `db:"latest_release_date"`
}
// ListOtherSourcesTitles returns the UUID, title and latest release date of
// every opportunity not from source whose latest release is from from
// (inclusive) to before (exclusive), so a fetcher can find the ones a new
// opportunity repeats.
func (s *Store) ListOtherSourcesTitles(
ctx context.Context, source string, from, before time.Time,
) ([]OpportunityTitle, error) {
titles := []OpportunityTitle{}
err := s.db.SelectContext(
ctx,
&titles,
selectOtherSourcesTitles,
source,
from.UTC(),
before.UTC(),
)
if err != nil {
return nil, fmt.Errorf("listing titles of sources but %s: %w", source, err)
}
return titles, nil
}
// ReleaseExists reports whether a release is already stored, so a fetcher
// can skip converting it.
func (s *Store) ReleaseExists(