Added an EU Supply source, scraping the UK portal's public tenders

internal/eusupply replaces the old Node scraper (eu-supply.js) for
uk.eu-supply.com. The site has no API, so it posts the public search
form, which now needs the form's current defaults and reads the HTML
with golang.org/x/net/html. Each tender not stored yet has its own
page read too, for its description, contract type, CPV codes and
buyer address. Tenders are stored with the source "EU Supply - UK"
as eusupply-uk-<tender id>, with one release holding the tender as
read.

The site publishes no rate limit, so the client sets its own: one
request at a time, with a random wait between
sources.eu_supply.request_min_gap and request_max_gap (2s to 10s).
The source has the same 7-day backfill as Find a Tender's.

Most of its tenders are also on Find a Tender, so a new one is stored
Hidden and Skipped when another source's opportunity released within
30 days has a title at least 90% alike, and its buyer or description
matches too (80% of the shorter one's significant words).

internal/eusupply/README.md

@@ -0,0 +1,47 @@
+# EU Supply
+
+A scraper for the public tenders on EU Supply's UK portal,
+[uk.eu-supply.com](https://uk.eu-supply.com). The site (Mercell CTM) has no
+API, so the client posts its public search form and reads the HTML. It
+replaces an old Node scraper, which also covered the Sellafield and Shared
+Systems portals; only the UK portal is still used.
+
+## Requests
+
+| Request | Path | Gives |
+|---|---|---|
+| Search | `POST /ctm/Supplier/PublicTenders` | Up to 100 tenders a page: id, reference, title, publication date, response deadline, process, buyers, countries |
+| Tender page | `GET /ctm/Supplier/PublicPurchase/<PID>/0/0?returnUrl=ctm/Supplier/PublicTenders&b=UK` | Description, type of contract, CPV codes, buyer address, lots |
+
+- The search filters on the **publication date**, by whole UK days
+  (`SearchFilter.FromDate`/`ToDate` as `DD/MM/YYYY`, both included). Only
+  issued tenders are searched (`SearchFilter.PublishType` 1), not awarded
+  ones.
+- The form must be posted whole, with the portal's own defaults
+  (`searchForm`). With the old scraper's values (`OperatorId` 1 and an
+  empty `SavedCategoryId`), the portal answers 500.
+- Times are UK time. Each date cell's tooltip `title` holds the full time
+  (`21/10/2026 12:45:00`); the cell's text has only the date.
+- A tender's PID (its pages' id) is only in its search result's link
+  (`rwlentrance_s.asp?PID=…`), which redirects to its page.
+- A tender page's fields are each a `div.ctm-content-label` (`CPV codes:`)
+  followed by a `<p>` with the value.
+- The system certificate authorities are enough: the old scraper's own CA
+  file isn't needed.
+
+## Rate limit
+
+The site publishes no rate limit, so the client sets its own: it makes one
+request at a time, waiting a random time from the minimum to the maximum gap
+(`WithRequestGap`; the source's `request_min_gap` and `request_max_gap`,
+2s to 10s by default) after each before making the next.
+
+## Opportunities
+
+`Convert` makes one opportunity per tender, with the source `EU Supply - UK`
+and the UUID `eusupply-uk-<tender id>`. It has one release (id: the tender
+id), dated its publication, tagged `tender`, whose JSON is the `Tender` as
+read. CPV codes are stored without their check digit, as Find a Tender's
+are. The package sets no marks: the fetcher marks the tenders it takes as
+repeats of another source's opportunities Hidden and Skipped (see
+`internal/fetcher/match.go`).

internal/eusupply/convert.go

@@ -0,0 +1,101 @@
+package eusupply
+
+import (
+	"encoding/json"
+	"strings"
+
+	"git.avsdev.uk/Go/job-search/internal/types"
+)
+
+// The opportunities Convert makes.
+const (
+	// UUIDPrefix starts the UUID of every opportunity from the UK portal;
+	// the tender's id follows it.
+	UUIDPrefix = "eusupply-uk-"
+	// uuidType is their UUIDType.
+	uuidType = "eusupply"
+	// releaseTag is their one release's tag: the portal lists open tenders.
+	releaseTag = "tender"
+)
+
+// mainCategories maps the portal's types of contract to OCDS's
+// mainProcurementCategory, as Find a Tender's opportunities have.
+var mainCategories = map[string]string{
+	"Supplies": "goods",
+	"Services": "services",
+	"Works":    "works",
+}
+
+// UUID returns the UUID of the opportunity for the tender with id.
+func UUID(id string) string {
+	return UUIDPrefix + id
+}
+
+// ReleaseID returns the id of the one release of the opportunity for
+// tender: its id, as the portal gives each tender one page.
+func ReleaseID(tender Tender) string {
+	return tender.ID
+}
+
+// Convert returns the opportunity for each tender, with one release whose
+// JSON is the tender as read.
+func Convert(tenders []Tender) []*types.Opportunity {
+	opportunities := make([]*types.Opportunity, 0, len(tenders))
+	for _, tender := range tenders {
+		opportunities = append(opportunities, convertOne(tender))
+	}
+	return opportunities
+}
+
+// convertOne builds the opportunity for one tender.
+func convertOne(tender Tender) *types.Opportunity {
+	published := tender.Published
+	opportunity := &types.Opportunity{
+		UUID:              UUID(tender.ID),
+		UUIDType:          uuidType,
+		Title:             tender.Title,
+		Status:            "active",
+		Description:       tender.Description,
+		BuyerName:         tender.Buyer,
+		Classifications:   classifications(tender),
+		MainCategory:      mainCategories[tender.ContractType],
+		MultiLot:          tender.Lots > 1,
+		StartDate:         &published,
+		EndDate:           tender.Deadline,
+		Source:            Source,
+		LatestReleaseDate: tender.Published,
+	}
+
+	release := types.OpportunityRelease{
+		UUID:        opportunity.UUID,
+		ReleaseID:   ReleaseID(tender),
+		ReleaseDate: tender.Published,
+		Tags: []types.OpportunityReleaseTag{
+			{
+				Tag: releaseTag,
+			},
+		},
+	}
+	// A Tender is plain data, so marshalling it can't fail.
+	if encoded, err := json.Marshal(tender); err == nil {
+		release.JSON = string(encoded)
+	}
+	opportunity.Releases = []types.OpportunityRelease{release}
+
+	return opportunity
+}
+
+// classifications returns the tender's CPV codes as Find a Tender's are
+// stored: without the check digit ("48000000", not "48000000-8").
+func classifications(tender Tender) []types.ClassificationCode {
+	codes := make([]types.ClassificationCode, 0, len(tender.CPVCodes))
+	for _, cpv := range tender.CPVCodes {
+		code, _, _ := strings.Cut(cpv.Code, "-")
+		codes = append(codes, types.ClassificationCode{
+			IsCPV:          true,
+			Code:           code,
+			Classification: cpv.Description,
+		})
+	}
+	return codes
+}

internal/eusupply/details.go

@@ -0,0 +1,108 @@
+package eusupply
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+
+	"golang.org/x/net/html"
+)
+
+// ErrNoDetails means a tender's page had none of the labelled fields the
+// public page always has: the page has changed, or isn't a tender's.
+var ErrNoDetails = errors.New("eusupply: no tender details on the page")
+
+// Details reads tender's own page and fills in its description, contract
+// type, CPV codes, buyer address and number of lots, and sets HasDetails.
+func (c *Client) Details(ctx context.Context, tender *Tender) error {
+	body, err := c.get(ctx, tenderPath(tender.PID))
+	if err != nil {
+		return fmt.Errorf("reading tender %s: %w", tender.ID, err)
+	}
+
+	if err := parseDetails(body, tender); err != nil {
+		return fmt.Errorf("reading tender %s: %w", tender.ID, err)
+	}
+
+	return nil
+}
+
+// parseDetails reads a tender's page into tender. Each field is a
+// "ctm-content-label" div ("CPV codes:") followed by a <p> holding its
+// value.
+func parseDetails(body []byte, tender *Tender) error {
+	document, err := html.Parse(bytes.NewReader(body))
+	if err != nil {
+		return err
+	}
+
+	values := make(map[string]*html.Node)
+	labels := findAll(document, hasClass("div", "ctm-content-label"))
+	for _, label := range labels {
+		value := nextElement(label)
+		if value == nil || value.Data != "p" {
+			continue
+		}
+		name := strings.TrimSuffix(text(label), ":")
+		if _, found := values[name]; !found {
+			values[name] = value
+		}
+	}
+	if len(values) == 0 {
+		return ErrNoDetails
+	}
+
+	if value := values["Detailed description"]; value != nil {
+		tender.Description = strings.TrimSpace(rawText(value))
+	}
+	if value := values["Type of contract"]; value != nil {
+		tender.ContractType = text(value)
+	}
+	if value := values["CPV codes"]; value != nil {
+		tender.CPVCodes = parseCPVCodes(value)
+	}
+	if value := values["Buyer"]; value != nil {
+		tender.BuyerAddress = buyerAddress(value)
+	}
+	if deadline := values["Response deadline (UK time)"]; deadline != nil &&
+		tender.Deadline == nil {
+		if parsed, err := parseUKTime(text(deadline)); err == nil {
+			tender.Deadline = &parsed
+		}
+	}
+	tender.Lots = len(findAll(document, hasClass("div", "package-container")))
+	tender.HasDetails = true
+
+	return nil
+}
+
+// parseCPVCodes reads the CPV codes listed in value, one per <span>: the
+// code, two non-breaking spaces, then its description.
+func parseCPVCodes(value *html.Node) []CPVCode {
+	var codes []CPVCode
+	for _, span := range findAll(value, isTag("span")) {
+		code, description, _ := strings.Cut(text(span), " ")
+		if code == "" {
+			continue
+		}
+		codes = append(codes, CPVCode{
+			Code:        code,
+			Description: description,
+		})
+	}
+	return codes
+}
+
+// buyerAddress reads the buyer's name and address, one line each, leaving
+// out the "View profile" link.
+func buyerAddress(value *html.Node) []string {
+	var address []string
+	for _, line := range lines(value) {
+		if line != "View profile" {
+			address = append(address, line)
+		}
+	}
+	return address
+}

internal/eusupply/eusupply.go

@@ -0,0 +1,208 @@
+// Package eusupply scrapes the public tenders listed on EU Supply's UK
+// portal (uk.eu-supply.com), and converts them to opportunities.
+//
+// The site has no API: the client posts the portal's search form and reads
+// the HTML it returns, then reads each tender's own page for its
+// description and CPV codes. The client limits its own request rate (see
+// WithRequestGap), as the site publishes no limit to wait out.
+package eusupply
+
+import (
+	"cmp"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"math/rand/v2"
+	"net/http"
+	"net/http/cookiejar"
+	"net/url"
+	"strings"
+	"sync"
+	"time"
+	_ "time/tzdata" // the portal's times are UK time, wherever this runs
+)
+
+// The defaults of the client's options.
+const (
+	// DefaultBaseURL is the UK portal's root.
+	DefaultBaseURL = "https://uk.eu-supply.com"
+	// DefaultMinRequestGap and DefaultMaxRequestGap bound the random wait
+	// between two requests (see WithRequestGap).
+	DefaultMinRequestGap = 2 * time.Second
+	DefaultMaxRequestGap = 10 * time.Second
+	// requestTimeout is how long one request may take.
+	requestTimeout = 60 * time.Second
+)
+
+// Source is the Source of the opportunities Convert makes.
+const Source = "EU Supply - UK"
+
+// ErrUnexpectedStatus wraps a response status other than 200 OK.
+var ErrUnexpectedStatus = errors.New("eusupply: unexpected response status")
+
+// ukTime is the zone of the portal's dates and times.
+var ukTime = mustLoadLocation("Europe/London")
+
+// Client scrapes one EU Supply portal. Create it with New; it is safe for
+// concurrent use, though its requests are made one at a time.
+type Client struct {
+	baseURL    string
+	httpClient *http.Client
+	log        *slog.Logger
+
+	minGap, maxGap time.Duration
+
+	// mu serialises requests, so the gap between them holds; lastRequest is
+	// when the previous one finished (zero: none yet).
+	mu          sync.Mutex
+	lastRequest time.Time
+}
+
+// Option configures a Client.
+type Option func(*Client)
+
+// WithLogger makes the client log to logger instead of slog.Default().
+func WithLogger(logger *slog.Logger) Option {
+	return func(c *Client) {
+		c.log = logger
+	}
+}
+
+// WithRequestGap makes the client wait a random time from minGap to maxGap
+// after each request before making the next. A maxGap less than minGap is
+// read as minGap; 0 or less for both means no wait, for tests.
+func WithRequestGap(minGap, maxGap time.Duration) Option {
+	return func(c *Client) {
+		c.minGap = minGap
+		c.maxGap = max(minGap, maxGap)
+	}
+}
+
+// New returns a client for the portal at baseURL (DefaultBaseURL if empty).
+// It waits DefaultMinRequestGap to DefaultMaxRequestGap between requests
+// unless given WithRequestGap, and logs to slog.Default() unless given
+// WithLogger.
+func New(baseURL string, opts ...Option) *Client {
+	jar, _ := cookiejar.New(nil) // cookiejar.New never returns an error
+	client := &Client{
+		baseURL: strings.TrimSuffix(cmp.Or(baseURL, DefaultBaseURL), "/"),
+		httpClient: &http.Client{
+			Timeout: requestTimeout,
+			Jar:     jar,
+		},
+		minGap: DefaultMinRequestGap,
+		maxGap: DefaultMaxRequestGap,
+	}
+	for _, opt := range opts {
+		opt(client)
+	}
+
+	return client
+}
+
+// logger returns the logger the client logs to, tagged with its component.
+func (c *Client) logger() *slog.Logger {
+	log := c.log
+	if log == nil {
+		log = slog.Default()
+	}
+	return log.With("component", "eusupply")
+}
+
+// do makes one request, once the gap since the previous request has passed,
+// and returns its body. A status other than 200 OK fails with
+// ErrUnexpectedStatus. It returns the request context's error if that is
+// cancelled while it waits.
+func (c *Client) do(req *http.Request) ([]byte, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	if err := c.wait(req.Context()); err != nil {
+		return nil, err
+	}
+	defer func() {
+		c.lastRequest = time.Now()
+	}()
+
+	c.logger().Info("fetching", "method", req.Method, "uri", req.URL.String())
+
+	res, err := c.httpClient.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer res.Body.Close() //nolint:errcheck // read only
+
+	if res.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("%w: %s", ErrUnexpectedStatus, res.Status)
+	}
+
+	body, err := io.ReadAll(res.Body)
+	if err != nil {
+		return nil, fmt.Errorf("reading %s: %w", req.URL, err)
+	}
+
+	return body, nil
+}
+
+// wait sleeps until a random gap (see WithRequestGap) has passed since the
+// previous request, or ctx is cancelled. c.mu must be held.
+func (c *Client) wait(ctx context.Context) error {
+	if c.lastRequest.IsZero() || c.maxGap <= 0 {
+		return ctx.Err()
+	}
+
+	gap := c.minGap
+	if c.maxGap > c.minGap {
+		//nolint:gosec // G404: a jitter, not a secret
+		gap += rand.N(c.maxGap - c.minGap + 1)
+	}
+
+	timer := time.NewTimer(time.Until(c.lastRequest.Add(gap)))
+	defer timer.Stop()
+
+	select {
+	case <-timer.C:
+		return nil
+	case <-ctx.Done():
+		return ctx.Err()
+	}
+}
+
+// get makes a GET request for the portal's path.
+func (c *Client) get(ctx context.Context, path string) ([]byte, error) {
+	req, err := http.NewRequestWithContext(
+		ctx, http.MethodGet, c.baseURL+path, nil,
+	)
+	if err != nil {
+		return nil, err
+	}
+	return c.do(req)
+}
+
+// post posts form to the portal's path.
+func (c *Client) post(
+	ctx context.Context, path string, form url.Values,
+) ([]byte, error) {
+	req, err := http.NewRequestWithContext(
+		ctx,
+		http.MethodPost,
+		c.baseURL+path,
+		strings.NewReader(form.Encode()),
+	)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	return c.do(req)
+}
+
+// mustLoadLocation loads a time zone known to be in the embedded tzdata.
+func mustLoadLocation(name string) *time.Location {
+	location, err := time.LoadLocation(name)
+	if err != nil {
+		panic(err)
+	}
+	return location
+}

internal/eusupply/eusupply_test.go

@@ -0,0 +1,326 @@
+package eusupply
+
+import (
+	"context"
+	"errors"
+	"io"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"os"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+// readFixture returns a file in testdata.
+func readFixture(t *testing.T, name string) []byte {
+	t.Helper()
+	body, err := os.ReadFile("testdata/" + name) //nolint:gosec // G304: a test fixture
+	if err != nil {
+		t.Fatal(err)
+	}
+	return body
+}
+
+// newTestClient returns a client for srv that doesn't wait between
+// requests and logs nothing.
+func newTestClient(srv *httptest.Server) *Client {
+	return New(
+		srv.URL,
+		WithRequestGap(0, 0),
+		WithLogger(slog.New(slog.DiscardHandler)),
+	)
+}
+
+func TestParseSearchPage(t *testing.T) {
+	c := New("https://uk.example", WithLogger(slog.New(slog.DiscardHandler)))
+	tenders, pages, err := c.parseSearchPage(readFixture(t, "search.html"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if pages != 2 {
+		t.Errorf("pages = %d; want 2", pages)
+	}
+	// The row with no link is skipped.
+	if len(tenders) != 2 {
+		t.Fatalf("tenders = %d; want 2: %+v", len(tenders), tenders)
+	}
+
+	first := tenders[0]
+	wantPublished := time.Date(2026, 10, 21, 11, 45, 0, 0, time.UTC) // BST
+	wantDeadline := time.Date(2026, 11, 4, 12, 0, 0, 0, time.UTC)    // GMT
+	switch {
+	case first.ID != "100114946", first.PID != "109912",
+		first.Reference != "Proc2852":
+		t.Errorf("ids = %q, %q, %q", first.ID, first.PID, first.Reference)
+	case first.Title != "Jisc Google for Education Reseller Framework Agreement":
+		t.Errorf("title = %q", first.Title)
+	case first.URL != "https://uk.example/ctm/Supplier/PublicPurchase/109912/0/0?returnUrl=ctm/Supplier/PublicTenders&b=UK":
+		t.Errorf("url = %q", first.URL)
+	case !first.Published.Equal(wantPublished):
+		t.Errorf("published = %v; want %v", first.Published, wantPublished)
+	case first.Deadline == nil || !first.Deadline.Equal(wantDeadline):
+		t.Errorf("deadline = %v; want %v", first.Deadline, wantDeadline)
+	case first.Process != "PA23 - Competitive Flexible", first.Buyer != "Jisc",
+		first.Countries != "United Kingdom":
+		t.Errorf("cells = %q, %q, %q", first.Process, first.Buyer, first.Countries)
+	case first.HasDetails:
+		t.Error("HasDetails set from the search page")
+	}
+
+	second := tenders[1]
+	if second.Reference != "" || second.Deadline != nil {
+		t.Errorf("second = %+v; want no reference or deadline", second)
+	}
+	// Zero width spaces are dropped from titles.
+	if second.Title != "York and North Yorkshire Transport Sub-strategies" {
+		t.Errorf("second title = %q", second.Title)
+	}
+}
+
+func TestParseSearchPageChanged(t *testing.T) {
+	c := New("", WithLogger(slog.New(slog.DiscardHandler)))
+	page := []byte("<html><body>Maintenance</body></html>")
+	_, _, err := c.parseSearchPage(page)
+	if !errors.Is(err, ErrNoResults) {
+		t.Errorf("err = %v; want ErrNoResults", err)
+	}
+}
+
+func TestParseDetails(t *testing.T) {
+	tender := Tender{
+		ID: "100114946",
+	}
+	if err := parseDetails(readFixture(t, "tender.html"), &tender); err != nil {
+		t.Fatal(err)
+	}
+
+	wantDescription := "This opportunity is for a framework agreement.\n" +
+		"The Supplier(s) shall manage the full licence lifecycle (the 'Agreement')."
+	if tender.Description != wantDescription {
+		t.Errorf("description = %q", tender.Description)
+	}
+	if tender.ContractType != "Supplies" {
+		t.Errorf("contract type = %q", tender.ContractType)
+	}
+	if len(tender.CPVCodes) != 2 ||
+		tender.CPVCodes[0].Code != "48000000-8" ||
+		tender.CPVCodes[0].Description !=
+			"Software package and information systems" {
+		t.Errorf("cpv codes = %+v", tender.CPVCodes)
+	}
+	wantAddress := "Jisc|4 Portwall Lane|BS1 6NB|Bristol|United Kingdom"
+	if got := strings.Join(tender.BuyerAddress, "|"); got != wantAddress {
+		t.Errorf("buyer address = %q", got)
+	}
+	// The search had no deadline, so the page's is used.
+	if tender.Deadline == nil ||
+		!tender.Deadline.Equal(time.Date(2026, 11, 4, 12, 0, 0, 0, time.UTC)) {
+		t.Errorf("deadline = %v", tender.Deadline)
+	}
+	if tender.Lots != 2 || !tender.HasDetails {
+		t.Errorf("lots = %d, HasDetails = %v", tender.Lots, tender.HasDetails)
+	}
+
+	err := parseDetails([]byte("<html><body>Not found</body></html>"), &tender)
+	if !errors.Is(err, ErrNoDetails) {
+		t.Errorf("err = %v; want ErrNoDetails", err)
+	}
+}
+
+// Search posts the portal's form for each page, and keeps the pages read
+// before one fails.
+func TestSearch(t *testing.T) {
+	page := readFixture(t, "search.html")
+	var (
+		mu    sync.Mutex
+		forms []url.Values
+	)
+	fail := false
+	srv := httptest.NewServer(
+		http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			if r.Method != http.MethodPost || r.URL.Path != searchPath {
+				t.Errorf("request = %s %s", r.Method, r.URL.Path)
+			}
+			r.ParseForm()
+			mu.Lock()
+			forms = append(forms, r.PostForm)
+			number := len(forms)
+			mu.Unlock()
+			if fail && number == 2 {
+				w.WriteHeader(http.StatusInternalServerError)
+				return
+			}
+			w.Write(page)
+		}),
+	)
+	t.Cleanup(srv.Close)
+	c := newTestClient(srv)
+
+	from := time.Date(2026, 10, 14, 23, 30, 0, 0, time.UTC) // 15th in the UK
+	to := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC)
+	tenders, err := c.Search(context.Background(), from, to)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(tenders) != 4 || len(forms) != 2 {
+		t.Fatalf(
+			"%d tenders from %d pages; want 4 from 2", len(tenders), len(forms),
+		)
+	}
+	for i, form := range forms {
+		wantPage := []string{"1", "2"}[i]
+		if form.Get("SearchFilter.FromDate") != "15/10/2026" ||
+			form.Get("SearchFilter.ToDate") != "21/10/2026" ||
+			form.Get("SearchFilter.PagingInfo.PageNumber") != wantPage ||
+			form.Get("SearchFilter.PagingInfo.PageSize") != "100" ||
+			form.Get("Branding") != "UK" {
+			t.Errorf("form %d = %v", i+1, form)
+		}
+	}
+
+	forms = nil
+	fail = true
+	tenders, err = c.Search(context.Background(), from, to)
+	if !errors.Is(err, ErrUnexpectedStatus) {
+		t.Errorf("err = %v; want ErrUnexpectedStatus", err)
+	}
+	if len(tenders) != 2 {
+		t.Errorf("%d tenders; want the first page's 2", len(tenders))
+	}
+}
+
+func TestDetails(t *testing.T) {
+	var path string
+	srv := httptest.NewServer(
+		http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			path = r.URL.RequestURI()
+			w.Write(readFixture(t, "tender.html"))
+		}),
+	)
+	t.Cleanup(srv.Close)
+
+	tender := Tender{
+		ID:  "100114946",
+		PID: "109912",
+	}
+	err := newTestClient(srv).Details(context.Background(), &tender)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if path != tenderPath("109912") || !tender.HasDetails {
+		t.Errorf("path = %q, HasDetails = %v", path, tender.HasDetails)
+	}
+}
+
+// The client waits from minGap to maxGap between requests, and a cancelled
+// context stops the wait.
+func TestRequestGap(t *testing.T) {
+	var (
+		mu    sync.Mutex
+		times []time.Time
+	)
+	srv := httptest.NewServer(
+		http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			mu.Lock()
+			times = append(times, time.Now())
+			mu.Unlock()
+			io.WriteString(w, "ok")
+		}),
+	)
+	t.Cleanup(srv.Close)
+
+	const minGap, maxGap = 50 * time.Millisecond, 80 * time.Millisecond
+	c := New(
+		srv.URL,
+		WithRequestGap(minGap, maxGap),
+		WithLogger(slog.New(slog.DiscardHandler)),
+	)
+	for range 3 {
+		if _, err := c.get(context.Background(), "/"); err != nil {
+			t.Fatal(err)
+		}
+	}
+	for i := 1; i < len(times); i++ {
+		if gap := times[i].Sub(times[i-1]); gap < minGap {
+			t.Errorf("gap %d = %v; want at least %v", i, gap, minGap)
+		}
+	}
+
+	slow := New(
+		srv.URL,
+		WithRequestGap(time.Hour, time.Hour),
+		WithLogger(slog.New(slog.DiscardHandler)),
+	)
+	slow.get(context.Background(), "/")
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
+	defer cancel()
+	start := time.Now()
+	if _, err := slow.get(ctx, "/"); !errors.Is(err, context.DeadlineExceeded) {
+		t.Errorf("err = %v; want context.DeadlineExceeded", err)
+	}
+	if time.Since(start) > 5*time.Second {
+		t.Error("cancelling did not stop the wait")
+	}
+}
+
+func TestConvert(t *testing.T) {
+	published := time.Date(2026, 10, 21, 11, 45, 0, 0, time.UTC)
+	tender := Tender{
+		ID:           "100114946",
+		PID:          "109912",
+		Title:        "Hazard Management Tool",
+		Published:    published,
+		Buyer:        "UK Atomic Energy Authority",
+		HasDetails:   true,
+		Description:  "A tool",
+		ContractType: "Services",
+		CPVCodes: []CPVCode{
+			{
+				Code:        "48000000-8",
+				Description: "Software package and information systems",
+			},
+		},
+		Lots: 2,
+	}
+
+	opportunities := Convert([]Tender{tender})
+	if len(opportunities) != 1 {
+		t.Fatalf("%d opportunities; want 1", len(opportunities))
+	}
+	opportunity := opportunities[0]
+	switch {
+	case opportunity.UUID != "eusupply-uk-100114946",
+		opportunity.Source != Source:
+		t.Errorf("uuid, source = %q, %q", opportunity.UUID, opportunity.Source)
+	case opportunity.Title != tender.Title, opportunity.Description != "A tool",
+		opportunity.BuyerName != tender.Buyer, opportunity.Status != "active":
+		t.Errorf("opportunity = %+v", opportunity)
+	case opportunity.MainCategory != "services", !opportunity.MultiLot:
+		t.Errorf(
+			"category, multi-lot = %q, %v",
+			opportunity.MainCategory, opportunity.MultiLot,
+		)
+	case len(opportunity.Classifications) != 1 ||
+		opportunity.Classifications[0].Code != "48000000" ||
+		!opportunity.Classifications[0].IsCPV:
+		t.Errorf("classifications = %+v", opportunity.Classifications)
+	case !opportunity.LatestReleaseDate.Equal(published),
+		opportunity.StartDate == nil || !opportunity.StartDate.Equal(published):
+		t.Errorf(
+			"dates = %v, %v", opportunity.LatestReleaseDate, opportunity.StartDate,
+		)
+	case opportunity.Marks.Hidden || opportunity.Marks.Skipped:
+		t.Error("Convert set marks")
+	}
+
+	release := opportunity.Releases[0]
+	if len(opportunity.Releases) != 1 || release.ReleaseID != "100114946" ||
+		len(release.Tags) != 1 || release.Tags[0].Tag != "tender" ||
+		!strings.Contains(release.JSON, `"pid":"109912"`) {
+		t.Errorf("releases = %+v", opportunity.Releases)
+	}
+}

internal/eusupply/html.go

@@ -0,0 +1,139 @@
+package eusupply
+
+import (
+	"slices"
+	"strings"
+
+	"golang.org/x/net/html"
+)
+
+// The helpers below read the parsed pages. The site's HTML is server
+// rendered, so the elements are found by their ids and classes, as the old
+// Node scraper found them with CSS selectors.
+
+// findAll returns the elements under node (node included) that match, in
+// document order.
+func findAll(node *html.Node, match func(*html.Node) bool) []*html.Node {
+	var found []*html.Node
+	var walk func(*html.Node)
+	walk = func(current *html.Node) {
+		if current.Type == html.ElementNode && match(current) {
+			found = append(found, current)
+		}
+		for child := current.FirstChild; child != nil; child = child.NextSibling {
+			walk(child)
+		}
+	}
+	walk(node)
+	return found
+}
+
+// findFirst returns the first element under node that matches, or nil.
+func findFirst(node *html.Node, match func(*html.Node) bool) *html.Node {
+	found := findAll(node, match)
+	if len(found) == 0 {
+		return nil
+	}
+	return found[0]
+}
+
+// isTag matches elements with the tag name.
+func isTag(tag string) func(*html.Node) bool {
+	return func(node *html.Node) bool {
+		return node.Data == tag
+	}
+}
+
+// hasID matches the element with the id.
+func hasID(id string) func(*html.Node) bool {
+	return func(node *html.Node) bool {
+		return attr(node, "id") == id
+	}
+}
+
+// hasClass matches elements with tag (any, if empty) and the class among
+// their classes.
+func hasClass(tag, class string) func(*html.Node) bool {
+	return func(node *html.Node) bool {
+		if tag != "" && node.Data != tag {
+			return false
+		}
+		return slices.Contains(strings.Fields(attr(node, "class")), class)
+	}
+}
+
+// children returns node's element children with the tag name.
+func children(node *html.Node, tag string) []*html.Node {
+	var found []*html.Node
+	for child := node.FirstChild; child != nil; child = child.NextSibling {
+		if child.Type == html.ElementNode && child.Data == tag {
+			found = append(found, child)
+		}
+	}
+	return found
+}
+
+// nextElement returns the element after node among its siblings, or nil.
+func nextElement(node *html.Node) *html.Node {
+	sibling := node.NextSibling
+	for sibling != nil && sibling.Type != html.ElementNode {
+		sibling = sibling.NextSibling
+	}
+	return sibling
+}
+
+// attr returns the value of node's attribute key, or "".
+func attr(node *html.Node, key string) string {
+	for _, attribute := range node.Attr {
+		if attribute.Key == key {
+			return attribute.Val
+		}
+	}
+	return ""
+}
+
+// rawText returns the text under node as written, with a newline for each
+// <br>.
+func rawText(node *html.Node) string {
+	var text strings.Builder
+	var walk func(*html.Node)
+	walk = func(current *html.Node) {
+		switch {
+		case current.Type == html.TextNode:
+			text.WriteString(current.Data)
+		case current.Type == html.ElementNode && current.Data == "br":
+			text.WriteString("\n")
+		}
+		for child := current.FirstChild; child != nil; child = child.NextSibling {
+			walk(child)
+		}
+	}
+	walk(node)
+	return text.String()
+}
+
+// text returns the text under node with its white space collapsed: runs of
+// spaces, non-breaking spaces and line breaks become one space, and the ends
+// are trimmed.
+func text(node *html.Node) string {
+	return collapseSpace(rawText(node))
+}
+
+// lines returns the non-empty lines of the text under node, each with its
+// white space collapsed.
+func lines(node *html.Node) []string {
+	var found []string
+	for line := range strings.SplitSeq(rawText(node), "\n") {
+		if collapsed := collapseSpace(line); collapsed != "" {
+			found = append(found, collapsed)
+		}
+	}
+	return found
+}
+
+// collapseSpace collapses runs of white space (non-breaking spaces and zero
+// width spaces included) into one space, and trims the ends.
+func collapseSpace(value string) string {
+	value = strings.ReplaceAll(value, "\u200b", "")
+	return strings.Join(strings.Fields(value), " ")
+}

internal/eusupply/search.go

@@ -0,0 +1,277 @@
+package eusupply
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"net/url"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+
+	"golang.org/x/net/html"
+)
+
+// The search form's endpoint and paging.
+const (
+	// searchPath is where the portal's public search form posts to.
+	searchPath = "/ctm/Supplier/PublicTenders"
+	// pageSize is the most results per page the form offers.
+	pageSize = 100
+	// defaultSearchDays is how far back a search with no from date goes,
+	// as the old scraper's first search did.
+	defaultSearchDays = 7
+	// dateLayout is the form's and the results' date layout; dateTimeLayout
+	// adds the time of day the results give in their cells' titles.
+	dateLayout     = "02/01/2006"
+	dateTimeLayout = "02/01/2006 15:04:05"
+)
+
+// ErrNoResults means a search page had no results table, which the portal
+// always returns, even for no tenders: the page has changed.
+var ErrNoResults = errors.New("eusupply: no search results table on the page")
+
+// pidPattern finds a tender's portal id (PID) in its link.
+var pidPattern = regexp.MustCompile(`[?&]PID=(\d+)`)
+
+// pagesPattern reads the number of pages from the pager's "of N".
+var pagesPattern = regexp.MustCompile(`^of (\d+)$`)
+
+// Tender is one public tender on the portal. The fields up to Countries are
+// from the search results; the rest are from the tender's own page (see
+// Client.Details), and are empty until it is read.
+type Tender struct {
+	// ID is the quote/tender id the portal shows ("100114946").
+	ID string `json:"id"`
+	// PID is the portal's internal id, which its pages are addressed by.
+	PID       string     `json:"pid"`
+	Reference string     `json:"reference"`
+	Title     string     `json:"title"`
+	URL       string     `json:"url"`
+	Published time.Time  `json:"published"`
+	Deadline  *time.Time `json:"deadline"`
+	Process   string     `json:"process"`
+	Buyer     string     `json:"buyer"`
+	Countries string     `json:"countries"`
+
+	// HasDetails is set once the tender's own page has been read.
+	HasDetails   bool      `json:"has_details"`
+	Description  string    `json:"description,omitempty"`
+	ContractType string    `json:"contract_type,omitempty"`
+	CPVCodes     []CPVCode `json:"cpv_codes,omitempty"`
+	BuyerAddress []string  `json:"buyer_address,omitempty"`
+	Lots         int       `json:"lots,omitempty"`
+}
+
+// CPVCode is one of a tender's CPV codes, as its page lists them: the code
+// with its check digit ("48000000-8") and its description.
+type CPVCode struct {
+	Code        string `json:"code"`
+	Description string `json:"description"`
+}
+
+// Search returns the tenders published from the date of from to the date
+// of until (UK dates, both included), reading every page of results. A
+// zero from searches the last 7 days; a zero until searches up to today. If a page
+// fails, the tenders of the pages before it are returned with the error.
+func (c *Client) Search(
+	ctx context.Context, from, until time.Time,
+) ([]Tender, error) {
+	if until.IsZero() {
+		until = time.Now()
+	}
+	if from.IsZero() {
+		from = until.AddDate(0, 0, -defaultSearchDays)
+	}
+
+	var tenders []Tender
+	for page := 1; ; page++ {
+		body, err := c.post(ctx, searchPath, searchForm(from, until, page))
+		if err != nil {
+			return tenders, fmt.Errorf("searching page %d: %w", page, err)
+		}
+
+		found, pages, err := c.parseSearchPage(body)
+		if err != nil {
+			return tenders, fmt.Errorf("reading search page %d: %w", page, err)
+		}
+		tenders = append(tenders, found...)
+
+		if page >= pages {
+			return tenders, nil
+		}
+	}
+}
+
+// searchForm returns the search form's fields for one page of the tenders
+// published from from to until. They are the fields the portal's own form
+// posts, with its defaults; the portal answers 500 if one is missing or
+// isn't valid.
+func searchForm(from, until time.Time, page int) url.Values {
+	return url.Values{
+		"SearchFilter.SortField":             {"None"},
+		"SearchFilter.SortDirection":         {"None"},
+		"SearchFilter.ShortDescription":      {""},
+		"SearchFilter.Reference":             {""},
+		"SearchFilter.TenderId":              {"0"},
+		"SearchFilter.OperatorId":            {"0"},
+		"Branding":                           {"UK"},
+		"SavedCategoryId":                    {"0"},
+		"SavedUnitAndName":                   {""},
+		"SearchFilter.PublishType":           {"1"}, // issued, not awarded
+		"SearchFilter.BrandingCode":          {""},
+		"TextFilter":                         {""},
+		"SearchFilter.FromDate":              {from.In(ukTime).Format(dateLayout)},
+		"SearchFilter.ToDate":                {until.In(ukTime).Format(dateLayout)},
+		"CpvContainer.CpvCodes":              {""},
+		"CpvContainer.CpvMain":               {""},
+		"CpvContainer.ContractType":          {""},
+		"CpvContainer.CpvIds":                {""},
+		"CpvContainer.IsMandatory":           {"True"},
+		"SearchFilter.ShowExpiredRft":        {"false"},
+		"SearchFilter.PagingInfo.PageNumber": {strconv.Itoa(page)},
+		"SearchFilter.PagingInfo.PageSize":   {strconv.Itoa(pageSize)},
+	}
+}
+
+// parseSearchPage reads the tenders on a page of search results, and the
+// number of pages of results. A row that can't be read is logged and
+// skipped.
+func (c *Client) parseSearchPage(body []byte) ([]Tender, int, error) {
+	document, err := html.Parse(bytes.NewReader(body))
+	if err != nil {
+		return nil, 0, err
+	}
+
+	container := findFirst(document, hasID("searchResultContainer"))
+	if container == nil {
+		return nil, 0, ErrNoResults
+	}
+	table := findFirst(container, isTag("table"))
+	if table == nil {
+		return nil, 0, ErrNoResults
+	}
+
+	var tenders []Tender
+	for _, body := range children(table, "tbody") {
+		for _, row := range children(body, "tr") {
+			tender, err := c.parseResultRow(row)
+			if err != nil {
+				c.logger().Warn(
+					"skipping unreadable search result",
+					"row", text(row),
+					"error", err,
+				)
+				continue
+			}
+			tenders = append(tenders, tender)
+		}
+	}
+
+	return tenders, searchPages(document), nil
+}
+
+// parseResultRow reads one row of search results. Its cells are the id,
+// reference, title (a link to the tender), publication date, response
+// deadline, process, buyers and countries; each date cell's title has the
+// full time.
+func (c *Client) parseResultRow(row *html.Node) (Tender, error) {
+	cells := children(row, "td")
+	if len(cells) < 8 {
+		return Tender{}, fmt.Errorf(
+			"eusupply: a result row has %d cells, not 8", len(cells),
+		)
+	}
+
+	link := findFirst(cells[2], isTag("a"))
+	if link == nil {
+		return Tender{}, errors.New("eusupply: a result row has no link")
+	}
+	match := pidPattern.FindStringSubmatch(attr(link, "href"))
+	if match == nil {
+		return Tender{}, fmt.Errorf(
+			"eusupply: no PID in the link %q", attr(link, "href"),
+		)
+	}
+
+	published, err := cellTime(cells[3])
+	if err != nil {
+		return Tender{}, fmt.Errorf("publication date: %w", err)
+	}
+
+	tender := Tender{
+		ID:        text(cells[0]),
+		PID:       match[1],
+		Reference: text(cells[1]),
+		Title:     text(link),
+		URL:       c.tenderURL(match[1]),
+		Published: published,
+		Process:   text(cells[5]),
+		Buyer:     text(cells[6]),
+		Countries: text(cells[7]),
+	}
+	if tender.ID == "" {
+		return Tender{}, errors.New("eusupply: a result row has no id")
+	}
+
+	// A tender may have no deadline yet (e.g. market engagement).
+	if deadline, err := cellTime(cells[4]); err == nil {
+		tender.Deadline = &deadline
+	}
+
+	return tender, nil
+}
+
+// tenderURL returns the address of the public page of the tender with pid.
+func (c *Client) tenderURL(pid string) string {
+	return c.baseURL + tenderPath(pid)
+}
+
+// tenderPath returns the path of the public page of the tender with pid,
+// which the search results' links redirect to.
+func tenderPath(pid string) string {
+	return "/ctm/Supplier/PublicPurchase/" + url.PathEscape(pid) +
+		"/0/0?returnUrl=ctm/Supplier/PublicTenders&b=UK"
+}
+
+// cellTime reads the UK time in the title of a result cell's tooltip.
+func cellTime(cell *html.Node) (time.Time, error) {
+	tooltip := findFirst(cell, hasClass("div", "js-tooltip-content"))
+	if tooltip == nil {
+		return time.Time{}, errors.New("eusupply: no time in the cell")
+	}
+	return parseUKTime(attr(tooltip, "title"))
+}
+
+// parseUKTime reads a portal time ("21/10/2026 12:45:00"; the seconds may
+// be left out) in UK time.
+func parseUKTime(value string) (time.Time, error) {
+	value = strings.TrimSpace(value)
+	parsed, err := time.ParseInLocation(dateTimeLayout, value, ukTime)
+	if err != nil {
+		parsed, err = time.ParseInLocation("02/01/2006 15:04", value, ukTime)
+	}
+	if err != nil {
+		return time.Time{}, fmt.Errorf("eusupply: time %q: %w", value, err)
+	}
+	return parsed, nil
+}
+
+// searchPages reads the number of pages from the pager ("Page 1 of 3"), or
+// 1 if there is no pager.
+func searchPages(document *html.Node) int {
+	for _, pager := range findAll(document, hasClass("ul", "ctm-pager-info")) {
+		for _, item := range children(pager, "li") {
+			match := pagesPattern.FindStringSubmatch(text(item))
+			if match == nil {
+				continue
+			}
+			if pages, err := strconv.Atoi(match[1]); err == nil && pages > 0 {
+				return pages
+			}
+		}
+	}
+	return 1
+}

internal/eusupply/testdata/search.html

@@ -0,0 +1,45 @@
+<!DOCTYPE html>
+<html>
+<body>
+<form action="/ctm/Supplier/PublicTenders" id="searchForm" method="post">
+<div id="searchResultContainer">
+<table class="table table-striped">
+<thead>
+<tr><th>Quote/tender Id</th><th>Reference</th><th>Name</th><th>Date of publication</th><th>Response deadline (UK time)</th><th>Process</th><th>Buyers</th><th>Countries</th></tr>
+</thead>
+<tbody>
+<tr>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="100114946"> 100114946 </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="Proc2852"> Proc2852 </div> </td>
+<td> <span data="OLD URL: /app/rfq/publicpurchase_frameset.asp?PID=109912&PP=ctm/Supplier/PublicTenders&B=UK,&PS=1&NR=1"></span> <a href="/app/rfq/rwlentrance_s.asp?PID=109912&amp;PP=ctm/Supplier/PublicTenders&amp;B=UK&amp;PS=1" target="_blank">Jisc Google for Education Reseller Framework Agreement</a> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="21/10/2026 12:45:00"> 21/10/2026 </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="04/11/2026 12:00:00"> 04/11/2026 12:00 </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="PA23 - Competitive Flexible "> PA23 - Competitive Flexible </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="Jisc"> Jisc </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="United Kingdom"> United Kingdom </div> </td>
+</tr>
+<tr>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="100115168"> 100115168 </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content"> </div> </td>
+<td> <a href="/app/rfq/rwlentrance_s.asp?PID=110143&amp;PP=ctm/Supplier/PublicTenders&amp;B=UK&amp;PS=1" target="_blank">&#8203;&#8203;York and North Yorkshire Transport Sub-strategies</a> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="20/10/2026 12:34:00"> 20/10/2026 </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content"> </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="PA23 - Preliminary Market Engagement"> PA23 - Preliminary Market Engagement </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="York and North Yorkshire Combined Authority"> York and North Yorkshire Combined Authority </div> </td>
+<td> <div data-toggle="tooltip" class="js-tooltip-content" title="United Kingdom"> United Kingdom </div> </td>
+</tr>
+<tr>
+<td> <div class="js-tooltip-content" title="100115999"> 100115999 </div> </td>
+<td> <div class="js-tooltip-content"> </div> </td>
+<td> A row with no link </td>
+<td> <div class="js-tooltip-content" title="20/10/2026 09:00:00"> 20/10/2026 </div> </td>
+<td></td><td></td><td></td><td></td>
+</tr>
+</tbody>
+</table>
+<ul class="ctm-pager-info"><li><span><label for="SearchFilter_PagingInfo_PageNumber">Page</label></span></li><li><span class="ctm-pager-input"><input id="SearchFilter_PagingInfo_PageNumber" name="SearchFilter.PagingInfo.PageNumber" type="text" value="1" /></span></li><li><span>of 2</span></li></ul>
+<ul class="ctm-pager-info pull-right"><li><span>items per page</span></li></ul>
+</div>
+</form>
+</body>
+</html>

internal/eusupply/testdata/tender.html

@@ -0,0 +1,42 @@
+<!DOCTYPE html>
+<html>
+<body>
+<div id="tenderInfoSection" class="ctm-section-box">
+<div class="row-fluid">
+<div class="span9 ctm-divider-right">
+<div class="ctm-content-label">Detailed description:</div>
+<p class="ctm-multi-line">This opportunity is for a framework agreement.
+The Supplier(s) shall manage the full licence lifecycle (the &#39;Agreement&#39;).</p>
+<div class="ctm-content-label">Type of contract:</div>
+<p>Supplies</p>
+<div class="ctm-content-label">Response deadline (UK time):</div>
+<p>04/11/2026 12:00:00</p>
+<div class="ctm-content-label">CPV codes:</div>
+<p>
+<span class="text-underline">48000000-8&nbsp;&nbsp;Software package and information systems</span><br />
+<span class="">72220000-3&nbsp;&nbsp;Systems and technical consultancy services</span><br />
+</p>
+</div>
+<div class="span3">
+<div class="ctm-content-label">Buyer:</div>
+<p>
+<br />
+Jisc        <br />
+4 Portwall Lane        <br />
+BS1 6NB        <br />
+Bristol        <br />
+United Kingdom
+<br />
+<a href="/ctm/Supplier/CompanyInformation/Index/87175" class="openPopup">
+View profile
+</a>
+</p>
+<div class="ctm-content-label">Contact:</div>
+<p><br /><a href="https://www.jisc.ac.uk" target="_blank">https://www.jisc.ac.uk</a></p>
+</div>
+</div>
+</div>
+<div class="package-container"><div class="row-fluid">Lot 1</div></div>
+<div class="package-container"><div class="row-fluid">Lot 2</div></div>
+</body>
+</html>