Added an EU Supply source, scraping the UK portal's public tenders
internal/eusupply replaces the old Node scraper (eu-supply.js) for uk.eu-supply.com. The site has no API, so it posts the public search form, which now needs the form's current defaults and reads the HTML with golang.org/x/net/html. Each tender not stored yet has its own page read too, for its description, contract type, CPV codes and buyer address. Tenders are stored with the source "EU Supply - UK" as eusupply-uk-<tender id>, with one release holding the tender as read. The site publishes no rate limit, so the client sets its own: one request at a time, with a random wait between sources.eu_supply.request_min_gap and request_max_gap (2s to 10s). The source has the same 7-day backfill as Find a Tender's. Most of its tenders are also on Find a Tender, so a new one is stored Hidden and Skipped when another source's opportunity released within 30 days has a title at least 90% alike, and its buyer or description matches too (80% of the shorter one's significant words).
This commit is contained in:
1 parent
5c62376b6e
commit
3a06132071
25 files changed
+2520
-8
No files matched your search
@@ -120,7 +120,8 @@ nothing stored, Find a Tender's client fetches the last day. `--since`
|
||||
overrides that for every source.
|
||||
|
||||
When the start is more than 7 days ago (an early `--since`, or a long time
|
||||
offline), the Find a Tender sources first backfill it in 7-day windows,
|
||||
offline), the Find a Tender and EU Supply sources first backfill it in 7-day
|
||||
windows,
|
||||
oldest first, saving each before fetching the next. A window that fails is
|
||||
saved as far as it got and fetched again after the source's interval. The
|
||||
API returns the newest releases first, so if a backfill is stopped partway
|
||||
@@ -132,6 +133,18 @@ A rate limit is waited out, as the API asks, up to
|
||||
`sources.<source>.rate_limit_retries` times per request. A failed fetch
|
||||
keeps what it fetched before the failure.
|
||||
|
||||
### EU Supply
|
||||
The `eu_supply` source scrapes the public tenders on EU Supply's UK portal
|
||||
(`uk.eu-supply.com`), stored with the source `EU Supply - UK`. The site has
|
||||
no API and publishes no rate limit, so the client waits a random time from
|
||||
`request_min_gap` to `request_max_gap` (2s to 10s) between its requests. Each
|
||||
tender not stored yet has its own page read too, for its description and
|
||||
CPV codes. Most of its tenders are also published on Find a Tender: a new
|
||||
one is stored Hidden and Skipped when another source's opportunity released
|
||||
within 30 days of it has a title at least 90% alike, and its buyer or its
|
||||
description matches too (80% of the shorter one's words). See
|
||||
[internal/eusupply](internal/eusupply/README.md).
|
||||
|
||||
`fts-once` can be limited with `--planning`, `--opportunities` or
|
||||
`--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
|
||||
of waiting.
|
||||
@@ -191,6 +204,7 @@ second one fails with "address already in use".
|
||||
| `internal/skiprules` | The rules that mark new opportunities Skipped (see [internal/skiprules](internal/skiprules/README.md)) |
|
||||
| `internal/store` | SQLite storage of opportunities, and its migrations (see [internal/store](internal/store/README.md)) |
|
||||
| `internal/findatender` | FTS API client and conversion from OCDS to JobSearch types |
|
||||
| `internal/eusupply` | EU Supply UK portal scraper and conversion to JobSearch types (see [internal/eusupply](internal/eusupply/README.md)) |
|
||||
| `internal/types` | JobSearch's own opportunity types |
|
||||
| `internal/util` | The periodic `Worker` used by the fetcher |
|
||||
| `internal/logging` | slog setup and the HTTP access log |
|
||||
|
||||
Reference in new issue
Block a user