bwilliams
3a06132071
internal/eusupply replaces the old Node scraper (eu-supply.js) for uk.eu-supply.com. The site has no API, so it posts the public search form, which now needs the form's current defaults and reads the HTML with golang.org/x/net/html. Each tender not stored yet has its own page read too, for its description, contract type, CPV codes and buyer address. Tenders are stored with the source "EU Supply - UK" as eusupply-uk-<tender id>, with one release holding the tender as read. The site publishes no rate limit, so the client sets its own: one request at a time, with a random wait between sources.eu_supply.request_min_gap and request_max_gap (2s to 10s). The source has the same 7-day backfill as Find a Tender's. Most of its tenders are also on Find a Tender, so a new one is stored Hidden and Skipped when another source's opportunity released within 30 days has a title at least 90% alike, and its buyer or description matches too (80% of the shorter one's significant words).
42 lines
1.4 KiB
AMPL
42 lines
1.4 KiB
AMPL
module git.avsdev.uk/Go/job-search
|
|
|
|
go 1.26.8
|
|
|
|
require (
|
|
github.com/coder/websocket v1.8.15
|
|
github.com/dustin/go-humanize v1.0.1
|
|
github.com/go-chi/chi/v5 v5.4.0
|
|
github.com/jmoiron/sqlx v1.4.0
|
|
github.com/joho/godotenv v1.5.1
|
|
github.com/klauspost/compress v1.20.3
|
|
github.com/lmittmann/tint v1.2.0
|
|
github.com/mattn/go-sqlite3 v1.14.52
|
|
github.com/pires/go-proxyproto v0.15.0
|
|
github.com/pressly/goose/v3 v3.28.0
|
|
github.com/spf13/cobra v1.10.2
|
|
github.com/spf13/viper v1.21.0
|
|
go.yaml.in/yaml/v3 v3.0.5
|
|
golang.org/x/net v0.58.0
|
|
golang.org/x/time v0.16.0
|
|
resty.dev/v3 v3.0.0-rc.4
|
|
)
|
|
|
|
require (
|
|
github.com/fsnotify/fsnotify v1.10.1 // indirect
|
|
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
|
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
|
github.com/mfridman/interpolate v0.0.2 // indirect
|
|
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
|
|
github.com/sagikazarmark/locafero v0.11.0 // indirect
|
|
github.com/sethvargo/go-retry v0.4.0 // indirect
|
|
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
|
|
github.com/spf13/afero v1.15.0 // indirect
|
|
github.com/spf13/cast v1.10.0 // indirect
|
|
github.com/spf13/pflag v1.0.10 // indirect
|
|
github.com/subosito/gotenv v1.6.0 // indirect
|
|
go.uber.org/multierr v1.11.0 // indirect
|
|
golang.org/x/sync v0.22.0 // indirect
|
|
golang.org/x/sys v0.47.0 // indirect
|
|
golang.org/x/text v0.41.0 // indirect
|
|
)
|