JobSearch

JobSearch is a service for scraping for contract opportunities. It is primarily a re-write of the existing nodeJS JobSearch codebase.

It currently fetches releases from the UK Government Find a Tender service (FTS), which publishes them in the OCDS format. A mock of the FTS API is included for development.

Building

scripts/build.sh                    # build/job-search-<version>
scripts/build.sh --release          # also package release/job-search-<version>-<arch>.tar.gz

The build script builds the frontend and copies it into place for embedding (npm run build and npm run sync in web/, which need Node and npm; it runs npm ci first for a release, or when web/node_modules is missing). It then runs go generate ./..., which needs ocdsgen and swag installed (scripts/install-dev-tools.sh; see Installation of dev tools), and stamps the version from the latest git tag. A build from untagged commits is marked as a development build and isn't packaged unless --force is also given.

The frontend (web/) is embedded in the binary. Without the build script, build it and copy it into place first, or the binary serves no frontend (job-search serve warns):

cd web && npm ci && npm run build && npm run sync   # web/dist -> internal/frontend/dist

Setting server.frontend.dir serves a build from disk instead.

During development, go run -tags sqlite_fts5 ./cmd/jobsearch works as well. The sqlite_fts5 tag is needed for the database's title search (see Testing).

Running as a service

deploy/job-search.service is a systemd unit running job-search serve (the server and every enabled source's fetcher) as a job-search system user, from /usr/local/bin/job-search with /etc/job-search/config.yaml. Its comments give the install steps. It keeps its data in /var/lib/job-search and any log file in /var/log/job-search, the only places it can write, so set database.file there. Logs go to the journal (journalctl -u job-search) unless logging.file is set to a file in /var/log/job-search, which then needs rotating (below).

• The database isn't migrated on start: run job-search migrate up (--create the first time) as the job-search user before the first start and after each upgrade.
• A failing source stops serve on purpose; the unit restarts it after 30s.

Rotating the logs

job-search opens its log file once and has no signal to reopen it, so logrotate must copy the file and truncate it in place (copytruncate); a line written between the two is lost. deploy/job-search.logrotate does this weekly for every *.log in /var/log/job-search (the log and, if server.access_log is set there, the access log), keeping 8 weeks, compressed. /var/log/job-search belongs to the service user, so it rotates as that user (su job-search job-search); logrotate refuses otherwise.

# from the release tarball's directory (deploy/job-search.logrotate in the repo)
sudo install -m 644 job-search.logrotate /etc/logrotate.d/job-search
sudo logrotate --debug /etc/logrotate.d/job-search   # dry run: what it would do
sudo logrotate --force /etc/logrotate.d/job-search   # rotate now, to check

Configuration

Configuration is read from, in increasing priority:

1. Built-in defaults
2. config.yaml in /etc/job-search or the working directory, or the file given with --config
3. JOBSEARCH_* environment variables, e.g. JOBSEARCH_LOGGING_LEVEL=debug

A .env file in the working directory, if there is one, is loaded into the environment first.

Example config.yaml:

logging:
    level: info
    file: /var/log/job-search.log
database:
    file: /var/lib/job-search/jobsearch.db
sources:
    find_a_tender:
        enabled: true
        interval: 10m

See internal/config for every key and how the configuration code works.

Usage

Global flags: --config/-c (config file), --debug/-d and --quiet/-q (errors only).

Command	Purpose
job-search config init [path]	Create a config file holding the defaults (--all also writes the test sources, fts_mock and sleeper)
job-search config show	Print the configuration (--json or --yaml for those formats)
job-search config get <key>	Print one value, e.g. logging.level
job-search config set <key> <value>	Validate and write one value to the config file
job-search config set --unset <key>	Remove a value from the config file
job-search config generate-csrf-key	Store a new random server.csrf.tokens.key in the config file, so CSRF tokens survive restarts and work across instances (--print only prints one; --force replaces an existing key)
job-search migrate	Apply every pending database migration (same as migrate up)
job-search migrate up	Apply pending migrations (--steps N for only N; --create to create a missing database file without asking)
job-search migrate down --steps N	Roll back the last N migrations (--steps 0 rolls back all of them, deleting all data, and needs --yes)
job-search migrate status	Show the database version and which migrations are applied or pending
job-search skip-rules check [file]	Check a skip rules file (default: sources.skip_rules_file)
job-search skip-rules apply [file]	Mark the stored opportunities that match the skip rules Skipped, listing them (--dry-run only lists them)
job-search version	Print version information

job-search serve describes its REST API in OpenAPI 3.1 at /api/openapi.json, and shows it in Swagger UI at /api/docs. The description is generated from the handlers' annotations in internal/api by go generate.

The migrate commands work on the database in database.file. The store refuses to open a database that isn't at this build's schema version, so run job-search migrate after installing a new version.

Skip rules

sources.skip_rules_file names a JSON file of rules that mark new opportunities Skipped as fetcher service and job-search serve fetch them, by buyer, title words or CPV codes. The file is read again when it changes; an invalid edit is logged and the last good rules kept. Opportunities already stored are only changed by job-search skip-rules apply. See internal/skiprules for the format.

Fetch tool

fetcher reads the same configuration as job-search:

Command	Purpose
fetcher service	Poll every enabled source on its interval until stopped (Ctrl+C or SIGTERM)
fetcher fts-once	Fetch from Find a Tender once and print the releases as JSON

service logs to logging.file if it's set, or to the terminal with --foreground. Both commands take --since "YYYY-MM-DD[ hh:mm:ss]". Without it, service (and job-search serve) starts each source from the newest release stored for it, so a restart carries on where it left off; with nothing stored, Find a Tender's client fetches the last day. --since overrides that for every source.

When the start is more than 7 days ago (an early --since, or a long time offline), the Find a Tender and EU Supply sources first backfill it in 7-day windows, oldest first, saving each before fetching the next. A window that fails is saved as far as it got and fetched again after the source's interval. The API returns the newest releases first, so if a backfill is stopped partway through a window, restarting without --since skips the rest of that window: restart with --since set to the window's start (logged as backfilling) to fill it.

A rate limit is waited out, as the API asks, up to sources.<source>.rate_limit_retries times per request. A failed fetch keeps what it fetched before the failure.

EU Supply

The eu_supply source scrapes the public tenders on EU Supply's UK portal (uk.eu-supply.com), stored with the source EU Supply - UK. The site has no API and publishes no rate limit, so the client waits a random time from request_min_gap to request_max_gap (2s to 10s) between its requests. Each tender not stored yet has its own page read too, for its description and CPV codes. Most of its tenders are also published on Find a Tender: a new one is stored Hidden and Skipped when another source's opportunity released within 30 days of it has a title at least 90% alike, and its buyer or its description matches too (80% of the shorter one's words). See internal/eusupply.

fts-once can be limited with --planning, --opportunities or --awards, and with --ignore-rate-limit it fails on a rate limit instead of waiting.

During development, go run ./cmd/fetcher works as well.

Mock Find a Tender server

To develop without using the real API, run the mock and enable the fts_mock source:

go run ./cmd/mock-fts --min-gap 5s --max-gap 10s --rate-limit 1000

sources:
    fts_mock:
        enabled: true
        interval: 1m
        api_root: http://localhost:8080/api/1.0

See internal/mock for its options and behaviour.

WebSocket test tool

ws-test reads the same configuration as job-search for its defaults:

go run ./cmd/ws-test dial --http-upgrade --ping  # ping job-search serve's /ws endpoint
go run ./cmd/ws-test dial --direct --ping        # ping ws-test serve
go run ./cmd/ws-test dial --http-upgrade         # send each line of stdin, print replies
go run ./cmd/ws-test serve                       # plain TCP server for websocket hub clients

dial needs one of --http-upgrade, to upgrade an HTTP connection to a WebSocket on /ws as job-search serve expects, or --direct, to connect straight to ws-test serve over plain TCP with one JSON message per line. It also takes --message, --listen and --timeout; --url and --origin (to test the Origin check) with --http-upgrade, and --addr with --direct. See ws-test dial --help.

ws-test serve is a test server only: don't run it alongside job-search serve. Both listen on server.host and server.port, so the second one fails with "address already in use".

Project layout

Path	Contents
cmd/jobsearch	The job-search command
cmd/fetcher	The fetcher command (Find a Tender fetch, the polling service)
cmd/mock-fts	The mock FTS server
cmd/ws-test	The ws-test WebSocket test tool
internal/cli	Cobra commands and shared CLI setup
internal/config	Configuration loading, validation and editing
internal/fetcher	Runs a worker per enabled source
internal/search	Parsing of the general opportunity search (title:, buyer:, buyer_id:, cpv:, !) and its help
internal/skiprules	The rules that mark new opportunities Skipped (see internal/skiprules)
internal/store	SQLite storage of opportunities, and its migrations (see internal/store)
internal/findatender	FTS API client and conversion from OCDS to JobSearch types
internal/eusupply	EU Supply UK portal scraper and conversion to JobSearch types (see internal/eusupply)
internal/types	JobSearch's own opportunity types
internal/util	The periodic Worker used by the fetcher
internal/logging	slog setup and the HTTP access log
internal/mock	Mock external services
internal/version	Version information set at build time
internal/frontend	Serves the frontend's build output; embeds dist/, copied there from web/dist by npm run sync
web	The frontend (see web)
deploy	The systemd unit and logrotate configuration (see Running as a service)

Testing

The store's title search needs SQLite's FTS5 module, which is only compiled in with the sqlite_fts5 build tag, so tests (and lint) need it. Without it the store tests fail with no such module: fts5.

go test -race -tags sqlite_fts5 ./internal/...
go test -tags sqlite_fts5 -coverprofile=cover.out ./internal/... && go tool cover -func=cover.out
golangci-lint run --build-tags sqlite_fts5 ./...

Tests don't need network access; HTTP calls go to local httptest servers.

Addition of a new opportunities source

Using a source called example as an illustration:

1. Configuration section: create internal/config/conf_source_example.go, copying an existing source such as conf_source_ftsmock.go:
   • Add a JSCKSourceExample = "example" constant.
   • Define JSCSourceExample, embedding JSCSourcesSource (for enabled and interval), plus any fields of its own with cfg and optional default tags, e.g. APIRoot string `cfg:"api_root"` .
   • Implement the Section interface (see internal/config/section.go): copy the Map, Flatten, TypeOf, ValueOf and SetValue methods from an existing source, under a // ---------- Section methods (see Section) ---------- banner, renaming the receiver type and using the new constant in Flatten.
2. Register the section: add Example JSCSourceExample `cfg:"example"` to JSCSources in internal/config/conf_sources.go. The cfg tag must match the constant. The source's keys (sources.example.*) are then loaded, validated and available to the config commands with no further code.
3. Client package: create a package that fetches from the source and returns []*types.Opportunity (see internal/types). If the source publishes OCDS, internal/findatender may already be suitable, as it is for fts_mock: create it with the source's own API root.
4. Worker: create internal/fetcher/worker_example.go, following worker_ftsmock.go:
   • Add an exampleSource = "Example" constant: the worker's name, and the Source its opportunities are stored with (set it on each one if the client doesn't, as fetchFTSMock does).
   • runExample(ctx, logger, conf, save, since) builds a util.Worker with exampleSource, conf.Interval and a task, then calls Run.
   • The task fetches everything since its since argument, then returns save(ctx, logger, opportunities), which stores them when the fetcher has a store. Wrap blocking calls in util.AwaitTask so shutdown isn't held up, and return ErrNoNewReleases when nothing is found, so the checkpoint still moves forward. Any other error makes the next run retry from the same point.
5. Start it: in Run in internal/fetcher/fetcher.go, add

   if f.conf.Example.Enabled {
       start(exampleSource, func(ctx context.Context, since time.Time) error {
           return runExample(ctx, logger, &f.conf.Example, f.save, since)
       })
   }
   

   start passes the worker its starting point: --since, or the newest release stored with exampleSource.
6. Tests: TestSections checks the new section implements Section and that its keys match its cfg tag, with no changes needed. In internal/config/conf_sections_test.go, add the section to TestConcreteSourceSections and update the key count in TestSourcesSection. Test the client and worker using httptest servers rather than the real service.
7. Docs: add the source and its keys to the key table in internal/config/README.md.

Installation of dev tools

scripts/install-dev-tools.sh installs everything needed to develop and build the project: the Go tools below, with go install, and the frontend's npm packages (npm ci in web/). It needs Go and Node.js 24 or later (with npm), as the frontend's scripts run TypeScript directly. Run it again after a tool's version in it changes.

scripts/install-dev-tools.sh

Tool	Version	Used for
ocdsgen (git.avsdev.uk/Go/go-ocds)	v1.0.0	Generates the OCDS types in internal/findatender/ocds (ocds_gen.go) from extensions.json, run by go generate
swag	v2.0.0-rc6	Generates the API's OpenAPI 3.1 description from the handlers' comments (swag init --v3.1), run by go generate
golangci-lint	latest	Lints the Go code with the project's .golangci.json
goose	latest	Creates new migration files (see internal/store); the migrations themselves are applied by job-search migrate
cobra-cli	latest	Adds cobra commands

The generators are pinned, as another version could generate different code. swag's v2, the version with OpenAPI 3 support, has only had release candidates so far. go generate ./... and the build script run the generators from PATH, so ensure that the GOBIN path ($HOME/go/bin by default) is added to the environment before running any of the tools; the script says if it isn't.