job-search/README.md

JobSearch

JobSearch is a service for scraping for contract opportunities. It is primarily a re-write of the existing nodeJS JobSearch codebase.

It currently fetches releases from the UK Government Find a Tender service (FTS), which publishes them in the OCDS format. A mock of the FTS API is included for development.

Building

scripts/build.sh                    # build/job-search-<version>
scripts/build.sh --release          # also package release/job-search-<version>-<arch>.tar.gz

The build script runs go generate ./..., which needs swag installed (see Installation of dev tools), and stamps the version from the latest git tag. A build from untagged commits is marked as a development build and isn't packaged unless --force is also given.

During development, go run -tags sqlite_fts5 ./cmd/jobsearch works as well. The sqlite_fts5 tag is needed for the database's title search (see Testing).

Configuration

Configuration is read from, in increasing priority:

1. Built-in defaults
2. config.yaml in /etc/job-search or the working directory, or the file given with --config
3. JOBSEARCH_* environment variables, e.g. JOBSEARCH_LOGGING_LEVEL=debug

A .env file in the working directory is loaded into the environment first, and must exist.

Example config.yaml:

logging:
    level: info
    file: /var/log/job-search.log
database:
    file: /var/lib/job-search/jobsearch.db
sources:
    find_a_tender:
        enabled: true
        interval: 10m

See internal/config for every key and how the configuration code works.

Usage

Global flags: --config/-c (config file), --debug/-d and --quiet/-q (errors only).

Command	Purpose
job-search config init [path]	Create a config file
job-search config show	Print the configuration (--json or --yaml for those formats)
job-search config get <key>	Print one value, e.g. logging.level
job-search config set <key> <value>	Validate and write one value to the config file
job-search config set --unset <key>	Remove a value from the config file
job-search config generate-csrf-key	Store a new random server.csrf.tokens.key in the config file, so CSRF tokens survive restarts and work across instances (--print only prints one; --force replaces an existing key)
job-search migrate	Apply every pending database migration (same as migrate up)
job-search migrate up	Apply pending migrations (--steps N for only N; --create to create a missing database file without asking)
job-search migrate down --steps N	Roll back the last N migrations (--steps 0 rolls back all of them, deleting all data, and needs --yes)
job-search migrate status	Show the database version and which migrations are applied or pending
job-search version	Print version information

job-search serve describes its REST API in OpenAPI 3.1 at /api/openapi.json, and shows it in Swagger UI at /api/docs. The description is generated from the handlers' annotations in internal/api by go generate.

The migrate commands work on the database in database.file. The store refuses to open a database that isn't at this build's schema version, so run job-search migrate after installing a new version.

Fetch tool

fetcher reads the same configuration as job-search:

Command	Purpose
fetcher service	Poll every enabled source on its interval until stopped (Ctrl+C or SIGTERM)
fetcher fts-once	Fetch from Find a Tender once and print the releases as JSON

service logs to logging.file if it's set, or to the terminal with --foreground. Both commands take --since "YYYY-MM-DD[ hh:mm:ss]". Without it, service (and job-search serve) starts each source from the newest release stored for it, so a restart carries on where it left off; with nothing stored, Find a Tender's client fetches the last day. --since overrides that for every source. fts-once can be limited with --planning, --opportunities or --awards, and with --ignore-rate-limit it fails on a rate limit instead of waiting.

During development, go run ./cmd/fetcher works as well.

Mock Find a Tender server

To develop without using the real API, run the mock and enable the fts_mock source:

go run ./cmd/mock-fts --min-gap 5s --max-gap 10s --rate-limit 1000

sources:
    fts_mock:
        enabled: true
        interval: 1m
        api_root: http://localhost:8080/api/1.0

See internal/mock for its options and behaviour.

WebSocket test tool

ws-test reads the same configuration as job-search for its defaults:

go run ./cmd/ws-test dial --http-upgrade --ping  # ping job-search serve's /ws endpoint
go run ./cmd/ws-test dial --direct --ping        # ping ws-test serve
go run ./cmd/ws-test dial --http-upgrade         # send each line of stdin, print replies
go run ./cmd/ws-test serve                       # plain TCP server for websocket hub clients

dial needs one of --http-upgrade, to upgrade an HTTP connection to a WebSocket on /ws as job-search serve expects, or --direct, to connect straight to ws-test serve over plain TCP with one JSON message per line. It also takes --message, --listen and --timeout; --url and --origin (to test the Origin check) with --http-upgrade, and --addr with --direct. See ws-test dial --help.

ws-test serve is a test server only: don't run it alongside job-search serve. Both listen on server.host and server.port, so the second one fails with "address already in use".

Project layout

Path	Contents
cmd/jobsearch	The job-search command
cmd/fetcher	The fetcher command (Find a Tender fetch, the polling service)
cmd/mock-fts	The mock FTS server
cmd/ws-test	The ws-test WebSocket test tool
internal/cli	Cobra commands and shared CLI setup
internal/config	Configuration loading, validation and editing
internal/fetcher	Runs a worker per enabled source
internal/store	SQLite storage of opportunities, and its migrations (see internal/store)
internal/findatender	FTS API client and conversion from OCDS to JobSearch types
internal/types	JobSearch's own opportunity types
internal/util	The periodic Worker used by the fetcher
internal/logging	slog setup and the HTTP access log
internal/mock	Mock external services
internal/version	Version information set at build time

Testing

The store's title search needs SQLite's FTS5 module, which is only compiled in with the sqlite_fts5 build tag, so tests (and lint) need it. Without it the store tests fail with no such module: fts5.

go test -race -tags sqlite_fts5 ./internal/...
go test -tags sqlite_fts5 -coverprofile=cover.out ./internal/... && go tool cover -func=cover.out
golangci-lint run --build-tags sqlite_fts5 ./...

Tests don't need network access; HTTP calls go to local httptest servers.

Addition of a new opportunities source

Using a source called example as an illustration:

1. Configuration section: create internal/config/conf_source_example.go, copying an existing source such as conf_source_ftsmock.go:
   • Add a JSCKSourceExample = "example" constant.
   • Define JSCSourceExample, embedding JSCSourcesSource (for enabled and interval), plus any fields of its own with cfg and optional default tags, e.g. APIRoot string `cfg:"api_root"` .
   • Implement the Section interface (see internal/config/section.go): copy the Map, Flatten, TypeOf, ValueOf and SetValue methods from an existing source, under a // ---------- Section methods (see Section) ---------- banner, renaming the receiver type and using the new constant in Flatten.
2. Register the section: add Example JSCSourceExample `cfg:"example"` to JSCSources in internal/config/conf_sources.go. The cfg tag must match the constant. The source's keys (sources.example.*) are then loaded, validated and available to the config commands with no further code.
3. Client package: create a package that fetches from the source and returns []*types.Opportunity (see internal/types). If the source publishes OCDS, internal/findatender may already be suitable, as it is for fts_mock: create it with the source's own API root.
4. Worker: create internal/fetcher/worker_example.go, following worker_ftsmock.go:
   • Add an exampleSource = "Example" constant: the worker's name, and the Source its opportunities are stored with (set it on each one if the client doesn't, as fetchFTSMock does).
   • runExample(ctx, logger, conf, save, since) builds a util.Worker with exampleSource, conf.Interval and a task, then calls Run.
   • The task fetches everything since its since argument, then returns save(ctx, logger, opportunities), which stores them when the fetcher has a store. Wrap blocking calls in util.AwaitTask so shutdown isn't held up, and return ErrNoNewReleases when nothing is found, so the checkpoint still moves forward. Any other error makes the next run retry from the same point.
5. Start it: in Run in internal/fetcher/fetcher.go, add

   if f.conf.Example.Enabled {
       start(exampleSource, func(ctx context.Context, since time.Time) error {
           return runExample(ctx, logger, &f.conf.Example, f.save, since)
       })
   }
   

   start passes the worker its starting point: --since, or the newest release stored with exampleSource.
6. Tests: TestSections checks the new section implements Section and that its keys match its cfg tag, with no changes needed. In internal/config/conf_sections_test.go, add the section to TestConcreteSourceSections and update the key count in TestSourcesSection. Test the client and worker using httptest servers rather than the real service.
7. Docs: add the source and its keys to the key table in internal/config/README.md.

Installation of dev tools

A number of tools have been used in the development of the project. Installation of these tools can be done using go install.

To install the tools:

go install github.com/spf13/cobra-cli@latest
go install git.avsdev.uk/Go/go-ocds/cmd/ocdsgen@latest
go install github.com/pressly/goose/v3/cmd/goose@latest
go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest
go install github.com/swaggo/swag/v2/cmd/swag@v2.0.0-rc6

goose creates new migration files (see internal/store); the migrations themselves are applied by job-search migrate. golangci-lint uses the project's .golangci.json. swag generates the API's OpenAPI 3.1 spec from the handlers' comments (swag init --v3.1). Its v2, the version with OpenAPI 3 support, has only had release candidates so far, so it is pinned to one rather than @latest.

Ensure that the GOBIN path ($HOME/go/bin) is added to the environment before running any of the tools.