JobSearch

JobSearch is a service for scraping for contract opportunities. It is primarily a re-write of the existing nodeJS JobSearch codebase.

It currently fetches releases from the UK Government Find a Tender service (FTS), which publishes them in the OCDS format. A mock of the FTS API is included for development.

Building

scripts/build.sh                    # build/job-search-<version>
scripts/build.sh --release          # also package release/job-search-<version>-<arch>.tar.gz

The build script runs go generate ./... and stamps the version from the latest git tag. A build from untagged commits is marked as a development build and isn't packaged unless --force is also given.

During development, go run ./cmd/jobsearch works as well.

Configuration

Configuration is read from, in increasing priority:

1. Built-in defaults
2. config.yaml in /etc/job-search or the working directory, or the file given with --config
3. JOBSEARCH_* environment variables, e.g. JOBSEARCH_LOGGING_LEVEL=debug

A .env file in the working directory is loaded into the environment first, and must exist.

Example config.yaml:

logging:
    level: info
    file: /var/log/job-search.log
sources:
    find_a_tender:
        enabled: true
        interval: 10m

See internal/config for every key and how the configuration code works.

Usage

Global flags: --config/-c (config file), --debug/-d and --quiet/-q (errors only).

Command	Purpose
job-search fetch service	Poll every enabled source on its interval until stopped (Ctrl+C or SIGTERM)
job-search fetch once	Fetch from Find a Tender once and print the releases as JSON
job-search config init [path]	Create a config file
job-search config show	Print the configuration (--json or --yaml for those formats)
job-search config get <key>	Print one value, e.g. logging.level
job-search config set <key> <value>	Validate and write one value to the config file
job-search config set --unset <key>	Remove a value from the config file
job-search version	Print version information

fetch service logs to logging.file if it's set, or to the terminal with --foreground. Both fetch commands take --since "YYYY-MM-DD[ hh:mm:ss]". fetch once can be limited with --planning, --opportunities or --awards, and with --ignore-rate-limit it fails on a rate limit instead of waiting.

Mock Find a Tender server

To develop without using the real API, run the mock and enable the fts_mock source:

go run ./cmd/mock-fts --min-gap 5s --max-gap 10s --rate-limit 1000

sources:
    fts_mock:
        enabled: true
        interval: 1m
        api_root: http://localhost:8080/api/1.0

See internal/mock for its options and behaviour.

Project layout

Path	Contents
cmd/jobsearch	The job-search command
cmd/mock-fts	The mock FTS server
internal/cli	Cobra commands and shared CLI setup
internal/config	Configuration loading, validation and editing
internal/fetcher	Runs a worker per enabled source
internal/findATender	FTS API client and conversion from OCDS to JobSearch types
internal/types	JobSearch's own opportunity types
internal/util	The periodic Worker used by the fetcher
internal/logging	slog setup and the HTTP access log
internal/mock	Mock external services
internal/version	Version information set at build time

Testing

go test -race ./internal/...
go test -coverprofile=cover.out ./internal/... && go tool cover -func=cover.out

Tests don't need network access; HTTP calls go to local httptest servers.

Addition of a new opportunities source

Using a source called example as an illustration:

1. Configuration section: create internal/config/conf_source_example.go, copying an existing source such as conf_source_ftsMock.go:
   • Add a JSCKSourceExample = "example" constant.
   • Define JobSearchConfigSourceExample, embedding JobSearchConfigSource (for enabled and interval), plus any fields of its own with cfg and optional default tags, e.g. APIRoot string `cfg:"api_root"` .
   • Copy the standard Map, Flatten, TypeOf, ValueOf and SetValue methods, renaming the receiver type and using the new constant in Flatten.
2. Register the section: add Example JobSearchConfigSourceExample `cfg:"example"` to JobSearchConfigSources in internal/config/conf_sources.go. The cfg tag must match the constant. The source's keys (sources.example.*) are then loaded, validated and available to the config commands with no further code.
3. Client package: create a package that fetches from the source and returns []*types.Opportunity (see internal/types). If the source publishes OCDS, internal/findATender may already be suitable, as it is for fts_mock: create it with the source's own API root.
4. Worker: create internal/fetcher/worker_example.go, following worker_ftsMock.go:
   • runExample(ctx, conf, since) builds a util.Worker with the source's name, conf.Sources.Example.Interval and a task, then calls Run.
   • The task fetches everything since its since argument. Wrap blocking calls in util.AwaitTask so shutdown isn't held up, and return ErrNoNewReleases when nothing is found, so the checkpoint still moves forward. Any other error makes the next run retry from the same point.
5. Start it: in Run in internal/fetcher/fetcher.go, add

   if conf.Sources.Example.Enabled {
       start("Example", runExample)
   }
   

6. Tests: add tests for the new config section to internal/config/conf_sections_test.go (add it to TestConcreteSourceSections and update the key count in TestSourcesSection), and for the client and worker, using httptest servers rather than the real service.
7. Docs: add the source and its keys to the key table in internal/config/README.md.

Installation of dev tools

A number of tools have been used in the development of the project. Installation of these tools can be done using go install.

To install the tools:

go install github.com/spf13/cobra-cli@latest
go install git.avsdev.uk/Go/go-ocds/cmd/ocdsgen@latest

Ensure that the GOBIN path ($HOME/go/bin) is added to the environment before running any of the tools.