job-search/README.md

# JobSearch

JobSearch is a service for scraping for contract opportunities. It is primarily
a re-write of the existing nodeJS JobSearch codebase.

It currently fetches releases from the UK Government
[Find a Tender](https://www.find-tender.service.gov.uk/) service (FTS), which
publishes them in the [OCDS](https://standard.open-contracting.org/) format.
A mock of the FTS API is included for development.


## Building
```bash
scripts/build.sh                    # build/job-search-<version>
scripts/build.sh --release          # also package release/job-search-<version>-<arch>.tar.gz
```

The build script builds the frontend and copies it into place for embedding
(`npm run build` and `npm run sync` in `web/`, which need Node and npm; it
runs `npm ci` first for a release, or when `web/node_modules` is missing). It
then runs `go generate ./...`, which needs `ocdsgen` and `swag` installed
(`scripts/install-dev-tools.sh`; see
[Installation of dev tools](#installation-of-dev-tools)), and stamps the
version from the latest git tag. A build from untagged commits is marked as a development build
and isn't packaged unless `--force` is also given.

The frontend (`web/`) is embedded in the binary. Without the build script,
build it and copy it into place first, or the binary serves no frontend
(`job-search serve` warns):

```bash
cd web && npm ci && npm run build && npm run sync   # web/dist -> internal/frontend/dist
```

Setting `server.frontend.dir` serves a build from disk instead.

During development, `go run -tags sqlite_fts5 ./cmd/jobsearch` works as well.
The `sqlite_fts5` tag is needed for the database's title search (see
[Testing](#testing)).


## Running as a service
`deploy/job-search.service` is a systemd unit running `job-search serve`
(the server and every enabled source's fetcher) as a `job-search` system
user, from `/usr/local/bin/job-search` with `/etc/job-search/config.yaml`.
Its comments give the install steps. It keeps its data in
`/var/lib/job-search` and any log file in `/var/log/job-search`, the only
places it can write, so set `database.file` there. Logs go to the journal
(`journalctl -u job-search`) unless `logging.file` is set to a file in
`/var/log/job-search`, which then needs rotating (below).

- The database isn't migrated on start: run `job-search migrate up`
  (`--create` the first time) as the `job-search` user before the first
  start and after each upgrade.
- A failing source stops `serve` on purpose; the unit restarts it after
  30s.

### Rotating the logs
`job-search` opens its log file once and has no signal to reopen it, so
logrotate must copy the file and truncate it in place (`copytruncate`); a
line written between the two is lost. `deploy/job-search.logrotate` does
this weekly for every `*.log` in `/var/log/job-search` (the log and, if
`server.access_log` is set there, the access log), keeping 8 weeks,
compressed. `/var/log/job-search` belongs to the service user, so it rotates
as that user (`su job-search job-search`); logrotate refuses otherwise.

```bash
# from the release tarball's directory (deploy/job-search.logrotate in the repo)
sudo install -m 644 job-search.logrotate /etc/logrotate.d/job-search
sudo logrotate --debug /etc/logrotate.d/job-search   # dry run: what it would do
sudo logrotate --force /etc/logrotate.d/job-search   # rotate now, to check
```


## Configuration
Configuration is read from, in increasing priority:
1. Built-in defaults
2. `config.yaml` in `/etc/job-search` or the working directory, or the file
   given with `--config`
3. `JOBSEARCH_*` environment variables, e.g. `JOBSEARCH_LOGGING_LEVEL=debug`

A `.env` file in the working directory, if there is one, is loaded into the
environment first.

Example `config.yaml`:
```yaml
logging:
    level: info
    file: /var/log/job-search.log
database:
    file: /var/lib/job-search/jobsearch.db
sources:
    find_a_tender:
        enabled: true
        interval: 10m
```

See [internal/config](internal/config/README.md) for every key and how the
configuration code works.


## Usage
Global flags: `--config`/`-c` (config file), `--debug`/`-d` and
`--quiet`/`-q` (errors only).

| Command | Purpose |
|---|---|
| `job-search config init [path]` | Create a config file holding the defaults (`--all` also writes the test sources, `fts_mock` and `sleeper`) |
| `job-search config show` | Print the configuration (`--json` or `--yaml` for those formats) |
| `job-search config get <key>` | Print one value, e.g. `logging.level` |
| `job-search config set <key> <value>` | Validate and write one value to the config file |
| `job-search config set --unset <key>` | Remove a value from the config file |
| `job-search config generate-csrf-key` | Store a new random `server.csrf.tokens.key` in the config file, so CSRF tokens survive restarts and work across instances (`--print` only prints one; `--force` replaces an existing key) |
| `job-search migrate` | Apply every pending database migration (same as `migrate up`) |
| `job-search migrate up` | Apply pending migrations (`--steps N` for only N; `--create` to create a missing database file without asking) |
| `job-search migrate down --steps N` | Roll back the last N migrations (`--steps 0` rolls back all of them, deleting all data, and needs `--yes`) |
| `job-search migrate status` | Show the database version and which migrations are applied or pending |
| `job-search skip-rules check [file]` | Check a skip rules file (default: `sources.skip_rules_file`) |
| `job-search skip-rules apply [file]` | Mark the stored opportunities that match the skip rules Skipped, listing them (`--dry-run` only lists them) |
| `job-search version` | Print version information |

`job-search serve` describes its REST API in OpenAPI 3.1 at
`/api/openapi.json`, and shows it in Swagger UI at `/api/docs`. The
description is generated from the handlers' annotations in `internal/api` by
`go generate`.

The `migrate` commands work on the database in `database.file`. The store
refuses to open a database that isn't at this build's schema version, so run
`job-search migrate` after installing a new version.

### Skip rules
`sources.skip_rules_file` names a JSON file of rules that mark new
opportunities Skipped as `fetcher service` and `job-search serve` fetch them,
by buyer, title words or CPV codes. The file is read again when it changes;
an invalid edit is logged and the last good rules kept. Opportunities already
stored are only changed by `job-search skip-rules apply`. See
[internal/skiprules](internal/skiprules/README.md) for the format.


## Fetch tool
`fetcher` reads the same configuration as `job-search`:

| Command | Purpose |
|---|---|
| `fetcher service` | Poll every enabled source on its interval until stopped (Ctrl+C or SIGTERM) |
| `fetcher fts-once` | Fetch from Find a Tender once and print the releases as JSON |

`service` logs to `logging.file` if it's set, or to the terminal with
`--foreground`. Both commands take `--since "YYYY-MM-DD[ hh:mm:ss]"`.
Without it, `service` (and `job-search serve`) starts each source from the
newest release stored for it, so a restart carries on where it left off; with
nothing stored, Find a Tender's client fetches the last day. `--since`
overrides that for every source.

When the start is more than 7 days ago (an early `--since`, or a long time
offline), the Find a Tender and EU Supply sources first backfill it in 7-day
windows,
oldest first, saving each before fetching the next. A window that fails is
saved as far as it got and fetched again after the source's interval. The
API returns the newest releases first, so if a backfill is stopped partway
through a window, restarting without `--since` skips the rest of that
window: restart with `--since` set to the window's start (logged as
`backfilling`) to fill it.

A rate limit is waited out, as the API asks, up to
`sources.<source>.rate_limit_retries` times per request. A failed fetch
keeps what it fetched before the failure.

### EU Supply
The `eu_supply` source scrapes the public tenders on EU Supply's UK portal
(`uk.eu-supply.com`), stored with the source `EU Supply - UK`. The site has
no API and publishes no rate limit, so the client waits a random time from
`request_min_gap` to `request_max_gap` (2s to 10s) between its requests. Each
tender not stored yet has its own page read too, for its description and
CPV codes. Most of its tenders are also published on Find a Tender: a new
one is stored Hidden and Skipped when another source's opportunity released
within 30 days of it has a title at least 90% alike, and its buyer or its
description matches too (80% of the shorter one's words). See
[internal/eusupply](internal/eusupply/README.md).

`fts-once` can be limited with `--planning`, `--opportunities` or
`--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
of waiting.

During development, `go run ./cmd/fetcher` works as well.


## Mock Find a Tender server
To develop without using the real API, run the mock and enable the `fts_mock`
source:
```bash
go run ./cmd/mock-fts --min-gap 5s --max-gap 10s --rate-limit 1000
```
```yaml
sources:
    fts_mock:
        enabled: true
        interval: 1m
        api_root: http://localhost:8080/api/1.0
```

See [internal/mock](internal/mock/README.md) for its options and behaviour.


## WebSocket test tool
`ws-test` reads the same configuration as `job-search` for its defaults:
```bash
go run ./cmd/ws-test dial --http-upgrade --ping  # ping job-search serve's /ws endpoint
go run ./cmd/ws-test dial --direct --ping        # ping ws-test serve
go run ./cmd/ws-test dial --http-upgrade         # send each line of stdin, print replies
go run ./cmd/ws-test serve                       # plain TCP server for websocket hub clients
```

`dial` needs one of `--http-upgrade`, to upgrade an HTTP connection to a
WebSocket on `/ws` as `job-search serve` expects, or `--direct`, to connect
straight to `ws-test serve` over plain TCP with one JSON message per line.
It also takes `--message`, `--listen` and `--timeout`; `--url` and `--origin`
(to test the Origin check) with `--http-upgrade`, and `--addr` with
`--direct`. See `ws-test dial --help`.

`ws-test serve` is a test server only: don't run it alongside
`job-search serve`. Both listen on `server.host` and `server.port`, so the
second one fails with "address already in use".


## Project layout

| Path | Contents |
|---|---|
| `cmd/jobsearch` | The `job-search` command |
| `cmd/fetcher` | The `fetcher` command (Find a Tender fetch, the polling service) |
| `cmd/mock-fts` | The mock FTS server |
| `cmd/ws-test` | The `ws-test` WebSocket test tool |
| `internal/cli` | Cobra commands and shared CLI setup |
| `internal/config` | Configuration loading, validation and editing |
| `internal/fetcher` | Runs a worker per enabled source |
| `internal/search` | Parsing of the general opportunity search (`title:`, `buyer:`, `buyer_id:`, `cpv:`, `!`) and its help |
| `internal/skiprules` | The rules that mark new opportunities Skipped (see [internal/skiprules](internal/skiprules/README.md)) |
| `internal/store` | SQLite storage of opportunities, and its migrations (see [internal/store](internal/store/README.md)) |
| `internal/findatender` | FTS API client and conversion from OCDS to JobSearch types |
| `internal/eusupply` | EU Supply UK portal scraper and conversion to JobSearch types (see [internal/eusupply](internal/eusupply/README.md)) |
| `internal/types` | JobSearch's own opportunity types |
| `internal/util` | The periodic `Worker` used by the fetcher |
| `internal/logging` | slog setup and the HTTP access log |
| `internal/mock` | Mock external services |
| `internal/version` | Version information set at build time |
| `internal/frontend` | Serves the frontend's build output; embeds `dist/`, copied there from `web/dist` by `npm run sync` |
| `web` | The frontend (see [web](web/README.md)) |
| `deploy` | The systemd unit and logrotate configuration (see [Running as a service](#running-as-a-service)) |


## Testing
The store's title search needs SQLite's FTS5 module, which is only compiled in
with the `sqlite_fts5` build tag, so tests (and lint) need it. Without it the
store tests fail with `no such module: fts5`.

```bash
go test -race -tags sqlite_fts5 ./internal/...
go test -tags sqlite_fts5 -coverprofile=cover.out ./internal/... && go tool cover -func=cover.out
golangci-lint run --build-tags sqlite_fts5 ./...
```

Tests don't need network access; HTTP calls go to local `httptest` servers.


## Addition of a new opportunities source
Using a source called `example` as an illustration:

1. **Configuration section:** create `internal/config/conf_source_example.go`,
   copying an existing source such as `conf_source_ftsmock.go`:
   - Add a `JSCKSourceExample = "example"` constant.
   - Define `JSCSourceExample`, embedding `JSCSourcesSource`
     (for `enabled` and `interval`), plus any fields of its own with `cfg` and
     optional `default` tags, e.g. ``APIRoot string `cfg:"api_root"` ``.
   - Implement the `Section` interface (see `internal/config/section.go`):
     copy the `Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue` methods
     from an existing source, under a
     `// ---------- Section methods (see Section) ----------` banner, renaming
     the receiver type and using the new constant in `Flatten`.
2. **Register the section:** add
   ``Example JSCSourceExample `cfg:"example"` `` to
   `JSCSources` in `internal/config/conf_sources.go`. The `cfg` tag
   must match the constant. The source's keys (`sources.example.*`) are then
   loaded, validated and available to the `config` commands with no further
   code.
3. **Client package:** create a package that fetches from the source and
   returns `[]*types.Opportunity` (see `internal/types`). If the source
   publishes OCDS, `internal/findatender` may already be suitable, as it is
   for `fts_mock`: create it with the source's own API root.
4. **Worker:** create `internal/fetcher/worker_example.go`, following
   `worker_ftsmock.go`:
   - Add an `exampleSource = "Example"` constant: the worker's name, and the
     `Source` its opportunities are stored with (set it on each one if the
     client doesn't, as `fetchFTSMock` does).
   - `runExample(ctx, logger, conf, save, since)` builds a `util.Worker`
     with `exampleSource`, `conf.Interval` and a task, then calls `Run`.
   - The task fetches everything since its `since` argument, then returns
     `save(ctx, logger, opportunities)`, which stores them when the fetcher
     has a store. Wrap blocking calls in `util.AwaitTask` so shutdown isn't
     held up, and return `ErrNoNewReleases` when nothing is found, so the
     checkpoint still moves forward. Any other error makes the next run
     retry from the same point.
5. **Start it:** in `Run` in `internal/fetcher/fetcher.go`, add
   ```go
   if f.conf.Example.Enabled {
       start(exampleSource, func(ctx context.Context, since time.Time) error {
           return runExample(ctx, logger, &f.conf.Example, f.save, since)
       })
   }
   ```
   `start` passes the worker its starting point: `--since`, or the newest
   release stored with `exampleSource`.
6. **Tests:** `TestSections` checks the new section implements `Section`
   and that its keys match its `cfg` tag, with no changes needed. In
   `internal/config/conf_sections_test.go`, add the section to
   `TestConcreteSourceSections` and update the key count in
   `TestSourcesSection`. Test the client and worker using `httptest` servers
   rather than the real service.
7. **Docs:** add the source and its keys to the key table in
   `internal/config/README.md`.


## Installation of dev tools
`scripts/install-dev-tools.sh` installs everything needed to develop and
build the project: the Go tools below, with `go install`, and the frontend's
npm packages (`npm ci` in `web/`). It needs Go and Node.js 24 or later (with
npm), as the frontend's scripts run TypeScript directly. Run it again after
a tool's version in it changes.

```bash
scripts/install-dev-tools.sh
```

| Tool | Version | Used for |
|---|---|---|
| `ocdsgen` (`git.avsdev.uk/Go/go-ocds`) | v1.0.0 | Generates the OCDS types in `internal/findatender/ocds` (`ocds_gen.go`) from `extensions.json`, run by `go generate` |
| `swag` | v2.0.0-rc6 | Generates the API's OpenAPI 3.1 description from the handlers' comments (`swag init --v3.1`), run by `go generate` |
| `golangci-lint` | latest | Lints the Go code with the project's `.golangci.json` |
| `goose` | latest | Creates new migration files (see `internal/store`); the migrations themselves are applied by `job-search migrate` |
| `cobra-cli` | latest | Adds cobra commands |

The generators are pinned, as another version could generate different code.
`swag`'s v2, the version with OpenAPI 3 support, has only had release
candidates so far. `go generate ./...` and the build script run the
generators from PATH, so ensure that the GOBIN path (`$HOME/go/bin` by
default) is added to the environment before running any of the tools; the
script says if it isn't.