job-search/README.md

# JobSearch

JobSearch is a service for scraping for contract opportunities. It is primarily
a re-write of the existing nodeJS JobSearch codebase.

It currently fetches releases from the UK Government
[Find a Tender](https://www.find-tender.service.gov.uk/) service (FTS), which
publishes them in the [OCDS](https://standard.open-contracting.org/) format.
A mock of the FTS API is included for development.


## Building
```bash
scripts/build.sh                    # build/job-search-<version>
scripts/build.sh --release          # also package release/job-search-<version>-<arch>.tar.gz
```

The build script runs `go generate ./...` and stamps the version from the
latest git tag. A build from untagged commits is marked as a development build
and isn't packaged unless `--force` is also given.

During development, `go run -tags sqlite_fts5 ./cmd/jobsearch` works as well.
The `sqlite_fts5` tag is needed for the database's title search (see
[Testing](#testing)).


## Configuration
Configuration is read from, in increasing priority:
1. Built-in defaults
2. `config.yaml` in `/etc/job-search` or the working directory, or the file
   given with `--config`
3. `JOBSEARCH_*` environment variables, e.g. `JOBSEARCH_LOGGING_LEVEL=debug`

A `.env` file in the working directory is loaded into the environment first,
and must exist.

Example `config.yaml`:
```yaml
logging:
    level: info
    file: /var/log/job-search.log
database:
    file: /var/lib/job-search/jobsearch.db
sources:
    find_a_tender:
        enabled: true
        interval: 10m
```

See [internal/config](internal/config/README.md) for every key and how the
configuration code works.


## Usage
Global flags: `--config`/`-c` (config file), `--debug`/`-d` and
`--quiet`/`-q` (errors only).

| Command | Purpose |
|---|---|
| `job-search fetch service` | Poll every enabled source on its interval until stopped (Ctrl+C or SIGTERM) |
| `job-search fetch once` | Fetch from Find a Tender once and print the releases as JSON |
| `job-search config init [path]` | Create a config file |
| `job-search config show` | Print the configuration (`--json` or `--yaml` for those formats) |
| `job-search config get <key>` | Print one value, e.g. `logging.level` |
| `job-search config set <key> <value>` | Validate and write one value to the config file |
| `job-search config set --unset <key>` | Remove a value from the config file |
| `job-search migrate` | Apply every pending database migration (same as `migrate up`) |
| `job-search migrate up` | Apply pending migrations (`--steps N` for only N; `--create` to create a missing database file without asking) |
| `job-search migrate down --steps N` | Roll back the last N migrations (`--steps 0` rolls back all of them, deleting all data, and needs `--yes`) |
| `job-search migrate status` | Show the database version and which migrations are applied or pending |
| `job-search version` | Print version information |

`fetch service` logs to `logging.file` if it's set, or to the terminal with
`--foreground`. Both fetch commands take `--since "YYYY-MM-DD[ hh:mm:ss]"`.
`fetch once` can be limited with `--planning`, `--opportunities` or
`--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
of waiting.

The `migrate` commands work on the database in `database.file`. The store
refuses to open a database that isn't at this build's schema version, so run
`job-search migrate` after installing a new version.


## Mock Find a Tender server
To develop without using the real API, run the mock and enable the `fts_mock`
source:
```bash
go run ./cmd/mock-fts --min-gap 5s --max-gap 10s --rate-limit 1000
```
```yaml
sources:
    fts_mock:
        enabled: true
        interval: 1m
        api_root: http://localhost:8080/api/1.0
```

See [internal/mock](internal/mock/README.md) for its options and behaviour.


## WebSocket test tool
`ws-test` reads the same configuration as `job-search` for its defaults:
```bash
go run ./cmd/ws-test dial --ping    # ping the HTTP server's /ws endpoint
go run ./cmd/ws-test dial           # send each line of stdin, print replies
go run ./cmd/ws-test serve          # plain TCP server for websocket hub clients
```

`dial` also takes `--message`, `--listen`, `--origin` (to test the Origin
check), `--url` and `--timeout`; see `ws-test dial --help`.


## Project layout

| Path | Contents |
|---|---|
| `cmd/jobsearch` | The `job-search` command |
| `cmd/mock-fts` | The mock FTS server |
| `cmd/ws-test` | The `ws-test` WebSocket test tool |
| `internal/cli` | Cobra commands and shared CLI setup |
| `internal/config` | Configuration loading, validation and editing |
| `internal/fetcher` | Runs a worker per enabled source |
| `internal/store` | SQLite storage of opportunities, and its migrations (see [internal/store](internal/store/README.md)) |
| `internal/findatender` | FTS API client and conversion from OCDS to JobSearch types |
| `internal/types` | JobSearch's own opportunity types |
| `internal/util` | The periodic `Worker` used by the fetcher |
| `internal/logging` | slog setup and the HTTP access log |
| `internal/mock` | Mock external services |
| `internal/version` | Version information set at build time |


## Testing
The store's title search needs SQLite's FTS5 module, which is only compiled in
with the `sqlite_fts5` build tag, so tests (and lint) need it. Without it the
store tests fail with `no such module: fts5`.

```bash
go test -race -tags sqlite_fts5 ./internal/...
go test -tags sqlite_fts5 -coverprofile=cover.out ./internal/... && go tool cover -func=cover.out
golangci-lint run --build-tags sqlite_fts5 ./...
```

Tests don't need network access; HTTP calls go to local `httptest` servers.


## Addition of a new opportunities source
Using a source called `example` as an illustration:

1. **Configuration section:** create `internal/config/conf_source_example.go`,
   copying an existing source such as `conf_source_ftsmock.go`:
   - Add a `JSCKSourceExample = "example"` constant.
   - Define `JobSearchConfigSourceExample`, embedding `JobSearchConfigSource`
     (for `enabled` and `interval`), plus any fields of its own with `cfg` and
     optional `default` tags, e.g. ``APIRoot string `cfg:"api_root"` ``.
   - Implement the `Section` interface (see `internal/config/section.go`):
     copy the `Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue` methods
     from an existing source, under a
     `// ---------- Section methods (see Section) ----------` banner, renaming
     the receiver type and using the new constant in `Flatten`.
2. **Register the section:** add
   ``Example JobSearchConfigSourceExample `cfg:"example"` `` to
   `JobSearchConfigSources` in `internal/config/conf_sources.go`. The `cfg` tag
   must match the constant. The source's keys (`sources.example.*`) are then
   loaded, validated and available to the `config` commands with no further
   code.
3. **Client package:** create a package that fetches from the source and
   returns `[]*types.Opportunity` (see `internal/types`). If the source
   publishes OCDS, `internal/findatender` may already be suitable, as it is
   for `fts_mock`: create it with the source's own API root.
4. **Worker:** create `internal/fetcher/worker_example.go`, following
   `worker_ftsmock.go`:
   - `runExample(ctx, conf, since)` builds a `util.Worker` with the source's
     name, `conf.Sources.Example.Interval` and a task, then calls `Run`.
   - The task fetches everything since its `since` argument. Wrap blocking
     calls in `util.AwaitTask` so shutdown isn't held up, and return
     `ErrNoNewReleases` when nothing is found, so the checkpoint still moves
     forward. Any other error makes the next run retry from the same point.
5. **Start it:** in `Run` in `internal/fetcher/fetcher.go`, add
   ```go
   if conf.Sources.Example.Enabled {
       start("Example", runExample)
   }
   ```
6. **Tests:** `TestSections` checks the new section implements `Section`
   and that its keys match its `cfg` tag, with no changes needed. In
   `internal/config/conf_sections_test.go`, add the section to
   `TestConcreteSourceSections` and update the key count in
   `TestSourcesSection`. Test the client and worker using `httptest` servers
   rather than the real service.
7. **Docs:** add the source and its keys to the key table in
   `internal/config/README.md`.


## Installation of dev tools
A number of tools have been used in the development of the project. Installation
of these tools can be done using `go install`.

To install the tools:
```bash
go install github.com/spf13/cobra-cli@latest
go install git.avsdev.uk/Go/go-ocds/cmd/ocdsgen@latest
go install github.com/pressly/goose/v3/cmd/goose@latest
go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest
```

`goose` creates new migration files (see `internal/store`); the migrations
themselves are applied by `job-search migrate`. `golangci-lint` uses the
project's `.golangci.json`.

Ensure that the GOBIN path ($HOME/go/bin) is added to the environment before
running any of the tools.