First pass implementation of an SQLite backing store for fetched opportunities

README.md

@@ -19,7 +19,9 @@ The build script runs `go generate ./...` and stamps the version from the
 latest git tag. A build from untagged commits is marked as a development build
 and isn't packaged unless `--force` is also given.
 
-During development, `go run ./cmd/jobsearch` works as well.
+During development, `go run -tags sqlite_fts5 ./cmd/jobsearch` works as well.
+The `sqlite_fts5` tag is needed for the database's title search (see
+[Testing](#testing)).
 
 
 ## Configuration
@@ -37,6 +39,8 @@ Example `config.yaml`:
 logging:
     level: info
     file: /var/log/job-search.log
+database:
+    file: /var/lib/job-search/jobsearch.db
 sources:
     find_a_tender:
         enabled: true
@@ -60,6 +64,10 @@ Global flags: `--config`/`-c` (config file), `--debug`/`-d` and
 | `job-search config get <key>` | Print one value, e.g. `logging.level` |
 | `job-search config set <key> <value>` | Validate and write one value to the config file |
 | `job-search config set --unset <key>` | Remove a value from the config file |
+| `job-search migrate` | Apply every pending database migration (same as `migrate up`) |
+| `job-search migrate up` | Apply pending migrations (`--steps N` for only N; `--create` to create a missing database file without asking) |
+| `job-search migrate down --steps N` | Roll back the last N migrations (`--steps 0` rolls back all of them, deleting all data, and needs `--yes`) |
+| `job-search migrate status` | Show the database version and which migrations are applied or pending |
 | `job-search version` | Print version information |
 
 `fetch service` logs to `logging.file` if it's set, or to the terminal with
@@ -68,6 +76,10 @@ Global flags: `--config`/`-c` (config file), `--debug`/`-d` and
 `--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
 of waiting.
 
+The `migrate` commands work on the database in `database.file`. The store
+refuses to open a database that isn't at this build's schema version, so run
+`job-search migrate` after installing a new version.
+
 
 ## Mock Find a Tender server
 To develop without using the real API, run the mock and enable the `fts_mock`
@@ -95,6 +107,7 @@ See [internal/mock](internal/mock/README.md) for its options and behaviour.
 | `internal/cli` | Cobra commands and shared CLI setup |
 | `internal/config` | Configuration loading, validation and editing |
 | `internal/fetcher` | Runs a worker per enabled source |
+| `internal/store` | SQLite storage of opportunities, and its migrations (see [internal/store](internal/store/README.md)) |
 | `internal/findATender` | FTS API client and conversion from OCDS to JobSearch types |
 | `internal/types` | JobSearch's own opportunity types |
 | `internal/util` | The periodic `Worker` used by the fetcher |
@@ -104,9 +117,14 @@ See [internal/mock](internal/mock/README.md) for its options and behaviour.
 
 
 ## Testing
+The store's title search needs SQLite's FTS5 module, which is only compiled in
+with the `sqlite_fts5` build tag, so tests (and lint) need it. Without it the
+store tests fail with `no such module: fts5`.
+
 ```bash
-go test -race ./internal/...
-go test -coverprofile=cover.out ./internal/... && go tool cover -func=cover.out
+go test -race -tags sqlite_fts5 ./internal/...
+go test -tags sqlite_fts5 -coverprofile=cover.out ./internal/... && go tool cover -func=cover.out
+golangci-lint run --build-tags sqlite_fts5 ./...
 ```
 
 Tests don't need network access; HTTP calls go to local `httptest` servers.
@@ -168,7 +186,13 @@ To install the tools:
 ```bash
 go install github.com/spf13/cobra-cli@latest
 go install git.avsdev.uk/Go/go-ocds/cmd/ocdsgen@latest
+go install github.com/pressly/goose/v3/cmd/goose@latest
+go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest
 ```
 
+`goose` creates new migration files (see `internal/store`); the migrations
+themselves are applied by `job-search migrate`. `golangci-lint` uses the
+project's `.golangci.json`.
+
 Ensure that the GOBIN path ($HOME/go/bin) is added to the environment before
 running any of the tools.