First pass implementation of an SQLite backing store for fetched opportunities

internal/store/README.md

@@ -0,0 +1,114 @@
+# Store
+
+A SQLite store for opportunities and their OCDS releases, built on sqlx + goose and mattn/go-sqlite3.
+
+## Build and test
+
+The title search uses SQLite FTS5, which go-sqlite3 only compiles in with a build tag, so **every build and test needs `-tags sqlite_fts5`**. Without it, opening the store fails with `no such module: fts5`.
+
+```bash
+go test -tags sqlite_fts5 ./...
+go run  -tags sqlite_fts5 ./cmd/jobsearch migrate status
+```
+
+The tag can be made the default for every build with
+`go env -w GOFLAGS=-tags=sqlite_fts5`.
+
+## Layout
+
+```
+internal/types/                    Go struct definitions for tables (opportunities.go, classifications.go)
+internal/store/migrations/         NNNNN_name.sql (Up and Down in one file)
+internal/store/migrate.go          OpenDB, NewMigrator (goose), schema version checks
+internal/store/migrate_test.go     version checks, stepping up and down, failed-migration rollback
+internal/store/queries.go          all SQL, as named constants grouped by table
+internal/store/queries_test.go     prepares every query against the migrated schema
+internal/store/store.go            Store: logic, using the queries via sqlx
+internal/store/store_test.go       behavioural tests
+internal/store/common.go           common for migrate, store and direct access
+```
+
+## Schema
+
+| Table | Key | Notes |
+|---|---|---|
+| `opportunities` | `uuid` (unique) | Updatable. `id INTEGER PRIMARY KEY` exists only to give the search index a stable rowid (fts5). |
+| `opportunity_marks` | `uuid` | `uuid` references `opportunities`. The user's own marks (`hidden`, `skipped`, `interested`, `bid_submitted`, `won`, `lost`), each `BOOLEAN NOT NULL DEFAULT 0`. A `CHECK` constraint stops `won` and `lost` both being set. |
+| `opportunity_releases` | (`uuid`, `release_id`) | Write-once. `uuid` references `opportunities`. `data` is TEXT, so `json_extract` works on it. |
+| `opportunity_release_tags` | (`uuid`, `release_id`, `tag`) | Write-once. (`uuid`, `release_id`) references `opportunity_releases`. |
+| `classification_codes` | `id` | Shared between opportunities. A row is reused only when every content column matches. The table grows over time to include codes not already seen in fetched opportunities. |
+| `opportunity_classifications` | (`opportunity_uuid`, `classification_id`) | Mapping table; replaced as a set when the opportunity is updated. |
+| `opportunities_fts` | rowid = `opportunities.id` | FTS5 index on `opportunities.title`, kept in sync by triggers. |
+
+Foreign keys are enforced (`_foreign_keys=on` in the connection string) and cascade on delete from `opportunities`. The connection also enables WAL mode, a 5s busy timeout, and `_txlock=immediate`, so a save takes the write lock up front rather than failing halfway through.
+
+**When adding a classification scheme flag** (e.g. `is_unspsc`), the migration adding the column must drop and recreate the `classification_codes_content` unique index with the new column included. Otherwise rows differing only by that flag would be merged.
+
+**internal/types**: The structs contain `db` tags to allow sqlx to map table columns to struct members. Any struct members not mapped to a column (including child slices) are referenced by `db:"-"`.
+
+
+## Migrations and Goose
+
+Migrations are embedded in the store package and applied explicitly, by the project's `migrate` command. `store.Open` doesn't migrate: it checks the schema version and fails with
+
+- `ErrNeedsMigration` if migrations are pending (resolved by running `migrate`);
+- `ErrSchemaTooNew` if a newer build has migrated the database.
+
+`store.WithAutoMigrate()` makes `Open` apply pending migrations instead, for tests and throwaway databases. For tooling, `store.OpenDB(path)` opens the database with the store's connection settings, and `store.NewMigrator(db)` returns a goose `Provider` over the embedded migrations.
+
+Each migration runs in its own transaction, so a failed migration is rolled back and leaves the database at the previous version. goose records applied migrations in `goose_db_version`. Each file has `-- +goose Up` and `-- +goose Down` sections. Each section is one `StatementBegin`/`StatementEnd` block, which sends it to SQLite in a single call; that's what lets trigger bodies (which contain `;`) work. A new migration is created with:
+```bash
+goose -dir internal/store/migrations create add_something sql
+```
+
+## Queries and sqlx
+
+All SQL is in `queries.go` as constants, grouped by table in migration order; `store.go` holds only the logic. Inserts use sqlx named parameters (`:column`, matched to the `db` struct tags in `internal/types`). sqlx treats any colon followed by a name inside named SQL as a parameter, including inside SQL comments, so comments go on the Go constant, never inside the string.
+
+`queries_test.go` checks the SQL without needing a test for every code path:
+
+- **TestQueriesPrepare** prepares every query constant against a freshly migrated database, so a syntax error, an unknown table or unknown column fails the test and names the failing statement.
+- **TestQueriesAllListed** parses `queries.go` and fails if a constant isn't listed in that test's `checks` map. Adding a query therefore means adding one line there, and it can't be skipped by accident.
+- **TestListOpportunitiesQueryAllFilters** builds the `ListOpportunities` query with every filter set and runs it, checking that the fragments combine and that the arguments line up with the placeholders.
+- **TestMarksCheckConstraint** checks the database itself rejects `won` and `lost` set together, for writes that bypass the Go validation.
+
+Adding a table involves adding its migration, putting its queries in a new section of `queries.go`, listing them in `checks`, and adding the store methods.
+
+Reads select whole rows (`SELECT *`, `t.*`, `c.*`) which map to a Go struct. sqlx refuses to read a result column that has no matching struct field, so when a migration adds a column, reads fail with `missing destination name <column>` until it is mapped: the field is added to the type in `internal/types` if callers should see it, or to the store's row wrapper if it is internal (like `opportunities.id` in `opportunityRow`). This means that queries should only return data for a single struct type at a time (nested structs are not supported - subsequent queries must be run to populate the referenced structs).
+
+`ListOpportunities` is assembled by `listOpportunitiesQuery` from fragments in `queries.go`: a base `SELECT`, one `filterOpportunity*` clause per filter that is set, and the ordering and paging tail. Each filter fragment takes exactly one argument, except the `filterOpportunityMark*` fragments, which take none (`MarkUnset` puts `NOT` in front of them). Adding a filter means one fragment and one line in `listOpportunitiesQuery`. The classifications and the marks for a page of opportunities are then loaded with one `IN (...)` query each.
+
+## Store API
+
+```go
+s, err := store.Open(ctx, "jobsearch.db") // fails with ErrNeedsMigration if not migrated
+s, err := store.Open(ctx, path, store.WithAutoMigrate()) // tests: migrate on open
+res, err := s.SaveOpportunity(ctx, &opportunity) // SaveResult{OpportunityUpdated, NewReleases}
+res, err := s.SaveOpportunities(ctx, opportunities) // all in one transaction; SaveMultipleResult
+err = s.SetMarks(ctx, types.OpportunityMarks{UUID: uuid, Interested: true}) // ErrNotFound
+t, err := s.GetOpportunity(ctx, uuid) // classifications + marks + releases + tags; ErrNotFound
+ts, err := s.ListOpportunities(ctx, store.OpportunityFilter{
+	Title:      "cloud migr",
+	Hidden:     store.MarkUnset, // MarkAny (default), MarkSet or MarkUnset per mark
+	Interested: store.MarkSet,
+	Limit:      20,
+})
+rs, err := s.ListReleases(ctx, uuid) // oldest first, with tags
+ok, err := s.ReleaseExists(ctx, uuid, releaseID)
+```
+
+**Saving.** `SaveOpportunity` writes everything in one transaction. `SaveOpportunities` does the same for several opportunities in a single transaction, reporting totals (`NewOpportunities`, `UpdatedOpportunities`, `NewReleases`) and each opportunity's `SaveResult`:
+
+- **Opportunity normalisation.** The opportunity is normalised in place. Releases, Marks and tags get the opportunity's `UUID` and their `ReleaseID`, times become UTC, `LatestReleaseDate` becomes the newest release date, and classifications get their database `ID`s.
+- **Minimal writing.** Releases and tags are write-once. A release already stored (same `uuid` and `release_id`) is skipped, and `NewReleases` counts only the new ones. `ReleaseExists` lets a fetcher skip converting a release it already has.
+- **Opportunity row updates.** The opportunity row and its classifications are replaced only if the save's newest release date is at least the stored `latest_release_date`. An older release arriving late is still stored, but never overwrites newer opportunity data; `OpportunityUpdated` reports which happened.
+- **Marks are kept.** An opportunity's marks are stored when it is first saved; saving it again never changes them, so fetched data (which carries no marks) can't reset what the user has set.
+
+**Marks.** `SetMarks` replaces an opportunity's marks, returning `ErrNotFound` if the opportunity isn't stored. `won` and `lost` exclude each other: `SetMarks`, `SaveOpportunity` and `SaveOpportunities` reject marks with both set with `types.ErrWonAndLost` (from `OpportunityMarks.Validate`), and the database's `CHECK` constraint backs this up.
+
+
+**Listing.** `ListOpportunities` filters by status, classification code, title text, a range on `LatestReleaseDate` and each mark (`MarkSet` or `MarkUnset`; an opportunity with no marks stored has every mark unset), newest first, 50 per page by default. It loads classifications and marks but not releases (which can be large); those are loaded by `ListReleases` or `GetOpportunity`.
+
+
+**Title search.** The words are matched against the title using FTS5: every word must appear, each also matches as a prefix, case and accents are ignored, and "migrating" finds "migration", etc. Input is sanitised, so FTS5 operators and quotes typed by a user are searched as plain words, never interpreted.
+