bwilliams
67215a6aaa
CODE_CONVENTIONS.md sets out the conventions for Go code in this repository: naming, layout, comments, errors, logging, configuration and tests. .golangci.json enables the linters that check the parts that can be checked (each limited to the rules the conventions use); the rest is applied by hand. Applied across the codebase: - Naming: full-word names in application code, initialisms in one case (BuyerID, PrintJSON, PrintYAML, ocdsGetBuyerID), no names shadowing built-ins (min/max are minGap/maxGap), mustRaw renamed rawJSON as it returns nil rather than panicking, and config.ConfigLevelToLogLevel renamed LevelToLogLevel. - Layout: lines kept to 80 characters where possible, assignments taken out of if and for statements that don't fit, wrapped arguments all on one line or one per line, steps separated by blank lines, no else after a return, function and struct literals wrapped. - Comments: doc comments on every exported name (including each section's Map, Flatten, TypeOf, ValueOf and SetValue), section banners, British spelling. - Errors: a created error starts with its package name, a wrapping error adds only what was being done, sentinels in one commented var block, errors.Is and errors.As in place of == and type assertions, and no err := shadowing of a named result. That fixes the mock server losing its access log's close error. - Logging: the fetcher passes its workers an untagged logger, so findatender's lines are tagged with their own component rather than with fetcher's as well; cmd/mock-fts logs "stopped" with slog. - Configuration: util.WithTimeout (moved from the websocket package) applies a timeout of 0 or less as no limit, used for the websocket write and ping, TCP close and tcpserver shutdown timeouts; every setting documents what 0 or empty means, including the rate limit rule's keys in internal/config/README.md. - Tests: related tests merged into table-driven ones with named cases. Separately from CODE_CONVENTIONS.md, internal/store/README.md gains a soft SQL style convention for the queries in internal/store/queries.go, and the queries are reformatted to follow it.
127 lines
11 KiB
Markdown
127 lines
11 KiB
Markdown
# Store
|
|
|
|
A SQLite store for opportunities and their OCDS releases, built on sqlx + goose and mattn/go-sqlite3.
|
|
|
|
## Build and test
|
|
|
|
The title search uses SQLite FTS5, which go-sqlite3 only compiles in with a build tag, so **every build and test needs `-tags sqlite_fts5`**. Without it, opening the store fails with `no such module: fts5`.
|
|
|
|
```bash
|
|
go test -tags sqlite_fts5 ./...
|
|
go run -tags sqlite_fts5 ./cmd/jobsearch migrate status
|
|
```
|
|
|
|
The tag can be made the default for every build with
|
|
`go env -w GOFLAGS=-tags=sqlite_fts5`.
|
|
|
|
## Layout
|
|
|
|
```
|
|
internal/types/ Go struct definitions for tables (opportunities.go, classifications.go)
|
|
internal/store/migrations/ NNNNN_name.sql (Up and Down in one file)
|
|
internal/store/migrate.go OpenDB, NewMigrator (goose), schema version checks
|
|
internal/store/migrate_test.go version checks, stepping up and down, failed-migration rollback
|
|
internal/store/queries.go all SQL, as named constants grouped by table
|
|
internal/store/queries_test.go prepares every query against the migrated schema
|
|
internal/store/store.go Store: logic, using the queries via sqlx
|
|
internal/store/store_test.go behavioural tests
|
|
internal/store/common.go common for migrate, store and direct access
|
|
```
|
|
|
|
## Schema
|
|
|
|
| Table | Key | Notes |
|
|
|---|---|---|
|
|
| `opportunities` | `uuid` (unique) | Updatable. `id INTEGER PRIMARY KEY` exists only to give the search index a stable rowid (fts5). |
|
|
| `opportunity_marks` | `uuid` | `uuid` references `opportunities`. The user's own marks (`hidden`, `skipped`, `interested`, `bid_submitted`, `won`, `lost`), each `BOOLEAN NOT NULL DEFAULT 0`. A `CHECK` constraint stops `won` and `lost` both being set. |
|
|
| `opportunity_releases` | (`uuid`, `release_id`) | Write-once. `uuid` references `opportunities`. `data` is TEXT, so `json_extract` works on it. |
|
|
| `opportunity_release_tags` | (`uuid`, `release_id`, `tag`) | Write-once. (`uuid`, `release_id`) references `opportunity_releases`. |
|
|
| `classification_codes` | `id` | Shared between opportunities. A row is reused only when every content column matches. The table grows over time to include codes not already seen in fetched opportunities. |
|
|
| `opportunity_classifications` | (`opportunity_uuid`, `classification_id`) | Mapping table; replaced as a set when the opportunity is updated. |
|
|
| `opportunities_fts` | rowid = `opportunities.id` | FTS5 index on `opportunities.title`, kept in sync by triggers. |
|
|
|
|
Foreign keys are enforced (`_foreign_keys=on` in the connection string) and cascade on delete from `opportunities`. The connection also enables WAL mode, a 5s busy timeout, and `_txlock=immediate`, so a save takes the write lock up front rather than failing halfway through.
|
|
|
|
**When adding a classification scheme flag** (e.g. `is_unspsc`), the migration adding the column must drop and recreate the `classification_codes_content` unique index with the new column included. Otherwise rows differing only by that flag would be merged.
|
|
|
|
**internal/types**: The structs contain `db` tags to allow sqlx to map table columns to struct members. Any struct members not mapped to a column (including child slices) are referenced by `db:"-"`.
|
|
|
|
|
|
## Migrations and Goose
|
|
|
|
Migrations are embedded in the store package and applied explicitly, by the project's `migrate` command. `store.Open` doesn't migrate: it checks the schema version and fails with
|
|
|
|
- `ErrNeedsMigration` if migrations are pending (resolved by running `migrate`);
|
|
- `ErrSchemaTooNew` if a newer build has migrated the database.
|
|
|
|
`store.WithAutoMigrate()` makes `Open` apply pending migrations instead, for tests and throwaway databases. For tooling, `store.OpenDB(path)` opens the database with the store's connection settings, and `store.NewMigrator(db)` returns a goose `Provider` over the embedded migrations.
|
|
|
|
Each migration runs in its own transaction, so a failed migration is rolled back and leaves the database at the previous version. goose records applied migrations in `goose_db_version`. Each file has `-- +goose Up` and `-- +goose Down` sections. Each section is one `StatementBegin`/`StatementEnd` block, which sends it to SQLite in a single call; that's what lets trigger bodies (which contain `;`) work. A new migration is created with:
|
|
```bash
|
|
goose -dir internal/store/migrations create add_something sql
|
|
```
|
|
|
|
## Queries and sqlx
|
|
|
|
All SQL is in `queries.go` as constants, grouped by table in migration order; `store.go` holds only the logic. Inserts use sqlx named parameters (`:column`, matched to the `db` struct tags in `internal/types`). sqlx treats any colon followed by a name inside named SQL as a parameter, including inside SQL comments, so comments go on the Go constant, never inside the string.
|
|
|
|
`queries_test.go` checks the SQL without needing a test for every code path:
|
|
|
|
- **TestQueriesPrepare** prepares every query constant against a freshly migrated database, so a syntax error, an unknown table or unknown column fails the test and names the failing statement.
|
|
- **TestQueriesAllListed** parses `queries.go` and fails if a constant isn't listed in that test's `checks` map. Adding a query therefore means adding one line there, and it can't be skipped by accident.
|
|
- **TestListOpportunitiesQueryAllFilters** builds the `ListOpportunities` query with every filter set and runs it, checking that the fragments combine and that the arguments line up with the placeholders.
|
|
- **TestMarksCheckConstraint** checks the database itself rejects `won` and `lost` set together, for writes that bypass the Go validation.
|
|
|
|
Adding a table involves adding its migration, putting its queries in a new section of `queries.go`, listing them in `checks`, and adding the store methods.
|
|
|
|
Reads select whole rows (`SELECT *`, `t.*`, `c.*`) which map to a Go struct. sqlx refuses to read a result column that has no matching struct field, so when a migration adds a column, reads fail with `missing destination name <column>` until it is mapped: the field is added to the type in `internal/types` if callers should see it, or to the store's row wrapper if it is internal (like `opportunities.id` in `opportunityRow`). This means that queries should only return data for a single struct type at a time (nested structs are not supported - subsequent queries must be run to populate the referenced structs).
|
|
|
|
`ListOpportunities` is assembled by `listOpportunitiesQuery` from fragments in `queries.go`: a base `SELECT`, one `filterOpportunity*` clause per filter that is set, and the ordering and paging tail. Each filter fragment takes exactly one argument, except the `filterOpportunityMark*` fragments, which take none (`MarkUnset` puts `NOT` in front of them). Adding a filter means one fragment and one line in `listOpportunitiesQuery`. The classifications and the marks for a page of opportunities are then loaded with one `IN (...)` query each.
|
|
|
|
### SQL style
|
|
|
|
A soft convention for the SQL in `queries.go`, followed so the queries look alike and read easily. A query that reads better another way can depart from it.
|
|
|
|
- A statement starts on the line after the opening backtick. A fragment combined into another query (`filterOpportunity*`) starts straight after it.
|
|
- Keywords are uppercase (`SELECT`, `INSERT INTO`, `ON CONFLICT`); table and column names are the lowercase snake_case of the schema.
|
|
- A short statement may stay on one line (`SELECT t.* FROM opportunities t WHERE t.uuid = ?`). A longer one has one clause per line, each at the start of the line: `SELECT`, `FROM`, `JOIN`, `WHERE`, `ORDER BY`, `LIMIT`.
|
|
- A column list that doesn't fit on one line goes inside its brackets, indented one tab and filled to about 80 characters per line. The `VALUES` list wraps at the same columns, so each line of values matches the line of columns above it.
|
|
- `ON CONFLICT … DO UPDATE SET` puts one `column = excluded.column` per line, indented one tab.
|
|
- A subquery (`EXISTS (…)`, `IN (…)`) is indented one tab inside its brackets, with the closing bracket at the start of a line.
|
|
- Tables are aliased consistently: `t` for `opportunities`, `c` for `classification_codes`, `tc` for `opportunity_classifications`, `m` for `opportunity_marks`.
|
|
|
|
## Store API
|
|
|
|
```go
|
|
s, err := store.Open(ctx, "jobsearch.db") // fails with ErrNeedsMigration if not migrated
|
|
s, err := store.Open(ctx, path, store.WithAutoMigrate()) // tests: migrate on open
|
|
res, err := s.SaveOpportunity(ctx, &opportunity) // SaveResult{OpportunityUpdated, NewReleases}
|
|
res, err := s.SaveOpportunities(ctx, opportunities) // all in one transaction; SaveMultipleResult
|
|
err = s.SetMarks(ctx, types.OpportunityMarks{UUID: uuid, Interested: true}) // ErrNotFound
|
|
t, err := s.GetOpportunity(ctx, uuid) // classifications + marks + releases + tags; ErrNotFound
|
|
ts, err := s.ListOpportunities(ctx, store.OpportunityFilter{
|
|
Title: "cloud migr",
|
|
Hidden: store.MarkUnset, // MarkAny (default), MarkSet or MarkUnset per mark
|
|
Interested: store.MarkSet,
|
|
Limit: 20,
|
|
})
|
|
rs, err := s.ListReleases(ctx, uuid) // oldest first, with tags
|
|
ok, err := s.ReleaseExists(ctx, uuid, releaseID)
|
|
```
|
|
|
|
**Saving.** `SaveOpportunity` writes everything in one transaction. `SaveOpportunities` does the same for several opportunities in a single transaction, reporting totals (`NewOpportunities`, `UpdatedOpportunities`, `NewReleases`) and each opportunity's `SaveResult`:
|
|
|
|
- **Opportunity normalisation.** The opportunity is normalised in place. Releases, Marks and tags get the opportunity's `UUID` and their `ReleaseID`, times become UTC, `LatestReleaseDate` becomes the newest release date, and classifications get their database `ID`s.
|
|
- **Minimal writing.** Releases and tags are write-once. A release already stored (same `uuid` and `release_id`) is skipped, and `NewReleases` counts only the new ones. `ReleaseExists` lets a fetcher skip converting a release it already has.
|
|
- **Opportunity row updates.** The opportunity row and its classifications are replaced only if the save's newest release date is at least the stored `latest_release_date`. An older release arriving late is still stored, but never overwrites newer opportunity data; `OpportunityUpdated` reports which happened.
|
|
- **Marks are kept.** An opportunity's marks are stored when it is first saved; saving it again never changes them, so fetched data (which carries no marks) can't reset what the user has set.
|
|
|
|
**Marks.** `SetMarks` replaces an opportunity's marks, returning `ErrNotFound` if the opportunity isn't stored. `won` and `lost` exclude each other: `SetMarks`, `SaveOpportunity` and `SaveOpportunities` reject marks with both set with `types.ErrWonAndLost` (from `OpportunityMarks.Validate`), and the database's `CHECK` constraint backs this up.
|
|
|
|
|
|
**Listing.** `ListOpportunities` filters by status, classification code, title text, a range on `LatestReleaseDate` and each mark (`MarkSet` or `MarkUnset`; an opportunity with no marks stored has every mark unset), newest first, 50 per page by default. It loads classifications and marks but not releases (which can be large); those are loaded by `ListReleases` or `GetOpportunity`.
|
|
|
|
|
|
**Title search.** The words are matched against the title using FTS5: every word must appear, each also matches as a prefix, case and accents are ignored, and "migrating" finds "migration", etc. Input is sanitised, so FTS5 operators and quotes typed by a user are searched as plain words, never interpreted.
|
|
|