Added change batches, so clients can catch up on changed opportunities

Every save that stores new opportunities or new releases records them in a
change batch (migration 00008: opportunity_change_batches and
opportunity_change_batch_opportunities; the opportunities table is
unchanged), as does every change of marks. A batch is named by a random
SHA-256 hash, so it can't be guessed, and ordered by its created_at, which
is kept strictly increasing even if the clock goes back.

GET /api/opportunities pages now carry the newest batch, read before the
page, and GET /api/opportunities?changed_after=<batch> returns every
opportunity changed since that batch, unpaged, with the newest batch it
includes. Offset and limit can't be combined with it, and an unknown batch
is 410 Gone, telling the client to reload the whole list.

OpportunitiesUpdated now carries its save's batch, and a "large" flag: over
50 opportunities, new and updated are left empty and only the summary
counts them, so a big catch-up no longer sends tens of thousands of UUIDs.
This commit is contained in:
bwilliams committed 2026-10-16 15:25:48 +01:00
1 parent 51d6470649
commit 155b84caea
13 files changed
+699 -29

No files matched your search

+8 -1
View File
@@ -39,6 +39,8 @@ internal/store/common.go common for migrate, store and direct access
| `classification_codes` | `id` | Shared between opportunities. A row is reused only when every content column matches. The table grows over time to include codes not already seen in fetched opportunities. |
| `opportunity_classifications` | (`opportunity_uuid`, `classification_id`) | Mapping table; replaced as a set when the opportunity is updated. |
| `opportunities_fts` | rowid = `opportunities.id` | FTS5 index on `opportunities.title`, kept in sync by triggers. |
| `opportunity_change_batches` | `id`; `hash` (unique) | Write-once. One row per save that adds opportunities or gives them new releases, and per change of marks. `hash` names the batch to clients (random, so unguessable); `created_at` orders the batches. |
| `opportunity_change_batch_opportunities` | (`batch_id`, `opportunity_uuid`) | The opportunities in each batch. References both tables, cascading on delete. |
Foreign keys are enforced (`_foreign_keys=on` in the connection string) and cascade on delete from `opportunities`. The connection also enables WAL mode, a 5s busy timeout, and `_txlock=immediate`, so a save takes the write lock up front rather than failing halfway through.
@@ -88,7 +90,7 @@ A soft convention for the SQL in `queries.go`, followed so the queries look alik
- A column list that doesn't fit on one line goes inside its brackets, indented one tab and filled to about 80 characters per line. The `VALUES` list wraps at the same columns, so each line of values matches the line of columns above it.
- `ON CONFLICT … DO UPDATE SET` puts one `column = excluded.column` per line, indented one tab.
- A subquery (`EXISTS (…)`, `IN (…)`) is indented one tab inside its brackets, with the closing bracket at the start of a line.
- Tables are aliased consistently: `t` for `opportunities`, `c` for `classification_codes`, `tc` for `opportunity_classifications`, `m` for `opportunity_marks`.
- Tables are aliased consistently: `t` for `opportunities`, `c` for `classification_codes`, `tc` for `opportunity_classifications`, `m` for `opportunity_marks`, `b` for `opportunity_change_batches`, `bo` for `opportunity_change_batch_opportunities`.
## Store API
@@ -106,6 +108,8 @@ ts, err := s.ListOpportunities(ctx, store.OpportunityFilter{
Limit: 20,
})
rs, err := s.ListReleases(ctx, uuid) // oldest first, with tags
batch, err := s.LatestChangeBatch(ctx) // hash of the newest change batch, or ""
ts, batch, err = s.ListChangedOpportunities(ctx, batch) // changed since; ErrUnknownBatch
ok, err := s.ReleaseExists(ctx, uuid, releaseID)
```
@@ -122,5 +126,8 @@ ok, err := s.ReleaseExists(ctx, uuid, releaseID)
**Listing.** `ListOpportunities` filters by status, classification code, title text, a range on `LatestReleaseDate` and each mark (`MarkSet` or `MarkUnset`; an opportunity with no marks stored has every mark unset), newest first, 50 per page by default. It loads classifications and marks but not releases (which can be large); those are loaded by `ListReleases` or `GetOpportunity`.
**Changes.** Each save that adds opportunities or gives them new releases records them in a change batch, in the same transaction, and `SaveOpportunities` reports the batch's hash (`ChangeBatch`). `SetMarks` records the opportunity in a batch of its own. A client that read the list after batch B (`LatestChangeBatch`, read before listing) gets everything changed since from `ListChangedOpportunities(ctx, B)`, with the newest batch to ask from next time; an unknown hash is `ErrUnknownBatch`. Batches are ordered by `created_at`, which `recordChangeBatch` keeps strictly increasing even if the clock goes back, and only batches up to the newest when the call starts are read, so one recorded meanwhile is left for next time rather than missed. Batches are never deleted yet.
**Title search.** The words are matched against the title using FTS5: every word must appear, each also matches as a prefix, case and accents are ignored, and "migrating" finds "migration", etc. Input is sanitised, so FTS5 operators and quotes typed by a user are searched as plain words, never interpreted.