Added database.change_batch_retention, deleting old change batches
Each save that records a batch also deletes, in the same transaction, the batches older than database.change_batch_retention (30 days by default; 0 or less keeps them for ever), with their records of opportunities. A client still holding a deleted batch gets 410 Gone from ?changed_after= and reloads the whole list, as for any unknown batch.
This commit is contained in:
1 parent
1988d07a29
commit
65e5bc6b26
10 files changed
+153
-19
No files matched your search
@@ -97,6 +97,7 @@ A soft convention for the SQL in `queries.go`, followed so the queries look alik
|
||||
```go
|
||||
s, err := store.Open(ctx, "jobsearch.db") // fails with ErrNeedsMigration if not migrated
|
||||
s, err := store.Open(ctx, path, store.WithAutoMigrate()) // tests: migrate on open
|
||||
s, err := store.Open(ctx, path, store.WithChangeBatchRetention(30*24*time.Hour))
|
||||
res, err := s.SaveOpportunity(ctx, &opportunity) // SaveResult{OpportunityUpdated, NewReleases}
|
||||
res, err := s.SaveOpportunities(ctx, opportunities) // all in one transaction; SaveMultipleResult
|
||||
err = s.SetMarks(ctx, types.OpportunityMarks{UUID: uuid, Interested: true}) // ErrNotFound
|
||||
@@ -119,14 +120,14 @@ ok, err := s.ReleaseExists(ctx, uuid, releaseID)
|
||||
- **Minimal writing.** Releases and tags are write-once. A release already stored (same `uuid` and `release_id`) is skipped, and `NewReleases` counts only the new ones. `ReleaseExists` lets a fetcher skip converting a release it already has.
|
||||
- **Opportunity row updates.** The opportunity row and its classifications are replaced only if the save's newest release date is at least the stored `latest_release_date`. An older release arriving late is still stored, but never overwrites newer opportunity data; `OpportunityUpdated` reports which happened.
|
||||
- **Marks are kept.** An opportunity's marks are stored when it is first saved; saving it again never changes them, so fetched data (which carries no marks) can't reset what the user has set.
|
||||
- **Change batches.** The opportunities a save adds or gives new releases are recorded, in the same transaction, in a change batch named by a random hash; `SaveOpportunities` reports it as `ChangeBatch` ("" if nothing changed). A batch's `created_at` is kept later than every earlier batch's, even if the clock goes back. Recording one deletes the batches older than the store's retention (`WithChangeBatchRetention`; without it, or with 0 or less, they are kept for ever).
|
||||
|
||||
**Marks.** `SetMarks` replaces an opportunity's marks, returning `ErrNotFound` if the opportunity isn't stored. `won` and `lost` exclude each other: `SetMarks`, `SaveOpportunity` and `SaveOpportunities` reject marks with both set with `types.ErrWonAndLost` (from `OpportunityMarks.Validate`), and the database's `CHECK` constraint backs this up.
|
||||
**Marks.** `SetMarks` replaces an opportunity's marks, returning `ErrNotFound` if the opportunity isn't stored. `won` and `lost` exclude each other: `SetMarks`, `SaveOpportunity` and `SaveOpportunities` reject marks with both set with `types.ErrWonAndLost` (from `OpportunityMarks.Validate`), and the database's `CHECK` constraint backs this up. A change of marks is recorded in a change batch of its own, so a client catching up sees it.
|
||||
|
||||
|
||||
**Listing.** `ListOpportunities` filters by status, classification code, title text, a range on `LatestReleaseDate` and each mark (`MarkSet` or `MarkUnset`; an opportunity with no marks stored has every mark unset), newest first, 50 per page by default. It loads classifications and marks but not releases (which can be large); those are loaded by `ListReleases` or `GetOpportunity`.
|
||||
|
||||
|
||||
**Changes.** Each save that adds opportunities or gives them new releases records them in a change batch, in the same transaction, and `SaveOpportunities` reports the batch's hash (`ChangeBatch`). `SetMarks` records the opportunity in a batch of its own. A client that read the list after batch B (`LatestChangeBatch`, read before listing) gets everything changed since from `ListChangedOpportunities(ctx, B)`, with the newest batch to ask from next time; an unknown hash is `ErrUnknownBatch`. Batches are ordered by `created_at`, which `recordChangeBatch` keeps strictly increasing even if the clock goes back, and only batches up to the newest when the call starts are read, so one recorded meanwhile is left for next time rather than missed. Batches are never deleted yet.
|
||||
`LatestChangeBatch` returns the hash of the newest change batch ("" if there is none). Read before listing, it is the batch to give `ListChangedOpportunities`, which returns the opportunities in every later batch, once each, unpaged and ordered as `ListOpportunities` orders them, with the hash of the newest of those batches to give it next time. Batches recorded while it runs are left for next time rather than missed. A hash it doesn't know, such as a deleted batch's, is `ErrUnknownBatch`.
|
||||
|
||||
|
||||
**Title search.** The words are matched against the title using FTS5: every word must appear, each also matches as a prefix, case and accents are ignored, and "migrating" finds "migration", etc. Input is sanitised, so FTS5 operators and quotes typed by a user are searched as plain words, never interpreted.
|
||||
|
||||
Reference in new issue
Block a user