Replaced the Title filter with a general search, run by the backend

GET /api/opportunities/search?q= returns the UUIDs of the matching
opportunities, and the frontend filters its list by them. Words without
a prompt search the title and description; title:, buyer:, buyer_id:
and cpv: search only their own field, and a leading ! leaves out what a
term matches. Repeats of one prompt are OR, everything else AND. CPV
terms are checked per code: cpv:72 !cpv:7221 needs a code starting 72
that doesn't start 7221.

Unknown prompts are reported, and every response carries the help that
the search box's "?" panel shows. Migration 00011 adds the description
and buyer name to the full-text index, and indexes CPV descriptions.
This commit is contained in:
bwilliams committed 2026-10-22 13:10:25 +01:00
1 parent 0a777792d8
commit b0230e6847
29 files changed
+1913 -43

No files matched your search

+13 -2
View File
@@ -4,7 +4,7 @@ A SQLite store for opportunities and their OCDS releases, built on sqlx + goose
## Build and test
The title search uses SQLite FTS5, which go-sqlite3 only compiles in with a build tag, so **every build and test needs `-tags sqlite_fts5`**. Without it, opening the store fails with `no such module: fts5`.
The title and general searches use SQLite FTS5, which go-sqlite3 only compiles in with a build tag, so **every build and test needs `-tags sqlite_fts5`**. Without it, opening the store fails with `no such module: fts5`.
```bash
go test -tags sqlite_fts5 ./...
@@ -24,7 +24,9 @@ internal/store/migrate_test.go version checks, stepping up and down, failed-
internal/store/queries.go all SQL, as named constants grouped by table
internal/store/queries_test.go prepares every query against the migrated schema
internal/store/store.go Store: logic, using the queries via sqlx
internal/store/search.go SearchOpportunities: the general search's query
internal/store/store_test.go behavioural tests
internal/store/search_test.go behavioural tests of the general search
internal/store/common.go common for migrate, store and direct access
```
@@ -38,7 +40,8 @@ internal/store/common.go common for migrate, store and direct access
| `opportunity_release_tags` | (`uuid`, `release_id`, `tag`) | Write-once. (`uuid`, `release_id`) references `opportunity_releases`. |
| `classification_codes` | `id` | Shared between opportunities. A CPV code has one row (unique on `code` where `is_cpv`), keeping the longest description it has been given: EU Supply cuts descriptions to 64 characters. A code of any other scheme gets a row per distinct content. The table grows over time to include codes not already seen in fetched opportunities. |
| `opportunity_classifications` | (`opportunity_uuid`, `classification_id`) | Mapping table; replaced as a set when the opportunity is updated. |
| `opportunities_fts` | rowid = `opportunities.id` | FTS5 index on `opportunities.title`, kept in sync by triggers. |
| `opportunities_fts` | rowid = `opportunities.id` | FTS5 index on `opportunities.title`, `description` and `buyer_name`, kept in sync by triggers. |
| `classification_codes_fts` | rowid = `classification_codes.id` | FTS5 index on `classification_codes.classification` (the description), kept in sync by triggers. |
| `opportunity_change_batches` | `id`; `hash` (unique) | Write-once. One row per save that adds opportunities or gives them new releases, and per change of marks. `hash` names the batch to clients (random, so unguessable); `created_at` orders the batches. |
| `opportunity_change_batch_opportunities` | (`batch_id`, `opportunity_uuid`) | The opportunities in each batch. References both tables, cascading on delete. |
@@ -71,6 +74,7 @@ All SQL is in `queries.go` as constants, grouped by table in migration order; `s
- **TestQueriesPrepare** prepares every query constant against a freshly migrated database, so a syntax error, an unknown table or unknown column fails the test and names the failing statement.
- **TestQueriesAllListed** parses `queries.go` and fails if a constant isn't listed in that test's `checks` map. Adding a query therefore means adding one line there, and it can't be skipped by accident.
- **TestSearchOpportunitiesQueryAllTerms** does the same for `SearchOpportunities`, with a term of every kind, plain and negated.
- **TestListOpportunitiesQueryAllFilters** builds the `ListOpportunities` query with every filter set and runs it, checking that the fragments combine and that the arguments line up with the placeholders.
- **TestMarksCheckConstraint** checks the database itself rejects `won` and `lost` set together, for writes that bypass the Go validation.
@@ -111,6 +115,7 @@ ts, err := s.ListOpportunities(ctx, store.OpportunityFilter{
rs, err := s.ListReleases(ctx, uuid) // oldest first, with tags
batch, err := s.LatestChangeBatch(ctx) // hash of the newest change batch, or ""
ts, batch, err = s.ListChangedOpportunities(ctx, batch, time.Time{}) // changed since; ErrUnknownBatch
uuids, err := s.SearchOpportunities(ctx, search.Parse("cloud buyer:leeds"), releasedFrom) // never nil
ok, err := s.ReleaseExists(ctx, uuid, releaseID)
```
@@ -132,3 +137,9 @@ ok, err := s.ReleaseExists(ctx, uuid, releaseID)
**Title search.** The words are matched against the title using FTS5: every word must appear, each also matches as a prefix, case and accents are ignored, and "migrating" finds "migration", etc. Input is sanitised, so FTS5 operators and quotes typed by a user are searched as plain words, never interpreted.
**General search.** `SearchOpportunities` runs a `search.Query` (parsed by `internal/search`, which documents the syntax) and returns only the matching UUIDs, for the frontend to filter its own list by. It is assembled by `searchOpportunitiesQuery` from the `searchOpportunity*` and `searchCPV*` fragments:
- Words without a prompt search `{title description}` in `opportunities_fts`, and must all match. `title:` and `buyer:` search the `title` and `buyer_name` columns; repeats of one prompt are joined by OR into one clause. Every negated full-text term goes into one `NOT IN` clause, joined by OR. Each term is one FTS5 phrase built from `Term.Words`, its last word a prefix, so typed input is never FTS5 syntax.
- `buyer_id:` matches the whole ID, ignoring case (`IN`, expanded by `sqlx.In`).
- CPV terms are checked against each code, inside one `EXISTS` on the opportunity's CPV codes: a code must match a `cpv:` term (a code prefix by `LIKE`, or words in `classification_codes_fts`) and no `!cpv:` term. With only `!cpv:` terms the clause is `NOT EXISTS` a code matching any of them, so an opportunity with no CPV codes passes it.
- `OpportunityFilter.Title` searches only the `title` column of the same index.