Added skip rules, which mark new opportunities Skipped when they're fetched

sources.skip_rules_file names a JSON file of filters, each with a
comment, an active flag, a logic mode ("or" by default, or "and") and
rules by buyer, title words or CPV codes. The fetcher applies them
before saving, so only new opportunities are marked; the marks of
stored ones are never touched. CPV rules match when the opportunity's
codes are all in the list, or any of them with exclusive false, and
by prefix with exact false.

The file is read again when its time or size changes: an invalid edit
is logged and the last good rules kept, and a deleted file means no
rules until it's back. fetcher service and serve refuse to start if it
is define but missing or invalid.

job-search skip-rules check validates a rules file, and skip-rules
apply marks the stored opportunities that match Skipped, or only lists
them with --dry-run.
This commit is contained in:
bwilliams committed 2026-10-20 08:55:19 +01:00
1 parent 2e6c21f9dd
commit 0543762330
21 files changed
+1766 -8

No files matched your search

+11
View File
@@ -82,6 +82,8 @@ Global flags: `--config`/`-c` (config file), `--debug`/`-d` and
| `job-search migrate up` | Apply pending migrations (`--steps N` for only N; `--create` to create a missing database file without asking) |
| `job-search migrate down --steps N` | Roll back the last N migrations (`--steps 0` rolls back all of them, deleting all data, and needs `--yes`) |
| `job-search migrate status` | Show the database version and which migrations are applied or pending |
| `job-search skip-rules check [file]` | Check a skip rules file (default: `sources.skip_rules_file`) |
| `job-search skip-rules apply [file]` | Mark the stored opportunities that match the skip rules Skipped, listing them (`--dry-run` only lists them) |
| `job-search version` | Print version information |
`job-search serve` describes its REST API in OpenAPI 3.1 at
@@ -93,6 +95,14 @@ The `migrate` commands work on the database in `database.file`. The store
refuses to open a database that isn't at this build's schema version, so run
`job-search migrate` after installing a new version.
### Skip rules
`sources.skip_rules_file` names a JSON file of rules that mark new
opportunities Skipped as `fetcher service` and `job-search serve` fetch them,
by buyer, title words or CPV codes. The file is read again when it changes;
an invalid edit is logged and the last good rules kept. Opportunities already
stored are only changed by `job-search skip-rules apply`. See
[internal/skiprules](internal/skiprules/README.md) for the format.
## Fetch tool
`fetcher` reads the same configuration as `job-search`:
@@ -164,6 +174,7 @@ second one fails with "address already in use".
| `internal/cli` | Cobra commands and shared CLI setup |
| `internal/config` | Configuration loading, validation and editing |
| `internal/fetcher` | Runs a worker per enabled source |
| `internal/skiprules` | The rules that mark new opportunities Skipped (see [internal/skiprules](internal/skiprules/README.md)) |
| `internal/store` | SQLite storage of opportunities, and its migrations (see [internal/store](internal/store/README.md)) |
| `internal/findatender` | FTS API client and conversion from OCDS to JobSearch types |
| `internal/types` | JobSearch's own opportunity types |