Made the fetcher save what it fetches, with fetcher.WithStore
With WithStore, the Find a Tender and mock workers save each run's opportunities with SaveOpportunities and log what changed; a failed save fails the run, so the checkpoint stays put and the next run retries. Without a store nothing is persisted. The mock's opportunities are saved with Source "FTSMock" rather than the converter's "FindATender". fetcher service and job-search serve open the store at database.file (cli.OpenConfiguredStore), or warn and run without one. A missing file isn't created, and a database needing migration says to run job-search migrate. Fixed upsertOpportunity, which never wrote the source column, so every saved opportunity had an empty Source.
This commit is contained in:
1 parent
eba6af680f
commit
b3d20c1db0
10 files changed
+407
-38
No files matched your search
@@ -29,9 +29,10 @@ func NewCommand(opts *cli.Options) *cobra.Command {
|
||||
Short: "Serve the REST API (/api) and WebSocket (/ws), and fetch from sources",
|
||||
Long: `Serve the REST API under /api and the WebSocket endpoint under /ws.
|
||||
|
||||
It also runs a worker for every enabled source, as fetcher service does. A
|
||||
source that stops with an error stops the server too, and the server
|
||||
stopping stops the sources.`,
|
||||
It also runs a worker for every enabled source, as fetcher service does,
|
||||
saving what they fetch to database.file if it is set. A source that stops
|
||||
with an error stops the server too, and the server stopping stops the
|
||||
sources.`,
|
||||
PersistentPreRun: func(cmd *cobra.Command, args []string) {
|
||||
slog.Debug("command persistent pre-run", "command", "serve")
|
||||
},
|
||||
@@ -118,7 +119,23 @@ stopping stops the sources.`,
|
||||
runCtx, cancelRun := context.WithCancel(ctx)
|
||||
defer cancelRun()
|
||||
|
||||
sourceFetcher := fetcher.New(&conf.Sources)
|
||||
var fetcherOptions []fetcher.Option
|
||||
opportunityStore, openErr := cli.OpenConfiguredStore(ctx, conf)
|
||||
switch {
|
||||
case errors.Is(openErr, cli.ErrNoDatabase):
|
||||
slog.Warn("not storing fetched opportunities", "reason", openErr)
|
||||
case openErr != nil:
|
||||
return openErr
|
||||
default:
|
||||
defer func() {
|
||||
err = errors.Join(err, opportunityStore.Close())
|
||||
}()
|
||||
fetcherOptions = append(
|
||||
fetcherOptions, fetcher.WithStore(opportunityStore),
|
||||
)
|
||||
}
|
||||
|
||||
sourceFetcher := fetcher.New(&conf.Sources, fetcherOptions...)
|
||||
fetcherDone := make(chan error, 1)
|
||||
go func() {
|
||||
fetchErr := sourceFetcher.Run(runCtx, time.Time{})
|
||||
|
||||
Reference in new issue
Block a user