Made the fetcher a Fetcher type with per-source config, and ran it in job-search serve

fetcher.Run(ctx, conf, since) becomes New(&conf.Sources, opts...) and
Run(ctx, since). Each worker now takes only its own source's config, and
Run starts each one through a closure that binds it. Cancelling ctx stops
the workers through the Fetcher's background context, and Run always waits
for every worker before returning their joined errors.

The findatender client no longer holds the configuration: New takes only
the API root, ReloadConfig is gone, and request tracing follows
logging.IsDebugEnabled(). The FindAOpportunity worker is now FindATender.

job-search serve runs the fetcher alongside the HTTP server. A source that
fails stops the server, and the server stopping stops the sources; with no
sources enabled the server keeps running. fetcher service and fetcher
fts-once use the new APIs.
This commit is contained in:
bwilliams committed 2026-10-16 10:44:38 +01:00
1 parent f0c171c8a8
commit 18db462d89
12 files changed
+269 -214

No files matched your search

+1 -12
View File
@@ -7,7 +7,6 @@ import (
resty "resty.dev/v3"
"git.avsdev.uk/Go/job-search/internal/config"
"git.avsdev.uk/Go/job-search/internal/types"
)
@@ -15,7 +14,6 @@ import (
// server). Create it with New and call Shutdown when finished with it.
type FindATender struct {
httpClient *resty.Client
config *config.JobSearchConfig
apiRoot string
ctx context.Context
log *slog.Logger
@@ -34,11 +32,8 @@ func WithLogger(logger *slog.Logger) Option {
// New returns a client for the OCDS API at apiRoot. Requests time out after
// 60 seconds and are retried once. Call Shutdown when finished with it. It
// logs to slog.Default() unless given WithLogger.
func New(
conf *config.JobSearchConfig, apiRoot string, opts ...Option,
) *FindATender {
func New(apiRoot string, opts ...Option) *FindATender {
findATender := &FindATender{
config: conf,
apiRoot: apiRoot,
}
for _, opt := range opts {
@@ -61,12 +56,6 @@ func (ft *FindATender) logger() *slog.Logger {
return log.With("component", "findatender")
}
// ReloadConfig replaces the configuration the client reads, such as the log
// level. The HTTP client and API root set by New are not changed.
func (ft *FindATender) ReloadConfig(conf *config.JobSearchConfig) {
ft.config = conf
}
// With returns a copy of the client whose requests use ctx, so they stop
// when it is cancelled. A nil ctx returns the client unchanged.
func (ft *FindATender) With(ctx context.Context) *FindATender {