Added unit tests, documentation and function descriptions across the board. Increased cyclop complexity threshold to 15 which is a bit more sane than 10 and excluded _test.go files from linters.
This commit is contained in:
bwilliams committed
1 parent
005b335371
commit
7fd321df97
31 files changed
+4106
-1
No files matched your search
@@ -3,6 +3,159 @@
|
||||
JobSearch is a service for scraping for contract opportunities. It is primarily
|
||||
a re-write of the existing nodeJS JobSearch codebase.
|
||||
|
||||
It currently fetches releases from the UK Government
|
||||
[Find a Tender](https://www.find-tender.service.gov.uk/) service (FTS), which
|
||||
publishes them in the [OCDS](https://standard.open-contracting.org/) format.
|
||||
A mock of the FTS API is included for development.
|
||||
|
||||
|
||||
## Building
|
||||
```bash
|
||||
scripts/build.sh # build/job-search-<version>
|
||||
scripts/build.sh --release # also package release/job-search-<version>-<arch>.tar.gz
|
||||
```
|
||||
|
||||
The build script runs `go generate ./...` and stamps the version from the
|
||||
latest git tag. A build from untagged commits is marked as a development build
|
||||
and isn't packaged unless `--force` is also given.
|
||||
|
||||
During development, `go run ./cmd/jobsearch` works as well.
|
||||
|
||||
|
||||
## Configuration
|
||||
Configuration is read from, in increasing priority:
|
||||
1. Built-in defaults
|
||||
2. `config.yaml` in `/etc/job-search` or the working directory, or the file
|
||||
given with `--config`
|
||||
3. `JOBSEARCH_*` environment variables, e.g. `JOBSEARCH_LOGGING_LEVEL=debug`
|
||||
|
||||
A `.env` file in the working directory is loaded into the environment first,
|
||||
and must exist.
|
||||
|
||||
Example `config.yaml`:
|
||||
```yaml
|
||||
logging:
|
||||
level: info
|
||||
file: /var/log/job-search.log
|
||||
sources:
|
||||
find_a_tender:
|
||||
enabled: true
|
||||
interval: 10m
|
||||
```
|
||||
|
||||
See [internal/config](internal/config/README.md) for every key and how the
|
||||
configuration code works.
|
||||
|
||||
|
||||
## Usage
|
||||
Global flags: `--config`/`-c` (config file), `--debug`/`-d` and
|
||||
`--quiet`/`-q` (errors only).
|
||||
|
||||
| Command | Purpose |
|
||||
|---|---|
|
||||
| `job-search fetch service` | Poll every enabled source on its interval until stopped (Ctrl+C or SIGTERM) |
|
||||
| `job-search fetch once` | Fetch from Find a Tender once and print the releases as JSON |
|
||||
| `job-search config init [path]` | Create a config file |
|
||||
| `job-search config show` | Print the configuration (`--json` or `--yaml` for those formats) |
|
||||
| `job-search config get <key>` | Print one value, e.g. `logging.level` |
|
||||
| `job-search config set <key> <value>` | Validate and write one value to the config file |
|
||||
| `job-search config set --unset <key>` | Remove a value from the config file |
|
||||
| `job-search version` | Print version information |
|
||||
|
||||
`fetch service` logs to `logging.file` if it's set, or to the terminal with
|
||||
`--foreground`. Both fetch commands take `--since "YYYY-MM-DD[ hh:mm:ss]"`.
|
||||
`fetch once` can be limited with `--planning`, `--opportunities` or
|
||||
`--awards`, and with `--ignore-rate-limit` it fails on a rate limit instead
|
||||
of waiting.
|
||||
|
||||
|
||||
## Mock Find a Tender server
|
||||
To develop without using the real API, run the mock and enable the `fts_mock`
|
||||
source:
|
||||
```bash
|
||||
go run ./cmd/mock-fts --min-gap 5s --max-gap 10s --rate-limit 1000
|
||||
```
|
||||
```yaml
|
||||
sources:
|
||||
fts_mock:
|
||||
enabled: true
|
||||
interval: 1m
|
||||
api_root: http://localhost:8080/api/1.0
|
||||
```
|
||||
|
||||
See [internal/mock](internal/mock/README.md) for its options and behaviour.
|
||||
|
||||
|
||||
## Project layout
|
||||
|
||||
| Path | Contents |
|
||||
|---|---|
|
||||
| `cmd/jobsearch` | The `job-search` command |
|
||||
| `cmd/mock-fts` | The mock FTS server |
|
||||
| `internal/cli` | Cobra commands and shared CLI setup |
|
||||
| `internal/config` | Configuration loading, validation and editing |
|
||||
| `internal/fetcher` | Runs a worker per enabled source |
|
||||
| `internal/findATender` | FTS API client and conversion from OCDS to JobSearch types |
|
||||
| `internal/types` | JobSearch's own opportunity types |
|
||||
| `internal/util` | The periodic `Worker` used by the fetcher |
|
||||
| `internal/logging` | slog setup and the HTTP access log |
|
||||
| `internal/mock` | Mock external services |
|
||||
| `internal/version` | Version information set at build time |
|
||||
|
||||
|
||||
## Testing
|
||||
```bash
|
||||
go test -race ./internal/...
|
||||
go test -coverprofile=cover.out ./internal/... && go tool cover -func=cover.out
|
||||
```
|
||||
|
||||
Tests don't need network access; HTTP calls go to local `httptest` servers.
|
||||
|
||||
|
||||
## Addition of a new opportunities source
|
||||
Using a source called `example` as an illustration:
|
||||
|
||||
1. **Configuration section:** create `internal/config/conf_source_example.go`,
|
||||
copying an existing source such as `conf_source_ftsMock.go`:
|
||||
- Add a `JSCKSourceExample = "example"` constant.
|
||||
- Define `JobSearchConfigSourceExample`, embedding `JobSearchConfigSource`
|
||||
(for `enabled` and `interval`), plus any fields of its own with `cfg` and
|
||||
optional `default` tags, e.g. ``APIRoot string `cfg:"api_root"` ``.
|
||||
- Copy the standard `Map`, `Flatten`, `TypeOf`, `ValueOf` and `SetValue`
|
||||
methods, renaming the receiver type and using the new constant in
|
||||
`Flatten`.
|
||||
2. **Register the section:** add
|
||||
``Example JobSearchConfigSourceExample `cfg:"example"` `` to
|
||||
`JobSearchConfigSources` in `internal/config/conf_sources.go`. The `cfg` tag
|
||||
must match the constant. The source's keys (`sources.example.*`) are then
|
||||
loaded, validated and available to the `config` commands with no further
|
||||
code.
|
||||
3. **Client package:** create a package that fetches from the source and
|
||||
returns `[]*types.Opportunity` (see `internal/types`). If the source
|
||||
publishes OCDS, `internal/findATender` may already be suitable, as it is
|
||||
for `fts_mock`: create it with the source's own API root.
|
||||
4. **Worker:** create `internal/fetcher/worker_example.go`, following
|
||||
`worker_ftsMock.go`:
|
||||
- `runExample(ctx, conf, since)` builds a `util.Worker` with the source's
|
||||
name, `conf.Sources.Example.Interval` and a task, then calls `Run`.
|
||||
- The task fetches everything since its `since` argument. Wrap blocking
|
||||
calls in `util.AwaitTask` so shutdown isn't held up, and return
|
||||
`ErrNoNewReleases` when nothing is found, so the checkpoint still moves
|
||||
forward. Any other error makes the next run retry from the same point.
|
||||
5. **Start it:** in `Run` in `internal/fetcher/fetcher.go`, add
|
||||
```go
|
||||
if conf.Sources.Example.Enabled {
|
||||
start("Example", runExample)
|
||||
}
|
||||
```
|
||||
6. **Tests:** add tests for the new config section to
|
||||
`internal/config/conf_sections_test.go` (add it to
|
||||
`TestConcreteSourceSections` and update the key count in
|
||||
`TestSourcesSection`), and for the client and worker, using `httptest`
|
||||
servers rather than the real service.
|
||||
7. **Docs:** add the source and its keys to the key table in
|
||||
`internal/config/README.md`.
|
||||
|
||||
|
||||
## Installation of dev tools
|
||||
A number of tools have been used in the development of the project. Installation
|
||||
|
||||
Reference in new issue
Block a user