Added the opportunities REST API, read-only apart from their marks, described in OpenAPI 3.1 with swag

GET /api/opportunities lists the stored opportunities, newest release first,
a page at a time (offset, and limit up to 100, default 50); has_more says
whether another page follows. GET /api/opportunities/{uuid} returns one with
its releases. PUT and DELETE /api/opportunities/{uuid}/marks/{mark} set and
clear one mark (hidden, skipped, interested, bid_submitted, won, lost) and
return them all; won and lost exclude each other (409). The API uses the
types in internal/types, and changing a mark is made atomic, as the store
replaces all of an opportunity's marks at once. The marks are found by
reflection: every bool field of types.OpportunityMarks, by its JSON name, so
a new mark needs no API change.

The API has one entry point, api.New, with its dependencies as options
(api.WithStore), and the server only mounts it under /api, keeping the
middleware. New resources are added inside internal/api. job-search serve
passes server.WithStore when database.file is set, opening the store before
the server; without one there are no opportunity routes.

The API is described by swag v2 annotations on the handlers. go generate
writes the OpenAPI 3.1 description (git-ignored, like the OCDS types), which
the server embeds and serves at /api/openapi.json, with Swagger UI at
/api/docs. A test checks the description and the routes match. swag is in
the README's dev tools, pinned to v2.0.0-rc6, and the build now needs it.
This commit is contained in:
bwilliams committed 2026-10-16 13:13:44 +01:00
1 parent 6e101d2b4d
commit 70d62f280e
14 files changed
+992 -16

No files matched your search

+13 -3
View File
@@ -15,8 +15,9 @@ scripts/build.sh # build/job-search-<version>
scripts/build.sh --release # also package release/job-search-<version>-<arch>.tar.gz scripts/build.sh --release # also package release/job-search-<version>-<arch>.tar.gz
``` ```
The build script runs `go generate ./...` and stamps the version from the The build script runs `go generate ./...`, which needs `swag` installed (see
latest git tag. A build from untagged commits is marked as a development build [Installation of dev tools](#installation-of-dev-tools)), and stamps the
version from the latest git tag. A build from untagged commits is marked as a development build
and isn't packaged unless `--force` is also given. and isn't packaged unless `--force` is also given.
During development, `go run -tags sqlite_fts5 ./cmd/jobsearch` works as well. During development, `go run -tags sqlite_fts5 ./cmd/jobsearch` works as well.
@@ -69,6 +70,11 @@ Global flags: `--config`/`-c` (config file), `--debug`/`-d` and
| `job-search migrate status` | Show the database version and which migrations are applied or pending | | `job-search migrate status` | Show the database version and which migrations are applied or pending |
| `job-search version` | Print version information | | `job-search version` | Print version information |
`job-search serve` describes its REST API in OpenAPI 3.1 at
`/api/openapi.json`, and shows it in Swagger UI at `/api/docs`. The
description is generated from the handlers' annotations in `internal/api` by
`go generate`.
The `migrate` commands work on the database in `database.file`. The store The `migrate` commands work on the database in `database.file`. The store
refuses to open a database that isn't at this build's schema version, so run refuses to open a database that isn't at this build's schema version, so run
`job-search migrate` after installing a new version. `job-search migrate` after installing a new version.
@@ -234,11 +240,15 @@ go install github.com/spf13/cobra-cli@latest
go install git.avsdev.uk/Go/go-ocds/cmd/ocdsgen@latest go install git.avsdev.uk/Go/go-ocds/cmd/ocdsgen@latest
go install github.com/pressly/goose/v3/cmd/goose@latest go install github.com/pressly/goose/v3/cmd/goose@latest
go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest
go install github.com/swaggo/swag/v2/cmd/swag@v2.0.0-rc6
``` ```
`goose` creates new migration files (see `internal/store`); the migrations `goose` creates new migration files (see `internal/store`); the migrations
themselves are applied by `job-search migrate`. `golangci-lint` uses the themselves are applied by `job-search migrate`. `golangci-lint` uses the
project's `.golangci.json`. project's `.golangci.json`. `swag` generates the API's OpenAPI 3.1 spec from
the handlers' comments (`swag init --v3.1`). Its v2, the version with OpenAPI
3 support, has only had release candidates so far, so it is pinned to one
rather than `@latest`.
Ensure that the GOBIN path ($HOME/go/bin) is added to the environment before Ensure that the GOBIN path ($HOME/go/bin) is added to the environment before
running any of the tools. running any of the tools.
+1
View File
@@ -0,0 +1 @@
openapi/
+45
View File
@@ -0,0 +1,45 @@
package api
import (
"github.com/go-chi/chi/v5"
"git.avsdev.uk/Go/job-search/internal/store"
)
// API is the REST API, mounted under /api. Create it with New, giving it what
// its resources need as options, and register its routes with Routes. The
// server applies the API's middleware (CSRF, NoStore, RequireJSON, rate
// limits); the API only registers handlers.
type API struct {
store *store.Store // nil: no opportunity routes
}
// Option configures the API.
type Option func(*API)
// WithStore makes the API serve the opportunities in opportunityStore under
// /opportunities. Without it there are no opportunity routes.
func WithStore(opportunityStore *store.Store) Option {
return func(a *API) {
a.store = opportunityStore
}
}
// New returns the API.
func New(opts ...Option) *API {
api := &API{}
for _, option := range opts {
option(api)
}
return api
}
// Routes registers the API's routes on router (mounted under /api): its
// description (see openapi.go), and the opportunities if it has a store.
func (a *API) Routes(router chi.Router) {
docsRoutes(router)
if a.store != nil {
newOpportunityRoutes(a.store).Routes(router)
}
}
+23
View File
@@ -0,0 +1,23 @@
package api
import (
"fmt"
"strconv"
)
// intParam parses the query parameter text as an integer from low to high,
// or returns fallback if it is empty. Its error is meant for the client.
func intParam(text string, fallback, low, high int) (int, error) {
if text == "" {
return fallback, nil
}
number, err := strconv.Atoi(text)
if err != nil || number < low || number > high {
return 0, fmt.Errorf(
"parameter must be an integer between %d and %d", low, high,
)
}
return number, nil
}
+24
View File
@@ -0,0 +1,24 @@
// Package api implements the REST API mounted under /api. New assembles it
// (api.go): the opportunities, read-only apart from their marks
// (opportunities.go), and the API's own OpenAPI 3.1 description and docs
// viewer (openapi.go). New resources are added in New and Routes, so the
// server only mounts the API.
//
// The description is generated from the swag annotations on this package's
// handlers by go generate (swag v2, see the README's dev tools), into
// openapi/swagger.json. The annotations below are the API's general
// information.
//
// @title JobSearch API
// @version 1.0
// @description Procurement opportunities fetched by JobSearch, and the user's
// @description marks on them. Errors are RFC 9457 problem details.
// @servers.url /api
//
// @securityDefinitions.apikey CSRF
// @in header
// @name X-CSRF-Token
// @description From GET /api/csrf; send its cookie too.
package api
//go:generate swag init --v3.1 --generalInfo doc.go --dir . --parseInternal --parseDependencyLevel 1 --outputTypes json --output openapi
+54
View File
@@ -0,0 +1,54 @@
package api
import (
_ "embed"
"net/http"
"github.com/go-chi/chi/v5"
)
// openAPISpec is the API's OpenAPI 3.1 description, generated by go generate
// (see doc.go).
//
//go:embed openapi/swagger.json
var openAPISpec []byte
// docsPage shows openapi.json, relative to /api/docs, in Swagger UI, loaded
// from jsDelivr at a fixed version. A content security policy
// (server.headers.content_security_policy) must allow that.
const docsPage = `<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>JobSearch API</title>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5.17.14/swagger-ui.css">
</head>
<body>
<div id="ui"></div>
<script src="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5.17.14/swagger-ui-bundle.js"></script>
<script>
SwaggerUIBundle({url: "openapi.json", dom_id: "#ui"});
</script>
</body>
</html>
`
// docsRoutes registers the API's description on router: GET /openapi.json,
// the OpenAPI 3.1 description, and GET /docs, a viewer of it.
func docsRoutes(router chi.Router) {
router.Get("/openapi.json", serveOpenAPISpec)
router.Get("/docs", serveDocsPage)
}
// serveOpenAPISpec serves GET /openapi.json.
func serveOpenAPISpec(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.Write(openAPISpec) //nolint:errcheck,gosec // the client may have gone
}
// serveDocsPage serves GET /docs.
func serveDocsPage(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.Write([]byte(docsPage)) //nolint:errcheck,gosec // the client may have gone
}
+136
View File
@@ -0,0 +1,136 @@
package api
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/go-chi/chi/v5"
"git.avsdev.uk/Go/job-search/internal/store"
)
// The description is served as JSON, and the docs page points at it.
func TestDocsRoutes(t *testing.T) {
router := chi.NewRouter()
New().Routes(router)
server := httptest.NewServer(router)
t.Cleanup(server.Close)
res, err := http.Get(server.URL + "/openapi.json")
if err != nil {
t.Fatal(err)
}
defer res.Body.Close()
var spec struct {
OpenAPI string `json:"openapi"`
}
if err := json.NewDecoder(res.Body).Decode(&spec); err != nil {
t.Fatal(err)
}
if res.Header.Get("Content-Type") != "application/json" ||
spec.OpenAPI != "3.1.0" {
t.Errorf(
"openapi.json: %s, openapi %q; want JSON, 3.1.0",
res.Header.Get("Content-Type"), spec.OpenAPI,
)
}
res, err = http.Get(server.URL + "/docs")
if err != nil {
t.Fatal(err)
}
defer res.Body.Close()
page, _ := io.ReadAll(res.Body)
if !strings.Contains(string(page), `url: "openapi.json"`) {
t.Errorf("docs page doesn't load openapi.json:\n%s", page)
}
}
// Every route the API serves, apart from its description's own, is in the
// description, and the description has no route that isn't served, so the
// annotations can't fall behind as resources are added.
func TestOpenAPICoversRoutes(t *testing.T) {
var spec struct {
Paths map[string]map[string]json.RawMessage `json:"paths"`
}
if err := json.Unmarshal(openAPISpec, &spec); err != nil {
t.Fatal(err)
}
documented := map[string]bool{}
for path, operations := range spec.Paths {
for method := range operations {
documented[strings.ToUpper(method)+" "+path] = true
}
}
// A store that is never opened: registering routes doesn't use it.
router := chi.NewRouter()
New(WithStore(&store.Store{})).Routes(router)
description := map[string]bool{
"GET /openapi.json": true,
"GET /docs": true,
}
served := map[string]bool{}
err := chi.Walk(router, func(
method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler,
) error {
if !description[method+" "+route] {
served[method+" "+route] = true
}
return nil
})
if err != nil {
t.Fatal(err)
}
for route := range served {
if !documented[route] {
t.Errorf("%s is served but not in the OpenAPI description", route)
}
}
for route := range documented {
if !served[route] {
t.Errorf("%s is in the OpenAPI description but not served", route)
}
}
}
// The description's list of marks matches OpportunityMarks, so its Enums
// annotation can't fall behind the struct.
func TestOpenAPIMarkEnum(t *testing.T) {
var spec struct {
Paths map[string]map[string]struct {
Parameters []struct {
Name string `json:"name"`
Schema struct {
Enum []string `json:"enum"`
} `json:"schema"`
} `json:"parameters"`
} `json:"paths"`
}
if err := json.Unmarshal(openAPISpec, &spec); err != nil {
t.Fatal(err)
}
want := strings.Join(markNames, ",")
checked := 0
operations := spec.Paths["/opportunities/{uuid}/marks/{mark}"]
for method, operation := range operations {
for _, parameter := range operation.Parameters {
if parameter.Name != "mark" {
continue
}
checked++
if got := strings.Join(parameter.Schema.Enum, ","); got != want {
t.Errorf("%s mark enum = %s; want %s", method, got, want)
}
}
}
if checked != 2 {
t.Errorf("checked the mark of %d routes; want 2 (PUT, DELETE)", checked)
}
}
+267
View File
@@ -0,0 +1,267 @@
package api
import (
"errors"
"fmt"
"net/http"
"reflect"
"strings"
"sync"
"github.com/go-chi/chi/v5"
"git.avsdev.uk/Go/job-search/internal/httpx"
"git.avsdev.uk/Go/job-search/internal/store"
"git.avsdev.uk/Go/job-search/internal/types"
)
// markIndex maps the name of each mark that can be set and cleared, as named
// in URLs, to the index of its field in types.OpportunityMarks. markNames
// lists the names in field order. Both are read from the struct (see
// findMarks), so a mark added to it can be changed over the API.
var markIndex, markNames = findMarks()
// OpportunityPage is a page of opportunities, as GET /opportunities returns
// it. HasMore reports that another page follows, at Offset+Limit.
type OpportunityPage struct {
Opportunities []types.Opportunity `json:"opportunities"`
Offset int `json:"offset"`
Limit int `json:"limit"`
HasMore bool `json:"has_more"`
}
// opportunityRoutes wires HTTP handlers to the opportunity store.
// Opportunities are read-only; only their marks can be changed.
type opportunityRoutes struct {
store *store.Store
// marksMu makes changing one mark (read the marks, change one, write
// them all back) atomic, so concurrent changes to different marks of an
// opportunity don't undo each other.
marksMu sync.Mutex
}
// newOpportunityRoutes returns the opportunity handlers for
// opportunityStore.
func newOpportunityRoutes(opportunityStore *store.Store) *opportunityRoutes {
return &opportunityRoutes{
store: opportunityStore,
}
}
// Routes registers the opportunity routes on router (mounted under /api).
func (h *opportunityRoutes) Routes(router chi.Router) {
router.Get("/opportunities", h.list)
router.Get("/opportunities/{uuid}", h.get)
router.Put("/opportunities/{uuid}/marks/{mark}", h.setMark)
router.Delete("/opportunities/{uuid}/marks/{mark}", h.clearMark)
}
// list serves GET /opportunities: a page of opportunities, newest release
// first, with their classifications and marks but not their releases. The
// page is set by the offset and limit (default 50, at most 100) query
// parameters.
//
// @Summary List opportunities
// @Description A page of opportunities, newest release first, with their
// @Description classifications and marks but not their releases. has_more
// @Description says whether another page follows, at offset+limit.
// @Tags opportunities
// @Produce json
// @Param offset query int false "Opportunities to skip" minimum(0) default(0)
// @Param limit query int false "Page size" minimum(1) maximum(100) default(50)
// @Success 200 {object} OpportunityPage
// @Failure 400 {object} httpx.Problem "Invalid offset or limit"
// @Failure 500 {object} httpx.Problem
// @Router /opportunities [get]
func (h *opportunityRoutes) list(w http.ResponseWriter, r *http.Request) {
query := r.URL.Query()
offset, offsetErr := intParam(query.Get("offset"), 0, 0, 1<<30)
limit, limitErr := intParam(query.Get("limit"), 50, 1, 100)
err := errors.Join(offsetErr, limitErr)
if err != nil {
httpx.WriteProblem(w, r, http.StatusBadRequest, err.Error())
return
}
// One more than the page, to tell whether another page follows.
opportunities, err := h.store.ListOpportunities(
r.Context(),
store.OpportunityFilter{
Limit: limit + 1,
Offset: offset,
},
)
if err != nil {
httpx.WriteProblem(w, r, http.StatusInternalServerError, "")
return
}
page := OpportunityPage{
Opportunities: opportunities,
Offset: offset,
Limit: limit,
HasMore: len(opportunities) > limit,
}
if page.HasMore {
page.Opportunities = opportunities[:limit]
}
httpx.WriteJSON(w, http.StatusOK, page)
}
// get serves GET /opportunities/{uuid}: one opportunity with its
// classifications, marks and releases.
//
// @Summary Get an opportunity
// @Description One opportunity with its classifications, marks and all of its
// @Description releases.
// @Tags opportunities
// @Produce json
// @Param uuid path string true "The opportunity's UUID (its OCID)"
// @Success 200 {object} types.Opportunity
// @Failure 404 {object} httpx.Problem "No such opportunity"
// @Failure 500 {object} httpx.Problem
// @Router /opportunities/{uuid} [get]
func (h *opportunityRoutes) get(w http.ResponseWriter, r *http.Request) {
opportunity, err := h.store.GetOpportunity(
r.Context(), chi.URLParam(r, "uuid"),
)
if err != nil {
h.fail(w, r, err)
return
}
httpx.WriteJSON(w, http.StatusOK, opportunity)
}
// setMark serves PUT /opportunities/{uuid}/marks/{mark}, setting the mark.
//
// @Summary Set a mark
// @Description Sets one of the opportunity's marks, keeping the others, and
// @Description returns all of them. Setting a mark that is already set
// @Description changes nothing. won and lost exclude each other.
// @Tags marks
// @Produce json
// @Security CSRF
// @Param uuid path string true "The opportunity's UUID (its OCID)"
// @Param mark path string true "The mark" Enums(hidden, skipped, interested, bid_submitted, won, lost)
// @Success 200 {object} types.OpportunityMarks
// @Failure 403 {object} httpx.Problem "No valid CSRF token"
// @Failure 404 {object} httpx.Problem "No such opportunity or mark"
// @Failure 409 {object} httpx.Problem "won and lost together"
// @Failure 500 {object} httpx.Problem
// @Router /opportunities/{uuid}/marks/{mark} [put]
func (h *opportunityRoutes) setMark(w http.ResponseWriter, r *http.Request) {
h.changeMark(w, r, true)
}
// clearMark serves DELETE /opportunities/{uuid}/marks/{mark}, clearing the
// mark.
//
// @Summary Clear a mark
// @Description Clears one of the opportunity's marks, keeping the others, and
// @Description returns all of them. Clearing a mark that isn't set changes
// @Description nothing.
// @Tags marks
// @Produce json
// @Security CSRF
// @Param uuid path string true "The opportunity's UUID (its OCID)"
// @Param mark path string true "The mark" Enums(hidden, skipped, interested, bid_submitted, won, lost)
// @Success 200 {object} types.OpportunityMarks
// @Failure 403 {object} httpx.Problem "No valid CSRF token"
// @Failure 404 {object} httpx.Problem "No such opportunity or mark"
// @Failure 500 {object} httpx.Problem
// @Router /opportunities/{uuid}/marks/{mark} [delete]
func (h *opportunityRoutes) clearMark(w http.ResponseWriter, r *http.Request) {
h.changeMark(w, r, false)
}
// changeMark sets the mark named in the URL to value, keeping the
// opportunity's other marks, and responds with all of its marks. Setting won
// while lost is set, or lost while won is, is refused with 409 Conflict.
func (h *opportunityRoutes) changeMark(
w http.ResponseWriter, r *http.Request, value bool,
) {
uuid, name := chi.URLParam(r, "uuid"), chi.URLParam(r, "mark")
h.marksMu.Lock()
defer h.marksMu.Unlock()
opportunity, err := h.store.GetOpportunity(r.Context(), uuid)
if err != nil {
h.fail(w, r, err)
return
}
marks := opportunity.Marks
marks.UUID = uuid
if !setMark(&marks, name, value) {
httpx.WriteProblem(
w,
r,
http.StatusNotFound,
fmt.Sprintf("no mark %q: marks are %v", name, markNames),
)
return
}
err = h.store.SetMarks(r.Context(), marks)
if err != nil {
h.fail(w, r, err)
return
}
httpx.WriteJSON(w, http.StatusOK, marks)
}
// findMarks returns the marks of types.OpportunityMarks: every bool field,
// named by its JSON tag, mapped to the field's index, and the names in field
// order.
func findMarks() (map[string]int, []string) {
index := map[string]int{}
var names []string
marksType := reflect.TypeFor[types.OpportunityMarks]()
for i := range marksType.NumField() {
field := marksType.Field(i)
if field.Type.Kind() != reflect.Bool {
continue
}
name, _, _ := strings.Cut(field.Tag.Get("json"), ",")
index[name] = i
names = append(names, name)
}
return index, names
}
// setMark sets the mark of marks named name (see markIndex) to value, and
// reports whether name is a mark.
func setMark(marks *types.OpportunityMarks, name string, value bool) bool {
i, ok := markIndex[name]
if !ok {
return false
}
reflect.ValueOf(marks).Elem().Field(i).SetBool(value)
return true
}
// fail writes the problem response for a store error.
func (h *opportunityRoutes) fail(
w http.ResponseWriter, r *http.Request, err error,
) {
switch {
case errors.Is(err, store.ErrNotFound):
httpx.WriteProblem(w, r, http.StatusNotFound, "opportunity not found")
case errors.Is(err, types.ErrWonAndLost):
httpx.WriteProblem(
w,
r,
http.StatusConflict,
"an opportunity cannot be both won and lost",
)
default:
httpx.WriteProblem(w, r, http.StatusInternalServerError, "")
}
}
+304
View File
@@ -0,0 +1,304 @@
package api
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/go-chi/chi/v5"
"git.avsdev.uk/Go/job-search/internal/store"
"git.avsdev.uk/Go/job-search/internal/types"
)
// opportunitiesServer serves the opportunity routes over a store holding
// count opportunities, ocds-1 the oldest, and returns its URL.
func opportunitiesServer(t *testing.T, count int) string {
t.Helper()
opportunityStore, err := store.Open(
context.Background(),
filepath.Join(t.TempDir(), "api.db"),
store.WithAutoMigrate(),
)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
opportunityStore.Close()
})
for i := 1; i <= count; i++ {
opportunity := &types.Opportunity{
UUID: fmt.Sprintf("ocds-%d", i),
UUIDType: "ocid",
Title: fmt.Sprintf("Opportunity %d", i),
Source: "FindATender",
Releases: []types.OpportunityRelease{
{
ReleaseID: "r1",
ReleaseDate: time.Date(2026, 10, i, 12, 0, 0, 0, time.UTC),
JSON: "{}",
},
},
}
_, err := opportunityStore.SaveOpportunity(
context.Background(), opportunity,
)
if err != nil {
t.Fatal(err)
}
}
router := chi.NewRouter()
New(WithStore(opportunityStore)).Routes(router)
server := httptest.NewServer(router)
t.Cleanup(server.Close)
return server.URL
}
// request sends a bodyless request and decodes a JSON reply into reply (if
// not nil), returning the status.
func request(t *testing.T, method, url string, reply any) int {
t.Helper()
req, err := http.NewRequest(method, url, nil)
if err != nil {
t.Fatal(err)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
defer res.Body.Close()
if reply != nil && res.StatusCode < 300 {
if err := json.NewDecoder(res.Body).Decode(reply); err != nil {
t.Fatal(err)
}
}
return res.StatusCode
}
// uuids returns the UUIDs of a page's opportunities.
func uuids(page OpportunityPage) []string {
var ids []string
for _, opportunity := range page.Opportunities {
ids = append(ids, opportunity.UUID)
}
return ids
}
// Opportunities are listed newest first, a page at a time.
func TestListOpportunitiesPaging(t *testing.T) {
url := opportunitiesServer(t, 5)
all := []string{"ocds-5", "ocds-4", "ocds-3", "ocds-2", "ocds-1"}
tests := []struct {
query string
want []string
hasMore bool
}{
{"?limit=2", []string{"ocds-5", "ocds-4"}, true},
{"?limit=2&offset=2", []string{"ocds-3", "ocds-2"}, true},
{"?limit=2&offset=4", []string{"ocds-1"}, false},
{"?limit=5", all, false},
{"?offset=10", nil, false},
{"", all, false},
}
for _, tt := range tests {
var page OpportunityPage
status := request(t, "GET", url+"/opportunities"+tt.query, &page)
if status != 200 {
t.Fatalf("%s: status %d", tt.query, status)
}
got := strings.Join(uuids(page), ",")
want := strings.Join(tt.want, ",")
if got != want || page.HasMore != tt.hasMore {
t.Errorf(
"%s: got %s, has_more %v; want %s, %v",
tt.query, got, page.HasMore, want, tt.hasMore,
)
}
}
var page OpportunityPage
request(t, "GET", url+"/opportunities", &page)
if page.Limit != 50 || page.Offset != 0 {
t.Errorf(
"defaults: limit %d, offset %d; want 50, 0", page.Limit, page.Offset,
)
}
}
// An empty page is [] in JSON, not null.
func TestListOpportunitiesEmpty(t *testing.T) {
url := opportunitiesServer(t, 0)
res, err := http.Get(url + "/opportunities")
if err != nil {
t.Fatal(err)
}
defer res.Body.Close()
var raw map[string]json.RawMessage
json.NewDecoder(res.Body).Decode(&raw)
if string(raw["opportunities"]) != "[]" {
t.Errorf("opportunities = %s; want []", raw["opportunities"])
}
}
func TestListOpportunitiesBadPaging(t *testing.T) {
url := opportunitiesServer(t, 1)
queries := []string{"?limit=0", "?limit=101", "?limit=x", "?offset=-1"}
for _, query := range queries {
status := request(t, "GET", url+"/opportunities"+query, nil)
if status != 400 {
t.Errorf("%s: status %d; want 400", query, status)
}
}
}
// One opportunity comes with its releases; a missing one is 404.
func TestGetOpportunity(t *testing.T) {
url := opportunitiesServer(t, 2)
var opportunity types.Opportunity
status := request(t, "GET", url+"/opportunities/ocds-2", &opportunity)
if status != 200 {
t.Fatalf("status %d", status)
}
if opportunity.UUID != "ocds-2" || len(opportunity.Releases) != 1 {
t.Errorf("got %+v; want ocds-2 with its release", opportunity)
}
status = request(t, "GET", url+"/opportunities/nope", nil)
if status != 404 {
t.Errorf("missing: status %d; want 404", status)
}
}
// Opportunities are read-only: there is no way to write one.
func TestOpportunitiesReadOnly(t *testing.T) {
url := opportunitiesServer(t, 1)
for _, method := range []string{"POST", "PUT", "PATCH", "DELETE"} {
status := request(t, method, url+"/opportunities/ocds-1", nil)
if status != http.StatusMethodNotAllowed {
t.Errorf("%s: status %d; want 405", method, status)
}
}
if status := request(t, "POST", url+"/opportunities", nil); status != 405 {
t.Errorf("POST /opportunities: status %d; want 405", status)
}
}
// A mark is set with PUT and cleared with DELETE, leaving the others, and
// both reply with all the marks.
func TestChangeMark(t *testing.T) {
url := opportunitiesServer(t, 1)
marks := url + "/opportunities/ocds-1/marks/"
var got types.OpportunityMarks
request(t, "PUT", marks+"interested", &got)
request(t, "PUT", marks+"bid_submitted", &got)
if !got.Interested || !got.BidSubmitted || got.UUID != "ocds-1" {
t.Fatalf("after two PUTs: %+v", got)
}
request(t, "DELETE", marks+"interested", &got)
if got.Interested || !got.BidSubmitted {
t.Errorf("after DELETE: %+v; want only bid_submitted", got)
}
// The change is stored, and setting a set mark again is harmless.
request(t, "PUT", marks+"bid_submitted", &got)
var opportunity types.Opportunity
request(t, "GET", url+"/opportunities/ocds-1", &opportunity)
if !opportunity.Marks.BidSubmitted || opportunity.Marks.Interested {
t.Errorf("stored marks: %+v", opportunity.Marks)
}
}
// Won and lost exclude each other; unknown marks and opportunities are 404.
func TestChangeMarkRefused(t *testing.T) {
url := opportunitiesServer(t, 1)
marks := url + "/opportunities/ocds-1/marks/"
request(t, "PUT", marks+"won", nil)
status := request(t, "PUT", marks+"lost", nil)
if status != http.StatusConflict {
t.Errorf("lost while won: status %d; want 409", status)
}
status = request(t, "PUT", marks+"shiny", nil)
if status != 404 {
t.Errorf("unknown mark: status %d; want 404", status)
}
status = request(t, "PUT", url+"/opportunities/nope/marks/won", nil)
if status != 404 {
t.Errorf("unknown opportunity: status %d; want 404", status)
}
// Clearing won makes lost allowed.
request(t, "DELETE", marks+"won", nil)
status = request(t, "PUT", marks+"lost", nil)
if status != 200 {
t.Errorf("lost after clearing won: status %d; want 200", status)
}
}
// Concurrent changes to different marks of one opportunity all stick.
func TestChangeMarkConcurrent(t *testing.T) {
url := opportunitiesServer(t, 1)
names := []string{"hidden", "skipped", "interested", "bid_submitted", "won"}
var wg sync.WaitGroup
for _, name := range names {
wg.Add(1)
go func() {
defer wg.Done()
request(t, "PUT", url+"/opportunities/ocds-1/marks/"+name, nil)
}()
}
wg.Wait()
var opportunity types.Opportunity
request(t, "GET", url+"/opportunities/ocds-1", &opportunity)
m := opportunity.Marks
if !m.Hidden || !m.Skipped || !m.Interested || !m.BidSubmitted || !m.Won {
t.Errorf("marks after concurrent PUTs: %+v; want all five set", m)
}
}
// The marks are OpportunityMarks' bool fields, by JSON name, and setting one
// by name sets only that field.
func TestFindMarks(t *testing.T) {
want := "hidden,skipped,interested,bid_submitted,won,lost"
if got := strings.Join(markNames, ","); got != want {
t.Fatalf("markNames = %s; want %s", got, want)
}
for _, name := range markNames {
var marks types.OpportunityMarks
if !setMark(&marks, name, true) {
t.Fatalf("setMark(%q) = false", name)
}
encoded, err := json.Marshal(marks)
if err != nil {
t.Fatal(err)
}
var fields map[string]any
json.Unmarshal(encoded, &fields)
for field, value := range fields {
if set := value == true; set != (field == name) {
t.Errorf("setting %s: %s = %v", name, field, value)
}
}
}
var marks types.OpportunityMarks
if setMark(&marks, "uuid", true) || setMark(&marks, "shiny", true) {
t.Error("setMark accepted a name that isn't a mark")
}
}
+20 -12
View File
@@ -34,7 +34,8 @@ It also runs a worker for every enabled source, as fetcher service does,
saving what they fetch to database.file if it is set, and pushing an saving what they fetch to database.file if it is set, and pushing an
OpportunitiesUpdated event to every WebSocket client when a save stores new OpportunitiesUpdated event to every WebSocket client when a save stores new
opportunities or new releases. A source that stops with an error stops the opportunities or new releases. A source that stops with an error stops the
server too, and the server stopping stops the sources.`, server too, and the server stopping stops the sources. With database.file
set, the API serves the stored opportunities under /api/opportunities.`,
PersistentPreRun: func(cmd *cobra.Command, args []string) { PersistentPreRun: func(cmd *cobra.Command, args []string) {
slog.Debug("command persistent pre-run", "command", "serve") slog.Debug("command persistent pre-run", "command", "serve")
}, },
@@ -112,21 +113,15 @@ server too, and the server stopping stops the sources.`,
stop() stop()
}() }()
appServer, err := server.New(&conf.Server, options...) // The store is shared by the sources, which save to it, and the API,
if err != nil { // which serves it.
return err
}
// runCtx stops whichever of the server and the sources is still
// running once the other has stopped.
runCtx, cancelRun := context.WithCancel(ctx)
defer cancelRun()
var fetcherOptions []fetcher.Option var fetcherOptions []fetcher.Option
opportunityStore, openErr := cli.OpenConfiguredStore(ctx, conf) opportunityStore, openErr := cli.OpenConfiguredStore(ctx, conf)
switch { switch {
case errors.Is(openErr, cli.ErrNoDatabase): case errors.Is(openErr, cli.ErrNoDatabase):
slog.Warn("not storing fetched opportunities", "reason", openErr) slog.Warn(
"opportunities aren't stored or served", "reason", openErr,
)
case openErr != nil: case openErr != nil:
return openErr return openErr
default: default:
@@ -136,8 +131,21 @@ server too, and the server stopping stops the sources.`,
fetcherOptions = append( fetcherOptions = append(
fetcherOptions, fetcher.WithStore(opportunityStore), fetcherOptions, fetcher.WithStore(opportunityStore),
) )
options = append(
options, server.WithStore(opportunityStore),
)
} }
appServer, err := server.New(&conf.Server, options...)
if err != nil {
return err
}
// runCtx stops whichever of the server and the sources is still
// running once the other has stopped.
runCtx, cancelRun := context.WithCancel(ctx)
defer cancelRun()
// Events from the sources, such as new opportunities, are pushed to // Events from the sources, such as new opportunities, are pushed to
// every WebSocket client. // every WebSocket client.
var bus events.Bus var bus events.Bus
+1
View File
@@ -60,6 +60,7 @@ func (s *Server) routes() (http.Handler, error) {
if s.csrf.TokenEnabled() { if s.csrf.TokenEnabled() {
apiRouter.Get("/csrf", s.csrf.IssueHandler) apiRouter.Get("/csrf", s.csrf.IssueHandler)
} }
s.api.Routes(apiRouter)
}) })
// WebSocket (no compression middleware: permessage-deflate is separate) // WebSocket (no compression middleware: permessage-deflate is separate)
+16
View File
@@ -17,9 +17,11 @@ import (
"github.com/pires/go-proxyproto" "github.com/pires/go-proxyproto"
"git.avsdev.uk/Go/job-search/internal/api"
"git.avsdev.uk/Go/job-search/internal/config" "git.avsdev.uk/Go/job-search/internal/config"
"git.avsdev.uk/Go/job-search/internal/logging" "git.avsdev.uk/Go/job-search/internal/logging"
"git.avsdev.uk/Go/job-search/internal/middleware" "git.avsdev.uk/Go/job-search/internal/middleware"
"git.avsdev.uk/Go/job-search/internal/store"
"git.avsdev.uk/Go/job-search/internal/util" "git.avsdev.uk/Go/job-search/internal/util"
"git.avsdev.uk/Go/job-search/internal/websocket" "git.avsdev.uk/Go/job-search/internal/websocket"
) )
@@ -61,6 +63,15 @@ func WithAccessLog(logger *log.Logger) Option {
} }
} }
// WithStore makes the server's API serve the opportunities in `store` under
// /api/opportunities (see api.WithStore). Without it there are no
// opportunity routes.
func WithStore(store *store.Store) Option {
return func(s *Server) {
s.store = store
}
}
// WithMiddleware appends middleware to a group. This is the generic hook // WithMiddleware appends middleware to a group. This is the generic hook
// (auth, metrics, tracing, custom header rules, ...). // (auth, metrics, tracing, custom header rules, ...).
func WithMiddleware(group Group, middlewares ...Middleware) Option { func WithMiddleware(group Group, middlewares ...Middleware) Option {
@@ -107,6 +118,10 @@ type Server struct {
proxyProtocolTrusted []netip.Prefix proxyProtocolTrusted []netip.Prefix
csrf *middleware.CSRF csrf *middleware.CSRF
hub *websocket.Hub hub *websocket.Hub
// store is the opportunities' database, or nil for none.
store *store.Store
// api is the REST API mounted under /api, built from the options.
api *api.API
started time.Time started time.Time
handler http.Handler handler http.Handler
// backgroundCtx is for background work, such as the rate limiters' // backgroundCtx is for background work, such as the rate limiters'
@@ -127,6 +142,7 @@ func New(conf *config.JSCServer, opts ...Option) (*Server, error) {
for _, option := range opts { for _, option := range opts {
option(server) option(server)
} }
server.api = api.New(api.WithStore(server.store))
var err error var err error
server.trusted, err = util.ParseNetPrefixes(conf.Proxy.TrustedProxies) server.trusted, err = util.ParseNetPrefixes(conf.Proxy.TrustedProxies)
+86
View File
@@ -9,6 +9,7 @@ import (
"net/http" "net/http"
"net/http/cookiejar" "net/http/cookiejar"
"net/http/httptest" "net/http/httptest"
"path/filepath"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -20,6 +21,8 @@ import (
"git.avsdev.uk/Go/job-search/internal/config" "git.avsdev.uk/Go/job-search/internal/config"
"git.avsdev.uk/Go/job-search/internal/middleware" "git.avsdev.uk/Go/job-search/internal/middleware"
"git.avsdev.uk/Go/job-search/internal/store"
"git.avsdev.uk/Go/job-search/internal/types"
ws "git.avsdev.uk/Go/job-search/internal/websocket" ws "git.avsdev.uk/Go/job-search/internal/websocket"
) )
@@ -87,6 +90,31 @@ func clientWithJar(t *testing.T) *http.Client {
} }
} }
// apiClient returns a function making requests to the test server ts with
// client c, sending body as JSON and, if withToken, the CSRF token tok.
func apiClient(
t *testing.T, ts *httptest.Server, c *http.Client, tok csrfToken,
) func(method, path, body string, withToken bool) *http.Response {
t.Helper()
return func(method, path, body string, withToken bool) *http.Response {
req, _ := http.NewRequest(method, ts.URL+path, strings.NewReader(body))
if body != "" {
req.Header.Set("Content-Type", "application/json")
}
if withToken {
req.Header.Set(tok.Header, tok.Token)
}
res, err := c.Do(req)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
res.Body.Close()
})
return res
}
}
// Unsafe API requests need the CSRF token: without it they are rejected // Unsafe API requests need the CSRF token: without it they are rejected
// before routing; with it they reach the router (here, 405 from a GET-only // before routing; with it they reach the router (here, 405 from a GET-only
// route). // route).
@@ -307,3 +335,61 @@ func (b *syncBuffer) String() string {
defer b.mu.Unlock() defer b.mu.Unlock()
return b.buffer.String() return b.buffer.String()
} }
// With WithOpportunityStore the opportunities are served, and changing a mark
// needs the CSRF token; without it there are no opportunity routes.
func TestOpportunityRoutes(t *testing.T) {
opportunityStore, err := store.Open(
context.Background(),
filepath.Join(t.TempDir(), "server.db"),
store.WithAutoMigrate(),
)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
opportunityStore.Close()
})
_, err = opportunityStore.SaveOpportunity(
context.Background(),
&types.Opportunity{
UUID: "ocds-1",
UUIDType: "ocid",
Releases: []types.OpportunityRelease{
{
ReleaseID: "r1",
ReleaseDate: time.Now(),
JSON: "{}",
},
},
},
)
if err != nil {
t.Fatal(err)
}
ts := newTestServer(t, nil, WithStore(opportunityStore))
c := clientWithJar(t)
do := apiClient(t, ts, c, getToken(t, c, ts.URL))
if res := do("GET", "/api/opportunities", "", false); res.StatusCode != 200 {
t.Errorf("list: %d; want 200", res.StatusCode)
}
mark := "/api/opportunities/ocds-1/marks/interested"
if res := do("PUT", mark, "", false); res.StatusCode != http.StatusForbidden {
t.Errorf("mark without token: %d; want 403", res.StatusCode)
}
if res := do("PUT", mark, "", true); res.StatusCode != 200 {
t.Errorf("mark with token: %d; want 200", res.StatusCode)
}
without := newTestServer(t, nil)
res, err := http.Get(without.URL + "/api/opportunities")
if err != nil {
t.Fatal(err)
}
res.Body.Close()
if res.StatusCode != http.StatusNotFound {
t.Errorf("no store: %d; want 404", res.StatusCode)
}
}
+2 -1
View File
@@ -58,7 +58,8 @@ var ErrWonAndLost = errors.New("types: an opportunity cannot be both won and los
// OpportunityMarks are the user's own marks on an opportunity, kept apart // OpportunityMarks are the user's own marks on an opportunity, kept apart
// from the fetched data so saving a newer release never changes them. The // from the fetched data so saving a newer release never changes them. The
// marks are independent except that Won and Lost exclude each other (see // marks are independent except that Won and Lost exclude each other (see
// Validate). // Validate). Every bool field is a mark, and the API sets and clears it by
// its JSON name, so a field that isn't a mark mustn't be a bool.
type OpportunityMarks struct { type OpportunityMarks struct {
UUID string `db:"uuid" json:"uuid"` UUID string `db:"uuid" json:"uuid"`
Hidden bool `db:"hidden" json:"hidden"` Hidden bool `db:"hidden" json:"hidden"`