Added websocket hub for handling websocket clients, with a wsserve command

which runs a plain TCP server for receiving websocket hub clients, with
server config, CSRF and rate limit middleware.

- internal/websocket: Hub pushing messages to WebSocket clients (ServeHTTP,
  example.com/ws upgrade with optional CSRF check) and plain TCP clients (ServeTCP,
  newline-delimited JSON envelopes); ping/pong keepalive, per-client message
  rate limit, slow consumer handling, graceful Shutdown, debug logging of
  messages sent and received
- internal/websocket/tcpserver: standalone TCP server for a hub, with accept
  back-off, a per-IP connection limit (server.websocket.connection_limit) and
  graceful shutdown
- internal/cli/wsserve: `wsserve` command running the TCP server, with
  --listen/--host/--port overrides
- internal/middleware: keyed token bucket rate limiter, real client IP from
  trusted proxies
- internal/httpx: RFC 9457 problem responses, request IDs in context
- internal/config: server section (host, port, shutdown timeout, rate limiter
  idle TTL, websocket); util.ByteSize settings (e.g. "32KiB"); lists of
  sections as single keys; empty lists stored as nil; isExplicitlySet for
  printLoadedConfig; JobSearchConfigSource renamed to JSCSourcesSource;
  TestSections checks every section's methods
- internal/util: ByteSize, parsed and formatted with go-humanize
This commit is contained in:
bwilliams committed 2026-10-14 15:55:25 +01:00
1 parent 803a8929e6
commit c9f42b62ad
42 files changed
+3619 -57

No files matched your search

+21
View File
@@ -40,11 +40,26 @@ is kept.
| `sources.find_a_tender.api_root` | string | `https://www.find-tender.service.gov.uk/api/1.0` | URL of the Find a Tender API |
| `sources.fts_mock.api_root` | string | | URL of the mock FTS server (see `internal/mock`) |
| `sources.sleeper.block_shutdown` | bool | `false` | Test source: if true, its sleep ignores shutdown |
| `server.host` | string | `127.0.0.1` | Address to listen on |
| `server.port` | int | `9393` | Port to listen on |
| `server.timeouts.shutdown` | duration | `15s` | How long shutdown waits for clients to close |
| `server.rate_limits.idle_ttl` | duration | `10m` | How long a client's rate limiter is kept after its last request |
| `server.websocket.enable_compression` | bool | `false` | permessage-deflate |
| `server.websocket.max_message_size` | byte size | `32KiB` | Largest message read from a client; `0B` for 32KiB |
| `server.websocket.connection_limit.*` | | | A rate limit rule for new connections from each client IP, WebSocket and plain TCP alike; limiters are kept for `server.rate_limits.idle_ttl` |
| `server.websocket.message_rate_limit.*` | | | A rate limit rule for messages from each connection |
| `server.websocket.write_timeout` | duration | `10s` | `0s` for none |
| `server.websocket.ping_interval` | duration | `30s` | |
| `server.websocket.pong_timeout` | duration | `10s` | `0s` for none |
The current sources are `find_a_tender`, `fts_mock` and `sleeper`. See
"Addition of a new opportunities source" in the top level README for how to add
a new source.
A rate limit rule (`JSCRateLimitRule`) has `enabled`, `per_second` and `burst`.
It is used under `server.websocket.connection_limit` and
`server.websocket.message_rate_limit`.
Example `config.yaml`:
```yaml
logging:
@@ -79,6 +94,9 @@ per key:
`time.Duration` or a duration string in `SetValue`, and are given out as
strings (`"10m0s"`) by `Map`, `Flatten` and `ValueOf`, and so by `JSON`,
`YAML` and `String` too.
- `util.ByteSize` fields work the same way, with size strings such as `"32KiB"`
or `"1MB"` (KB, MB, ... are powers of 1000; KiB, MiB, ... powers of 1024),
and are given out exactly in the largest unit that divides them (`"1MB"`).
- A section can reject values by implementing `validate(key, value) error` on
its pointer type. `JobSearchConfigLogging` does this for `level`.
@@ -138,6 +156,9 @@ Nothing in `section.go` needs changing.
| `conf_logging.go` | `logging` section and log level helpers |
| `conf_sources.go` | `sources` section and the shared source fields |
| `conf_source_*.go` | One file per source section |
| `conf_server.go` | `server` section |
| `conf_server_*.go` | One file per `server` sub-section (`rate_limits`, `timeouts`, `websocket`) |
| `conf_ratelimitrule.go` | `JSCRateLimitRule`, the rate limit rule shared by several keys |
| `print.go`, `debug.go` | Printing and debug logging of the configuration |