HydraDistrict

District registry and geo-proximity service for the Hydra infrastructure. Manages district definitions, service endpoints, and geographic coordinates.

Infrastructure

Health Check

curl -s https://hydradistrict.experiencenet.com/api/v1/health

The response includes the district count:

{
  "service": "hydradistrict",
  "status": "ok",
  "district_count": 3
}

API Endpoints

All mutating endpoints require a bearer token in the Authorization header. Read endpoints (GET) are unauthenticated.

Create a district

curl -X POST https://hydradistrict.experiencenet.com/api/v1/districts \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "eu-nbg1",
    "name": "EU Nuremberg",
    "location": "Nuremberg, Germany",
    "latitude": 49.45,
    "longitude": 11.08,
    "provider": "hetzner",
    "services": [
      {"type": "cluster", "url": "https://hydracluster.experiencenet.com"},
      {"type": "venues",  "url": "https://hydravenues.experiencenet.com"}
    ]
  }'

List districts

curl -s https://hydradistrict.experiencenet.com/api/v1/districts | jq .

Filter by provider:

curl -s "https://hydradistrict.experiencenet.com/api/v1/districts?provider=hetzner" | jq .

Get a single district

curl -s https://hydradistrict.experiencenet.com/api/v1/districts/eu-nbg1 | jq .

Update a district

curl -X PUT https://hydradistrict.experiencenet.com/api/v1/districts/eu-nbg1 \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "EU Nuremberg",
    "location": "Nuremberg, Germany",
    "latitude": 49.45,
    "longitude": 11.08,
    "provider": "hetzner"
  }'

Delete a district

curl -X DELETE https://hydradistrict.experiencenet.com/api/v1/districts/eu-nbg1 \
  -H "Authorization: Bearer <token>"

Get service URLs for a district

Returns a map of service type to URL for the district. No auth required.

curl -s https://hydradistrict.experiencenet.com/api/v1/districts/eu-nbg1/services | jq .

Example response:

{
  "district_id": "eu-nbg1",
  "services": {
    "cluster": "https://hydracluster.experiencenet.com",
    "venues":  "https://hydravenues.experiencenet.com"
  }
}

Find nearest district

Requires coordinates with geo-data set on at least one district. Returns the closest district and the distance in kilometers.

curl -s "https://hydradistrict.experiencenet.com/api/v1/nearest?lat=50.85&lon=4.35" | jq .

Example response:

{
  "district": { "id": "eu-bxl1", "name": "EU Brussels", ... },
  "distance_km": 12.3
}

How It Works

HydraDistrict is a Go service (Cobra CLI, systemd) backed by a YAML store with no database dependency.

Data layout on disk:

/root/.hydradistrict/
  config.yaml                    # server config
  districts.yaml                 # index of all districts
  districts/
    <id>/
      district.yaml              # full district record
  certs/                         # autocert TLS cache

Writes are atomic: all YAML files are written to a .tmp file and renamed into place, so a crash mid-write never leaves a partially written file.

Events: on every create, update, or delete the service emits an SSE event (district.created, district.updated, district.deleted) via hydramonitor. Other services subscribe on GET /api/v1/events.

Auto-updates: the service polls releases.experiencenet.com every 6 hours and applies any newer version automatically. Releases are triggered by pushing a v* tag to GitHub, which runs CI, builds cross-platform binaries, and uploads them to the release server. Setting HYDRA_AUTO_UPDATE=off disables the poll entirely — the container image sets it, because there the image is the unit of update and there is no systemd unit to restart.

Container image: the same v* tag also builds a multi-arch (linux/amd64 + linux/arm64) OCI image and pushes it to scaleregistry.experiencenet.com/hydradistrict:<tag> and :latest. The image runs serve --dev --listen :8080 (plain HTTP, TLS terminated upstream) and carries no VOLUME directive: persistence must be attached as an Incus disk device (or a Docker bind mount) at /root/.hydradistrict, pre-staged with a config.yaml containing at least server.admin_token — the service refuses to start without it.

TLS: autocert issues and renews certificates automatically for hydradistrict.experiencenet.com. The certificate cache lives in /root/.hydradistrict/certs/.

Operations

Check status

ssh root@46.224.179.9 systemctl status hydradistrict

View logs

ssh root@46.224.179.9 journalctl -u hydradistrict -f

Show last 100 lines:

ssh root@46.224.179.9 journalctl -u hydradistrict -n 100 --no-pager

Restart

ssh root@46.224.179.9 systemctl restart hydradistrict

Check current version

ssh root@46.224.179.9 hydradistrict version

Release a new version

Never deploy manually. Use the release pipeline:

1. Tag the new version: git tag v<X.Y.Z>
2. Push the tag: git push origin v<X.Y.Z>
3. GitHub Actions builds binaries and uploads them to the release server.
4. The running service picks up the update within 6 hours, or restart it to apply immediately.
5. Verify: hydrarelease verify --project hydradistrict

Troubleshooting

Service not responding

1. Check the health endpoint: curl -s https://hydradistrict.experiencenet.com/api/v1/health
2. SSH to the server: ssh root@46.224.179.9
3. Check service status: systemctl status hydradistrict
4. Check logs: journalctl -u hydradistrict -n 50 --no-pager
5. Restart if needed: systemctl restart hydradistrict

Service won't start

1. Check logs for the error: journalctl -u hydradistrict -n 50 --no-pager
2. Verify config exists and is valid: cat /root/.hydradistrict/config.yaml
3. Check if the port is already in use: ss -tlnp | grep 443
4. Verify the binary exists and is executable: ls -la $(which hydradistrict)
5. Try running manually to see output: hydradistrict serve

YAML store corruption

If the YAML store becomes corrupted (partial writes, disk issues):

1. Stop the service: systemctl stop hydradistrict
2. Inspect the data directory: ls -la /root/.hydradistrict/
3. Check the index file: cat /root/.hydradistrict/districts.yaml
4. Check individual district files: ls /root/.hydradistrict/districts/
5. Restore from backup if needed (see Backup and Recovery below), or manually fix the YAML
6. Start the service: systemctl start hydradistrict

Auth token rejected

1. Verify the token is set in config: cat /root/.hydradistrict/config.yaml
2. Check clock sync -- token validation can fail if the server clock is skewed: ssh root@46.224.179.9 date
3. If the token was rotated, update server.admin_token in config and restart the service

Geo-proximity returns no results

The nearest endpoint only considers districts that have non-zero latitude and longitude values. If it returns 404, either no districts have coordinates set or the store is empty. Set coordinates via the update endpoint (see API Endpoints above).

Backup and Recovery

What is backed up

Hetzner automated daily server snapshots are enabled on hydradistrict (46.224.179.9), context hydraexperiencenet. Backup window: 06:00-10:00 UTC, 7-day retention.

The snapshot covers the entire server disk, including all district YAML files, the TLS certificate cache, and the service configuration.

Restore procedure

1. In the Hetzner Cloud Console (hydraexperiencenet project), open Servers -> hydradistrict -> Backups.
2. Power off the server, restore the desired snapshot, then power on.
3. Verify the service is healthy:

   curl -s https://hydradistrict.experiencenet.com/api/v1/health
   

4. Confirm the district count matches expectations:

   curl -s https://hydradistrict.experiencenet.com/api/v1/districts | jq length