HydraOrganization

Organization registry managing customers and agencies in the Hydra platform.

Infrastructure

Health Check

curl -s https://hydraorganization.experiencenet.com/api/v1/health

The response includes the service version and the current count of organizations stored.

API Endpoints

All mutating endpoints (POST, PUT, DELETE) require a bearer token in the Authorization header. Read endpoints (GET) do not require auth.

Create organization

curl -X POST https://hydraorganization.experiencenet.com/api/v1/organizations \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "visit-flanders",
    "name": "Visit Flanders",
    "type": "customer",
    "description": "Flemish tourism board",
    "contact_name": "Jane Doe",
    "contact_email": "jane@visitflanders.com"
  }'

The id field is required and must be unique. The name field is required. The type field accepts any string; the conventional values are customer (end users operating venues) and agency (content creators submitting builds).

Get organization

curl -s https://hydraorganization.experiencenet.com/api/v1/organizations/visit-flanders

List organizations

curl -s https://hydraorganization.experiencenet.com/api/v1/organizations | jq .

Filter by type:

curl -s "https://hydraorganization.experiencenet.com/api/v1/organizations?type=customer" | jq .

Update organization

curl -X PUT https://hydraorganization.experiencenet.com/api/v1/organizations/visit-flanders \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Visit Flanders",
    "type": "customer",
    "contact_email": "new-contact@visitflanders.com"
  }'

Delete organization

curl -X DELETE https://hydraorganization.experiencenet.com/api/v1/organizations/visit-flanders \
  -H "Authorization: Bearer <token>"

A successful delete returns HTTP 204 with no body.

How It Works

HydraOrganization is a Go service (Cobra CLI + YAML store + systemd) that acts as the central registry for organizations in the Hydra platform.

YAML store: organization records live in /root/.hydraorganization/. The store has two layers:

• organizations.yaml -- index file with lightweight entries (id, name, type, timestamps) used for fast listing
• organizations/<id>/organization.yaml -- full per-organization record including description and contact fields

All writes use atomic rename (write to .tmp then rename into place) to prevent partial writes from corrupting the store.

Auth: mutating endpoints are protected by a static bearer token configured in config.yaml under server.admin_token. Token validation is handled by hydraauth.

SSE events: every create, update, and delete emits a server-sent event on /api/v1/events. Downstream services (HydraPipeline, HydraVenues) subscribe to this stream to react to organization changes.

Auto-updates: on startup the service registers a background poller that checks releases.experiencenet.com every 6 hours. If a newer version is available it downloads, replaces the binary, and restarts via systemd. Setting HYDRA_AUTO_UPDATE=off skips the poller entirely (logged as Auto-update: disabled (HYDRA_AUTO_UPDATE=off)); the container image sets this, since there the image is the unit of update and there is no systemd unit to restart.

Container image: v* tags also publish a multi-arch (amd64 + arm64) OCI image to scaleregistry.experiencenet.com/hydraorganization:<tag> and :latest. It runs serve --dev --listen :8080 — plain HTTP, with TLS terminated upstream — and expects /root/.hydraorganization to be supplied as an Incus disk device (or a Docker bind mount); the image deliberately declares no VOLUME, because Incus's OCI runtime cannot satisfy an anonymous volume and the container would fail to start.

TLS: production mode uses autocert (Let's Encrypt) with the certificate cache at /root/.hydraorganization/certs.

Operations

Check status

ssh root@46.225.184.72 systemctl status hydraorganization

View logs

ssh root@46.225.184.72 journalctl -u hydraorganization -f

Show last 100 lines:

ssh root@46.225.184.72 journalctl -u hydraorganization -n 100 --no-pager

Restart

ssh root@46.225.184.72 systemctl restart hydraorganization

Check version

ssh root@46.225.184.72 hydraorganization version

Release a new version

Never manually deploy. Use the release pipeline:

1. Tag the commit: git -C /home/claude-user/hydraorganization tag v<X.Y.Z>
2. Push the tag: git -C /home/claude-user/hydraorganization push origin v<X.Y.Z>
3. GitHub Actions builds cross-platform binaries, creates a GitHub Release, and publishes to releases.experiencenet.com.
4. The running service picks up the new version within 6 hours. To apply immediately: ssh root@46.225.184.72 hydraorganization update
5. Verify: hydrarelease verify --project hydraorganization

Troubleshooting

YAML corruption

If the YAML store becomes corrupted (partial writes, disk issues):

1. Stop the service: ssh root@46.225.184.72 systemctl stop hydraorganization
2. Check the data directory: ssh root@46.225.184.72 ls -la /root/.hydraorganization/
3. Inspect the index and any suspect per-organization files: ssh root@46.225.184.72 cat /root/.hydraorganization/organizations.yaml
4. Look for orphaned .tmp files left by a crashed write: ssh root@46.225.184.72 find /root/.hydraorganization -name '*.tmp'
5. Remove any .tmp files: ssh root@46.225.184.72 find /root/.hydraorganization -name '*.tmp' -delete
6. Restore from backup if the YAML is unrecoverable (see Backup & Recovery below)
7. Start the service: ssh root@46.225.184.72 systemctl start hydraorganization

Service won't start

1. Check logs for the error: ssh root@46.225.184.72 journalctl -u hydraorganization -n 50 --no-pager
2. Verify config exists and is valid YAML: ssh root@46.225.184.72 cat /root/.hydraorganization/config.yaml
3. Check that server.admin_token is set in the config -- the service refuses to start without it
4. Check if the port is already in use: ssh root@46.225.184.72 ss -tlnp | grep 443
5. Verify the binary exists and is executable: ssh root@46.225.184.72 ls -la $(which hydraorganization)
6. Try running manually to see startup output: ssh root@46.225.184.72 hydraorganization serve

Auth token issues

1. Verify the token against hydraauth: curl -s https://hydraauth.experiencenet.com/api/v1/verify -H "Authorization: Bearer <token>"
2. Check that hydraauth is reachable from the organization server: ssh root@46.225.184.72 curl -s https://hydraauth.experiencenet.com/api/v1/health
3. If tokens are consistently rejected, check clock sync -- token validation may fail if the server clock is skewed: ssh root@46.225.184.72 date

HydraVenues references an organization that no longer exists

If a venue's organization_id points to a deleted organization:

1. List all organizations to confirm the ID is gone: curl -s https://hydraorganization.experiencenet.com/api/v1/organizations | jq .
2. Either recreate the organization with the same ID, or update the venue to reference a valid organization ID via the HydraVenues API

SSE event stream drops

If downstream subscribers (HydraPipeline, HydraVenues) stop receiving organization events:

1. Check the events endpoint is reachable: curl -s -N -H "Authorization: Bearer <token>" https://hydraorganization.experiencenet.com/api/v1/events
2. Restart the service to reset the hydramonitor broadcast state: ssh root@46.225.184.72 systemctl restart hydraorganization
3. Confirm downstream services reconnect by checking their logs

Backup & Recovery

What is backed up

Hetzner automated daily server snapshots are enabled on hydraorganization (46.225.184.72), context hydraexperiencenet. Backup window: 22:00--02:00 UTC, 7-day retention.

The snapshot covers the entire server disk, including:

• All organization YAML files under /root/.hydraorganization/
• The service config at /root/.hydraorganization/config.yaml
• TLS certificate cache at /root/.hydraorganization/certs
• The binary at /usr/local/bin/hydraorganization

Restore procedure

1. Open the Hetzner Cloud Console and switch to the hydraexperiencenet project.
2. Go to Servers, select hydraorganization, then open the Backups tab.
3. Identify the snapshot to restore (most recent, or a specific date if rolling back to a known-good state).
4. Power off the server from the Console (or via ssh root@46.225.184.72 poweroff).
5. Click Restore on the chosen snapshot and confirm.
6. Power the server back on from the Console.
7. Verify the service is running and data is intact:

   curl -s https://hydraorganization.experiencenet.com/api/v1/health
   curl -s https://hydraorganization.experiencenet.com/api/v1/organizations | jq 'length'
   

8. If the restored snapshot is more than a few hours old, check with the team whether any organizations were created or modified after the snapshot timestamp and recreate them manually via the API.