HydraExperienceLibrary

Experience lifecycle manager handling the full journey from draft through live deployment to retirement.

Infrastructure

Operations

Check status

# via hydracluster exec on node-50ab5309
incus exec hydraexperiencelibrary -- systemctl status hydraexperiencelibrary

Or via the health endpoint:

curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/health

View logs

# via hydracluster exec on node-50ab5309
incus exec hydraexperiencelibrary -- journalctl -u hydraexperiencelibrary -n 100 --no-pager

Show last 100 lines:

# via hydracluster exec on node-50ab5309
incus exec hydraexperiencelibrary -- journalctl -u hydraexperiencelibrary -n 100 --no-pager

Restart

The service runs as the Incus scale hydraexperiencelibrary on pi-node-004-nvme (node-50ab5309), behind the edge router. The old host 46.225.120.6 is gone; its IP was recycled and belongs to a foreign machine now. Never ssh there.

# via hydracluster exec on node-50ab5309
incus restart hydraexperiencelibrary

Update

A scale has no self-update path: the image is the unit of update. Tag + push triggers CI, which publishes the OCI image. Then rebuild the scale (via hydracluster exec on node-50ab5309):

hydraskin update hydraexperiencelibrary          # dry run
hydraskin update hydraexperiencelibrary --apply  # rebuild onto the new image

The rebuild preserves state disks, the expose proxy and labels, and verifies /api/v1/health afterward (verified 2026-08-26, v0.10.6 to v0.10.7 in 12 s).

Create experience (API)

Creates a new experience in draft state:

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Experience Name",
    "organization_id": "<organization-id>",
    "description": "Description of the experience"
  }'

List experiences (API)

curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences \
  -H "Authorization: Bearer <token>" | jq .

Promote experience (API)

Move an experience to the next lifecycle state (e.g., draft to staging, staging to live):

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<id>/promote \
  -H "Authorization: Bearer <token>"

Pause experience (API)

Pause a live experience (can be resumed later):

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<id>/pause \
  -H "Authorization: Bearer <token>"

Rollback experience (API)

Roll back an experience to a previous build version:

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<id>/rollback \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "target_version": "<version>"
  }'

Retire experience (API)

Permanently retire an experience (terminal state):

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<id>/retire \
  -H "Authorization: Bearer <token>"

List live experiences (API)

Returns all experiences with status live, a production build, and a non-empty build URL. Used by hydrabody to discover which experience builds to download. Supports optional district and venue query parameters to filter by location:

# All live experiences
curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/live \
  -H "Authorization: Bearer <token>" | jq .

# Live experiences for a specific venue
curl -s "https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/live?district=brussels&venue=mercator-hall" \
  -H "Authorization: Bearer <token>" | jq .

Response fields:

[
  {"name": "rupelmonde-castle-viewer", "build_number": 1, "build_url": "https://releases.experiencenet.com/builds/...", "exe_path": "C:\\experiences\\rupelmonde-castle-viewer\\Rupelmonde.exe"}
]

Start stream (API)

Returns a URL to hydraheadwebstream for browser streaming of the given experience:

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<name>/stream \
  -H "Authorization: Bearer <token>"

Response:

{
  "stream_url": "https://hydraheadwebstream.experiencenet.com/?experience=<name>"
}

Requires hydraheadwebstream_url to be configured.

Pipeline health check (API)

End-to-end health check of the entire deployment pipeline: hydraperforce, hydrarelease, hydradistrict, hydracluster, hydramirror, and local experience status. Supports filtering by district and experience name.

# Full pipeline health for a district and experience
curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/pipeline/health?district=bxl1-test\&experience=mercator-talks \
  -H "Authorization: Bearer <token>" | jq .

# District-only check
curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/pipeline/health?district=bxl1-test \
  -H "Authorization: Bearer <token>" | jq .

# Full pipeline check (all experiences, no district filtering)
curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/pipeline/health \
  -H "Authorization: Bearer <token>" | jq .

Requires the pipeline config section:

pipeline:
  hydraperforce_url: "https://hydraperforce.experiencenet.com"
  hydrarelease_url: "https://releases.experiencenet.com"
  hydradistrict_url: "https://hydradistrict.experiencenet.com"
  hydramirror_url_pattern: "https://{district}.hydramirror.experiencenet.com"

Key Concepts

• Lifecycle states: experiences progress through draft -> staging -> live -> paused -> retired. Promotion moves forward, rollback reverts to a previous build version. Paused experiences can be resumed to live. Retired is a terminal state.
• Build integration via SSE: the service listens for SSE events from HydraRelease. When a new build is released, it automatically becomes available for experiences to pick up. Agencies submit Unreal Engine builds via Perforce, which flow through HydraPerforce -> HydraRelease -> HydraExperienceLibrary.
• YAML store: index file plus per-entity YAML files in /root/.hydraexperiencelibrary/. Each experience file tracks its current state, build version history, and metadata. No database required.
• REST API with bearer token auth: all mutating endpoints require a valid bearer token issued by hydraauth.
• Browser streaming via hydraheadwebstream: the hydraheadwebstream_url config field enables stream links. Stream buttons redirect to hydraheadwebstream, which manages its own streaming sessions (moonlight-web-stream instances, Sunshine pairing, WebRTC).
• SSE events via hydramonitor: experience lifecycle changes are broadcast as server-sent events, consumed by HydraPipeline and other services.
• Orientation: experiences can set orientation: portrait to stream in portrait mode (e.g. 720x1280). The field is exposed via the REST API and consumed by hydraheadwebstream, which passes it to hydraneckwebrtc. Default is landscape (empty or "landscape").
• Auto-updates from releases.experiencenet.com: the service checks for new releases and updates itself without manual intervention.
• Release channel routing: when promoting builds, reprovision targets nodes based on their release_channel field in hydracluster. Staging promotions only reach nodes with release_channel="staging". Production promotions reach all other nodes (default = production). This allows testing new builds on specific nodes before rolling out to the full fleet.
• Test districts: districts ending in -test (e.g. bxl1-test) share infrastructure with their base district (bxl1). The pipeline health check resolves mirrors via the base district. Test district nodes are invisible to production consumers (e.g. hydraheadwebstream querying eligible bodies in bxl1).

Troubleshooting

YAML corruption

If the YAML store becomes corrupted (partial writes, disk issues):

1. Stop the service: systemctl stop hydraexperiencelibrary
2. Check the data directory: ls -la /root/.hydraexperiencelibrary/
3. Look for malformed YAML files -- inspect the index and individual entity files
4. Restore from backup if needed, or manually fix the YAML syntax
5. Start the service: systemctl start hydraexperiencelibrary

Service won't start

1. Check logs for the error: journalctl -u hydraexperiencelibrary -n 50 --no-pager
2. Verify config exists and is valid: cat /root/.hydraexperiencelibrary/config.yaml
3. Check if the port is already in use: ss -tlnp | grep <port>
4. Verify the binary exists and is executable: ls -la $(which hydraexperiencelibrary)
5. Try running manually to see output: hydraexperiencelibrary serve

Auth token issues

1. Verify the token by calling hydraauth directly: curl -s https://hydraauth.experiencenet.com/api/v1/verify -H "Authorization: Bearer <token>"
2. Check that hydraauth is reachable from the experience library server: incus exec hydraexperiencelibrary -- curl -s https://hydraauth.experiencenet.com/api/v1/health (via exec on node-50ab5309)
3. If tokens are consistently rejected, check clock sync: incus exec hydraexperiencelibrary -- date -- token validation may fail if the clock is skewed

Experience stuck in wrong state

1. Check the experience YAML file in /root/.hydraexperiencelibrary/ for the current state
2. Review logs for failed state transitions: journalctl -u hydraexperiencelibrary --since "1 hour ago" | grep <experience-id>
3. If the state is inconsistent, stop the service, manually correct the YAML file, and restart

Not picking up new builds from HydraRelease

1. Check that the SSE connection to HydraRelease is active in the logs
2. Verify HydraRelease is healthy: curl -s https://hydrarelease.experiencenet.com/api/v1/health
3. Check for network issues between the servers: incus exec hydraexperiencelibrary -- curl -s https://hydrarelease.experiencenet.com/api/v1/health
4. Restart the service to re-establish the SSE connection if needed

Backup & Recovery

What is backed up

OBSOLETE: the Hetzner snapshot arrangement covered the retired dedicated server. The scale's state lives as a host disk device on pi-node-004-nvme and survives image rebuilds; a scale-level backup policy is an open item.

The snapshot covers the entire server disk, including all experience and build records.

Restore procedure

1. In the Hetzner Cloud Console (hydraexperiencenet project), open Servers → hydraexperiencelibrary → Backups.
2. Power off the server, restore the snapshot, then power on.
3. Verify:

   curl https://hydraexperiencelibrary.experiencenet.com/api/v1/health