HydraExperienceLibrary

Experience lifecycle manager handling the full journey from draft through live deployment to retirement.

Infrastructure

Operations

Check status

ssh root@46.225.120.6 systemctl status hydraexperiencelibrary

Or via the health endpoint:

curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/health

View logs

ssh root@46.225.120.6 journalctl -u hydraexperiencelibrary -f

Show last 100 lines:

ssh root@46.225.120.6 journalctl -u hydraexperiencelibrary -n 100 --no-pager

Restart

ssh root@46.225.120.6 systemctl restart hydraexperiencelibrary

Update

Updates happen automatically. The service polls releases.experiencenet.com for new versions and applies them. To check the current version:

ssh root@46.225.120.6 hydraexperiencelibrary version

Never manually deploy -- always use the release pipeline (tag + push to trigger CI, release server distributes, service auto-updates).

Create experience (API)

Creates a new experience in draft state:

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Experience Name",
    "organization_id": "<organization-id>",
    "description": "Description of the experience"
  }'

List experiences (API)

curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences \
  -H "Authorization: Bearer <token>" | jq .

Promote experience (API)

Move an experience to the next lifecycle state (e.g., draft to staging, staging to live):

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<id>/promote \
  -H "Authorization: Bearer <token>"

Pause experience (API)

Pause a live experience (can be resumed later):

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<id>/pause \
  -H "Authorization: Bearer <token>"

Rollback experience (API)

Roll back an experience to a previous build version:

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<id>/rollback \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "target_version": "<version>"
  }'

Retire experience (API)

Permanently retire an experience (terminal state):

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<id>/retire \
  -H "Authorization: Bearer <token>"

List live experiences (API)

Returns all experiences with status live, a production build, and a non-empty build URL. Used by hydrabody to discover which experience builds to download. Supports optional district and venue query parameters to filter by location:

# All live experiences
curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/live \
  -H "Authorization: Bearer <token>" | jq .

# Live experiences for a specific venue
curl -s "https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/live?district=brussels&venue=mercator-hall" \
  -H "Authorization: Bearer <token>" | jq .

Response fields:

[
  {"name": "rupelmonde-castle-viewer", "build_number": 1, "build_url": "https://releases.experiencenet.com/builds/...", "exe_path": "C:\\experiences\\rupelmonde-castle-viewer\\Rupelmonde.exe"}
]

Start stream (API)

Returns a URL to hydraheadwebstream for browser streaming of the given experience:

curl -X POST https://hydraexperiencelibrary.experiencenet.com/api/v1/experiences/<name>/stream \
  -H "Authorization: Bearer <token>"

Response:

{
  "stream_url": "https://hydraheadwebstream.experiencenet.com/?experience=<name>"
}

Requires hydraheadwebstream_url to be configured.

Pipeline health check (API)

End-to-end health check of the entire deployment pipeline: hydraperforce, hydrarelease, hydradistrict, hydracluster, hydramirror, and local experience status. Supports filtering by district and experience name.

# Full pipeline health for a district and experience
curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/pipeline/health?district=bxl1-test\&experience=mercator-talks \
  -H "Authorization: Bearer <token>" | jq .

# District-only check
curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/pipeline/health?district=bxl1-test \
  -H "Authorization: Bearer <token>" | jq .

# Full pipeline check (all experiences, no district filtering)
curl -s https://hydraexperiencelibrary.experiencenet.com/api/v1/pipeline/health \
  -H "Authorization: Bearer <token>" | jq .

Requires the pipeline config section:

pipeline:
  hydraperforce_url: "https://hydraperforce.experiencenet.com"
  hydrarelease_url: "https://releases.experiencenet.com"
  hydradistrict_url: "https://hydradistrict.experiencenet.com"
  hydramirror_url_pattern: "https://{district}.hydramirror.experiencenet.com"

Key Concepts

• Lifecycle states: experiences progress through draft -> staging -> live -> paused -> retired. Promotion moves forward, rollback reverts to a previous build version. Paused experiences can be resumed to live. Retired is a terminal state.
• Build integration via SSE: the service listens for SSE events from HydraRelease. When a new build is released, it automatically becomes available for experiences to pick up. Agencies submit Unreal Engine builds via Perforce, which flow through HydraPerforce -> HydraRelease -> HydraExperienceLibrary.
• YAML store: index file plus per-entity YAML files in /root/.hydraexperiencelibrary/. Each experience file tracks its current state, build version history, and metadata. No database required.
• REST API with bearer token auth: all mutating endpoints require a valid bearer token issued by hydraauth.
• Browser streaming via hydraheadwebstream: the hydraheadwebstream_url config field enables stream links. Stream buttons redirect to hydraheadwebstream, which manages its own streaming sessions (moonlight-web-stream instances, Sunshine pairing, WebRTC).
• SSE events via hydramonitor: experience lifecycle changes are broadcast as server-sent events, consumed by HydraPipeline and other services.
• Orientation: experiences can set orientation: portrait to stream in portrait mode (e.g. 720x1280). The field is exposed via the REST API and consumed by hydraheadwebstream, which passes it to hydraneckwebrtc. Default is landscape (empty or "landscape").
• Auto-updates from releases.experiencenet.com: the service checks for new releases and updates itself without manual intervention.
• Release channel routing: when promoting builds, reprovision targets nodes based on their release_channel field in hydracluster. Staging promotions only reach nodes with release_channel="staging". Production promotions reach all other nodes (default = production). This allows testing new builds on specific nodes before rolling out to the full fleet.
• Test districts: districts ending in -test (e.g. bxl1-test) share infrastructure with their base district (bxl1). The pipeline health check resolves mirrors via the base district. Test district nodes are invisible to production consumers (e.g. hydraheadwebstream querying eligible bodies in bxl1).

Troubleshooting

YAML corruption

If the YAML store becomes corrupted (partial writes, disk issues):

1. Stop the service: systemctl stop hydraexperiencelibrary
2. Check the data directory: ls -la /root/.hydraexperiencelibrary/
3. Look for malformed YAML files -- inspect the index and individual entity files
4. Restore from backup if needed, or manually fix the YAML syntax
5. Start the service: systemctl start hydraexperiencelibrary

Service won't start

1. Check logs for the error: journalctl -u hydraexperiencelibrary -n 50 --no-pager
2. Verify config exists and is valid: cat /root/.hydraexperiencelibrary/config.yaml
3. Check if the port is already in use: ss -tlnp | grep <port>
4. Verify the binary exists and is executable: ls -la $(which hydraexperiencelibrary)
5. Try running manually to see output: hydraexperiencelibrary serve

Auth token issues

1. Verify the token by calling hydraauth directly: curl -s https://hydraauth.experiencenet.com/api/v1/verify -H "Authorization: Bearer <token>"
2. Check that hydraauth is reachable from the experience library server: ssh root@46.225.120.6 curl -s https://hydraauth.experiencenet.com/api/v1/health
3. If tokens are consistently rejected, check clock sync: ssh root@46.225.120.6 date -- token validation may fail if the server clock is skewed

Experience stuck in wrong state

1. Check the experience YAML file in /root/.hydraexperiencelibrary/ for the current state
2. Review logs for failed state transitions: journalctl -u hydraexperiencelibrary --since "1 hour ago" | grep <experience-id>
3. If the state is inconsistent, stop the service, manually correct the YAML file, and restart

Not picking up new builds from HydraRelease

1. Check that the SSE connection to HydraRelease is active in the logs
2. Verify HydraRelease is healthy: curl -s https://hydrarelease.experiencenet.com/api/v1/health
3. Check for network issues between the servers: ssh root@46.225.120.6 curl -s https://hydrarelease.experiencenet.com/api/v1/health
4. Restart the service to re-establish the SSE connection if needed