HydraNeck

Venue network management service that polls routers and correlates network clients with hydracluster nodes.

Infrastructure

Operations

Check status

ssh root@46.225.8.28 systemctl status hydraneck

Or via the health endpoint:

curl -s https://hydraneck.experiencenet.com/api/v1/health

View logs

ssh root@46.225.8.28 journalctl -u hydraneck -f

Show last 100 lines:

ssh root@46.225.8.28 journalctl -u hydraneck -n 100 --no-pager

Restart

ssh root@46.225.8.28 systemctl restart hydraneck

Update

Updates happen automatically. The service polls releases.experiencenet.com for new versions and applies them. To check the current version:

ssh root@46.225.8.28 hydraneck version

Never manually deploy -- always use the release pipeline (tag + push to trigger CI, release server distributes, service auto-updates).

Scan venues (API)

Trigger a network scan for a specific venue:

curl -X POST https://hydraneck.experiencenet.com/api/v1/venues/<venue-id>/scan \
  -H "Authorization: Bearer <token>"

List venue network status (API)

curl -s https://hydraneck.experiencenet.com/api/v1/venues/<venue-id>/network \
  -H "Authorization: Bearer <token>" | jq .

View bandwidth history (API)

curl -s https://hydraneck.experiencenet.com/api/v1/venues/<venue-id>/bandwidth \
  -H "Authorization: Bearer <token>" | jq .

Returns 24h of bandwidth samples (tx/rx in bps) and the venue's capacity in bps. Also visible as a chart on the venue detail page.

Check WireGuard tunnels (API)

curl -s https://hydraneck.experiencenet.com/api/v1/venues/<venue-id>/tunnels \
  -H "Authorization: Bearer <token>" | jq .

Diagnose node connectivity (API)

Check correlation state and connectivity for a specific node:

curl -s https://hydraneck.experiencenet.com/api/v1/nodes/<node-id>/connectivity \
  -H "Authorization: Bearer <token>" | jq .

List all nodes with correlation states (API)

curl -s https://hydraneck.experiencenet.com/api/v1/nodes \
  -H "Authorization: Bearer <token>" | jq .

Routes

Public (no auth)

• GET / -- Landing page
• GET /docs -- Redirects to /docs/overview
• GET /docs/{section} -- Documentation viewer (partner network design, traffic flows, etc.)

Auth required (admin token)

• GET /admin -- Dashboard (venue overview, WireGuard tunnels, air units)
• GET /admin/venues/{name} -- Venue detail page (routers, nodes, bandwidth)
• POST /admin/nodes/{id}/assign -- Assign node to district/venue

API (bearer token)

• GET /api/v1/health -- Health endpoint
• GET /api/v1/runbook -- Raw markdown runbook
• GET /api/v1/venues -- List venues
• GET /api/v1/venues/{name} -- Venue details
• GET /api/v1/venues/{name}/network -- Node network status
• POST /api/v1/venues/{name}/scan -- Trigger scan
• GET /api/v1/venues/{name}/bandwidth -- Bandwidth history
• GET /api/v1/venues/{name}/wg -- WireGuard status

Key Concepts

• Router polling: HydraNeck periodically polls venue routers (Omada and MikroTik) to discover connected network clients and their status.
• Client-node correlation: discovered network clients are correlated with known hydracluster nodes. Each node is assigned one of four correlation states:
  • healthy -- node is on the network and its agent is responding normally
  • agent-down -- node is visible on the network but its agent is not responding
  • not-on-network -- node's agent is reporting in but the node is not visible on the venue router (network issue)
  • unmanaged -- a network client that does not match any known hydracluster node. When HydraCluster is configured (cluster.url), each unmanaged device row shows an Enrol button that deep-links to HydraCluster's /enroll page with the device name and IP pre-filled. The admin fills in the owner field and submits; HydraCluster creates a pending node with the correct IP and shows the install script to run on the device.
• WireGuard tunnels: venues connect to the Hydra infrastructure via WireGuard tunnels. HydraNeck monitors tunnel status and reports connectivity issues.
• YAML store: index file plus per-entity YAML files in /root/.hydraneck/. Stores venue network state, node correlations, and router configurations. Bandwidth history is stored per venue in /root/.hydraneck/bandwidth/<venue>.yaml (24h rolling window, ~288 samples at 5min intervals). No database required.
• REST API with bearer token auth: all mutating endpoints require a valid bearer token issued by hydraauth.
• SSE events via hydramonitor: network state changes and correlation updates are broadcast as server-sent events, consumed by HydraPipeline and other services.
• Auto-updates from releases.experiencenet.com: the service checks for new releases and updates itself without manual intervention.

Troubleshooting

YAML corruption

If the YAML store becomes corrupted (partial writes, disk issues):

1. Stop the service: systemctl stop hydraneck
2. Check the data directory: ls -la /root/.hydraneck/
3. Look for malformed YAML files -- inspect the index and individual entity files
4. Restore from backup if needed, or manually fix the YAML syntax
5. Start the service: systemctl start hydraneck

Service won't start

1. Check logs for the error: journalctl -u hydraneck -n 50 --no-pager
2. Verify config exists and is valid: cat /root/.hydraneck/config.yaml
3. Check if the port is already in use: ss -tlnp | grep <port>
4. Verify the binary exists and is executable: ls -la $(which hydraneck)
5. Try running manually to see output: hydraneck serve

Auth token issues

1. Verify the token by calling hydraauth directly: curl -s https://hydraauth.experiencenet.com/api/v1/verify -H "Authorization: Bearer <token>"
2. Check that hydraauth is reachable from the neck server: ssh root@46.225.8.28 curl -s https://hydraauth.experiencenet.com/api/v1/health
3. If tokens are consistently rejected, check clock sync: ssh root@46.225.8.28 date -- token validation may fail if the server clock is skewed

Router polling failures

1. Check logs for polling errors: journalctl -u hydraneck --since "1 hour ago" | grep -i "poll\|router\|omada\|mikrotik"
2. Verify the router is reachable from the server: ssh root@46.225.8.28 ping <router-ip>
3. Check router credentials in the config: ssh root@46.225.8.28 cat /root/.hydraneck/config.yaml
4. For Omada controllers, verify the API endpoint is accessible
5. For MikroTik routers, verify the API or SSH credentials are correct

Node stuck in wrong correlation state

1. Check the node's current state: curl -s https://hydraneck.experiencenet.com/api/v1/nodes/<node-id>/connectivity -H "Authorization: Bearer <token>" | jq .
2. Trigger a manual venue scan to refresh: curl -X POST https://hydraneck.experiencenet.com/api/v1/venues/<venue-id>/scan -H "Authorization: Bearer <token>"
3. Check if the node's agent is running on the node itself
4. Verify the node is visible on the venue router's client list
5. If state is stale, restart the service to clear caches: ssh root@46.225.8.28 systemctl restart hydraneck

WireGuard tunnel down

1. Check tunnel status via API: curl -s https://hydraneck.experiencenet.com/api/v1/venues/<venue-id>/tunnels -H "Authorization: Bearer <token>" | jq .
2. SSH to the server and check WireGuard directly: ssh root@46.225.8.28 wg show
3. Verify the venue's WireGuard peer configuration is correct
4. Check if the venue's router has outbound connectivity to the WireGuard endpoint
5. Review firewall rules if the tunnel was recently configured

Backup & Recovery

What is backed up

Hetzner automated daily server snapshots are enabled on hydraneck (46.225.8.28), context hydraexperiencenet. Backup window: 02:00–06:00 UTC, 7-day retention.

The snapshot covers the entire server disk, including:

• /root/.hydraneck/ — venue network state, node correlations, router configurations
• /root/.hydraneck/bandwidth/ — per-venue bandwidth history (24h rolling window)
• /root/.hydraneck/config.yaml — service config and auth tokens

Restore procedure

1. In the Hetzner Cloud Console (hydraexperiencenet project), open Servers → hydraneck → Backups.
2. Power off the server, restore the snapshot, then power on.
3. Verify the service came back up:

   curl https://hydraneck.experiencenet.com/api/v1/health
   

4. Spot-check venue network state is intact:

   curl -H "Authorization: Bearer $TOKEN" https://hydraneck.experiencenet.com/api/v1/venues