Skip to main content

Run on Mesh

A hospital group often has data in several places that cannot be centralized. Run on Mesh lets you build one workflow and execute different parts of it at different sites. Data passes between sites through Connect.

How it works​

Each node in a workflow can carry a placement: the site where it runs. When you start a mesh run, the coordinator splits the workflow into segments, one group of nodes per placement, and each segment runs on the workflow engine at that site (a spoke). A segment's output travels to the next segment as a hop through Connect.

Run on Mesh topology: a coordinator, executors at Site A and Site B, a Connect hop between them, and status events returning to the coordinator

  • The coordinator is a role of the same workflow engine server (HYDRA_ROLE=coordinator, or the -coordinator flag). It compiles the plan, keeps a ledger of every run and segment in Postgres, and dispatches only the first (entry) segment. The spokes chain the rest.
  • Plans and envelopes carry Ed25519 signatures over canonical JSON. Signature verification defaults to enforcing.
  • Status events sent back to the coordinator are PHI-free by default: no per-node inputs or outputs, and errors are reduced to a closed set of codes.
  • Data that leaves a data-holder site through the seam is pseudonymised, not anonymised. It can still be personal data under GDPR.

What the mesh supports​

Owner statement

These capabilities are owner-confirmed available. See Capability status for the source and verification boundary, including older design records that remain unreconciled. Confirm the supported deployment with your administrator.

  • Joins that combine data, including PHI, from multiple sites.
  • Durable, exactly-once delivery between sites.
  • Federated learning across sites.

Run a workflow on the mesh​

  1. Open the workflow in Tiet · Studio → Workflows → Builder.
  2. Select a node and open the Runtime section of its inspector. In Runs on (placement), pick a site or type one. The field suggests the hub and every registered spoke. Leave it empty to run the node on the hydra-platform VM, outside the mesh.
  3. Repeat for the nodes that belong to other sites.
  4. Open the Run menu. Run on mesh (description "Distributed across sites") is now enabled. While no node has a placement it is disabled, with the hint "Set a node's Runs-on placement to enable".
  5. Choose Run on mesh. A dialog asks for inputs, then the workflow is saved and the run starts.

After the run starts the bottom panel opens a MESH tab. If you start from an empty panel it reads "No mesh run yet. Tag a node's placement and choose Run on mesh."

Run sequence: Studio, coordinator, Connect and two spokes

Read the MESH tab​

The tab polls the coordinator every 2 seconds until the run ends. It shows the run status, run id and plan id, then the segments grouped by placement. Each segment row shows its id, the spoke (@spoke), a ×N badge when it has been attempted more than once, its status, and a failure code when it failed.

Run statusMeaning
pending, runningIn progress
doneAll segments finished
failedA segment failed
stalledNo progress before a deadline (the default schedule, execution and join timeouts are 15 minutes)
revokedThe run's access policy was revoked and the kill switch stopped it at its next hop
cancelledStopped by a user

Segment statuses are pending, running, done, skipped, failed, dlq and cancelled. Per-node results appear under a segment only when the process is configured with HYDRA_MESH_STATUS_DETAIL=full. Data-holder sites never send them.

Using the Builder's stop control while a mesh run is active cancels it.

Where else to look​

PlaceWhat you see
Tiet · Connect → Network → MeshSites, spokes and their health
Tiet · Studio → Workflows → Builder, bottom panel MESH tabPer-segment status of the run you started

Tiet · Connect → Network → Mesh: the hub and registered spokes

Safety controls​

ControlEffect
Fail-closed authenticationEvery /api/mesh route except /api/mesh/health needs a bearer token. An empty token configuration refuses everything.
Signed plans and envelopesTampering is detected. A mesh process without its trust store refuses to start.
Placement-scoped tokensA site's token can fetch, report on and join only segments of its own placement.
Access-policy gateA run that spans more than one placement needs a published, unrevoked, unexpired policy that matches the pipeline, the compiled plan hash and the placements. Otherwise the start is refused with SEAM_POLICY_MISSING, SEAM_POLICY_REVOKED, SEAM_POLICY_MISMATCH or SEAM_REGIME_MIXED.
Kill switchRevoking a policy stops the runs it admitted (revoked).
PHI-free status eventsMonitoring does not expose patient data.

Limits​

  • Run on mesh is offered only for a saved workflow (one that has a pipeline id).
  • Cancellation stops further work but does not undo side effects at a site.

Something unclear? Send documentation feedback by email.