Docs · Running

Cluster and federation

Two ways to run actors on more than one machine. A cluster is a set of trusted nodes sharing one population of actors, placed by shard. Federation connects actors of different owners through relays nobody needs to trust. A chart cannot tell which case it is in: <send target="…"> looks the same for all of them.

Two separate problems

Both reuse the runtime's three layers: an address resolves through a directory to a mailbox, and transports deliver into mailboxes as SCXML Event I/O Processors (see the actor runtime). What differs is trust.

The cluster

The facade's cluster feature brings in vertep-cluster with its gossip, quic and runtime features. The etcd and Kubernetes authorities are separate features (etcd, kubernetes) that you enable on vertep-cluster directly.

Cluster metadata and shards

A cluster is fixed at creation by its ClusterMeta: an identifier, a shard count, a hash version and a replication factor, kept in cluster.meta. A node built with different values is refused when it joins.

• The shard count is a power of two. It bounds write parallelism, so choose it generously (256 or 1024). A power of two turns resharding into a split: under mod 2N, an actor in shard s lands in s or s + N.
• The hash is versioned and platform-independent. Version 1 is FNV-1a 64-bit over the IRI's UTF-8 bytes, the same hash vertep-tiramemsu uses for single-node shard files, so a single-node directory and a cluster agree on placement. Golden vectors are checked in CI on Linux and macOS, x86_64 and arm64.
• One shard is one journal store with one writer, typically one tiramemsu file. Placement is per shard, not per actor: the directory has N entries, not one per actor. A hot shard concentrates load on one writer; actors are not moved individually.

Membership

One Membership trait reports the current view, notifies subscribers, and grants, renews, releases and revokes shard leases. The runtime never depends on which implementation is configured. Views carry strictly increasing version numbers, and a node ignores a view not newer than the one it holds.

Joining checks the cluster id, shard count and node id. Leaving drains first: the node refuses new activations, finishes running macrosteps, flushes outboxes, ships each shard and hands it over. Failure detection marks a silent member suspect after 3 s and failed after 10 s by default (DetectorConfig), and a failed member's leases become free only after they expire.

Gossip lease granting is weaker than consensus under partitions. Fencing keeps it safe, at the cost of availability. A gossip leader's epochs carry a tag from its identifier and start, so a leader that restarts never grants an epoch twice; the fault-injection matrix found that bug and the fix has a test.

Leases and fencing epochs

A node owns a shard through a time-bounded lease. Every new grant carries a strictly greater fencing epoch; a renewal keeps the epoch. By default a lease lasts 10 s, and the owner stops starting macrosteps 2 s before it ends, by its own monotonic clock (LeaseConfig), to absorb clock drift and pauses.

At most one activation of a virtual actor can commit macrosteps under any epoch. Three checks give that, in order of cost:

1. Routing. A node activates an actor only while it holds an unexpired lease on its shard. Other nodes forward the message to the owner, with a hop limit.
2. Lease timing. The owner stops before expiry, so in the normal case two owners never overlap.
3. Fenced append. Under clock skew or a long pause, a stale owner's append carries an epoch lower than the one the journal store has accepted, and is rejected as Fenced. Its outbound frames carry the old (shard, epoch) too, and receivers drop them into the dead-letter graph. The append is also a compare-and-set on the actor's journal position.

Lean proves the invariant behind this: for every shard and epoch at most one writer's appends are accepted, and accepted journal positions are totally ordered across epochs (see Verification).

The cluster inside a node

A ClusterNode joins one runtime Node to the cluster. It is the node's placement, its storage and its cluster Event I/O Processor. Before the runtime delivers a message or activates an actor, it asks the placement, which answers:

• Local, when this node holds an unexpired lease on the actor's shard;
• Remote, a mailbox that frames the envelope and sends it with retries until it is acknowledged or the delivery deadline passes, after which the sender gets error.communication;
• Hold, when no live owner exists: the message waits in the local mailbox and the actor is not activated.

Every append goes through ClusterStorage, which routes it to the shard's store and stamps the node's lease epoch as the fencing token. The wiring, from crates/vertep-cluster/tests/rt/mod.rs:

let cluster = ClusterNode::new(membership.clone() as Arc<dyn Membership>, config);
wire.connect(&cluster);                       // MemoryWire in tests; QuicWire in production

let node = Node::builder(storage as Arc<dyn Storage>, Shell::tokio())   // a ClusterStorage
    .datamodels(Arc::new(vertep_dm_ecma::EcmaRegistry))
    .placement(cluster.placement())
    .transport(cluster.processor())            // <send type="cluster">
    .build();
cluster.attach(node.handle());

ClusterConfig holds the transport settings:

The QUIC transport

Nodes talk over QUIC with mutual TLS. Every node has a certificate from the cluster's CA for <node>.<cluster>.vertep.internal, and a server accepts a client only if the certificate chains to the cluster CA and names a current member, so a foreign certificate or a removed member is refused during the handshake.

• Ordering. Frames of one sender and receiver pair travel on one stream, in order; other pairs use other streams and never wait for it. Past a stream cap per peer, pairs share streams by hash, still in order per pair.
• Backpressure. The receiver reads the next frame of a stream only after the previous one is stored in a mailbox, so a full mailbox slows exactly the streams that feed it. Accepted messages are never dropped.
• Delivery. At least once: the sender retries until the receiver confirms the message is stored. Virtual actors deduplicate by the journal's message ids, other receivers by a bounded id cache. A frame is at most 1 MiB by default.
• Stale routing. A node that is no longer a shard's owner forwards the message and sends a redirect, so the sender updates its table.

Worker pools

Stateless workers are pooled, never sharded, and never take a lease. Any node that declares a worker type may run activations of it. A request is routed by the sending node: it picks an activation by policy, creates one on a declaring member when none has capacity, queues within a bound, or refuses at once with an error event.

Results are journaled by the caller, so replay never calls the worker again. A worker node that fails fails its requests to their callers as error events or request timeouts. A request is retried once on another activation only if the worker type is declared idempotent, because a retried LLM or tool call can have outside effects.

Replication and moving shards

With a replication factor above one, every shard has rf - 1 standbys: the members after the owner in the sorted list of active members. The owner ships to its standbys what they lack and acknowledges an append only after the configured number of them stored it. If the quorum is not reached, the append fails, and the actor's journal is neither read nor extended until a later shipment catches the standbys up. So an entry that missed the quorum never produces effects, and a standby that takes over holds every entry whose effects anyone saw. A standby refuses a stream from an epoch lower than one it has accepted.

With replication factor 1 there is no standby. A failed owner's shards stay unavailable until its disk returns, and no other node may take such a shard with an empty journal. Messages wait in mailboxes rather than being lost.

To move a shard, the rebalancer drains the old owner, ships the journal suffix the new owner lacks, verifies the end position, and transfers the lease at a higher epoch. Late messages at the old owner are forwarded with a redirect. With a shared journal store a move is a lease transfer only.

Resharding is offline. vt cluster reshard <dir> <out> doubles the shard count of a stopped cluster: it reads the data directory (cluster.meta, the shard files and the leases dump), refuses if the cluster is running, re-hashes every actor IRI into shard s or s + N, and writes the new files and metadata to <out>.

Fault injection

Two harnesses check the cluster. tests/faults.rs runs in process over static, gossip and a stand-in for the external authorities on simulated time, and over etcd and Kubernetes in containers, each with a shared journal and with replication factors 1 and 3. tests/containers.rs runs every node as its own process (the vertep-fault-node binary) and injects network disconnects, pauses, kills and restarts, a lease clock at half speed, and lost frames. A checker requires every actor journal to be linear, at most one writer per shard and epoch, every acknowledged message processed exactly once, and no fenced side effect accepted. Every history is replayed through the Lean lease model, which must reach the same accept and reject decisions.

The nightly CI job runs:

VERTEP_CONTAINER_TESTS=require \
  cargo test -p vertep-cluster --release --features differential,etcd,kubernetes \
  --test faults --test simulation --test runtime --test external --test containers -- --test-threads 2

Without VERTEP_CONTAINER_TESTS the container tests skip; =1 runs them when Docker is available and =require fails without it. Set VERTEP_HISTORY_DIR to keep the histories as JSON lines.

Federation

The facade's federation feature brings in vertep-federation with DIDComm v2 and TSP. Actors on ordinary IRIs never touch any of it: their sends are delivered without DID resolution, keys or cryptography.

DID identities

A DID is optional and per actor. An actor that crosses a trust boundary uses its DID as its address, so the DID is also the subject of its journal and current-state graph. A DID resolver acts as the directory: the DID document's services become routes, and its verification methods become keys.

Resolution is cached with a bounded size and time to live, and re-resolved on expiry or endpoint failure. Applications can register or override method resolvers in the ResolverRegistry.

Private keys stay behind the KeyManager trait (InMemoryKeyManager, FileKeyManager) and never enter journals, snapshots, the graph or datamodels: a never-forget store could only erase them by destroying the key that encrypts them. Rotating a key keeps the DID, with a grace period for retired keys. Federation::provision creates an actor's keys and DID and binds the DID to a chart type and key on the node.

Authenticated senders and grants

An inbound message from across a trust boundary passes these checks before any journal append, in order:

1. decrypt it with the KeyManager;
2. verify the sender against its resolved DID document (an anonymous message is refused unless the receiver's AnonymousPolicy is Allow);
3. find a grant for this sender, target and event;
4. check the payload against the receiver's declared event type;
5. drop it if its message id was already seen.

An admitted event carries the sender's DID in _event.origin and _event.originverified set to true; an anonymous one has neither. The verification outcome is journaled with the event, so replay never resolves a DID or checks a signature again, and the pure core sees only the origin and one boolean. A refused message is recorded in the graph <actor>/inbound, acknowledged to the relay, and answered with error.communication when the sender is verified and had contacted the actor first. An ungranted message never costs the receiver a journal append.

Grants are checked at both ends. A grant names a grantee, a target and SCXML event descriptors, with an optional expiry:

// from crates/vertep-federation/examples/two_orgs.rs
a.fed.grant_outbound(Grant::new(buyer.did.as_str(), shop.did.as_str(), ["order.*"]));
b.fed.grant_inbound(Grant::new(buyer.did.as_str(), shop.did.as_str(), ["order.*"]));

A grant can also arrive as a verifiable credential from a trusted issuer (accept_credential). Grants are published into the actor's identity graph, <did>/identity, so "who may talk to this agent" is a query:

PREFIX vt-id: <https://w3id.org/vertep/vt-id#>
SELECT ?grantee ?event ?until WHERE {
  GRAPH ?g {
    ?grant a vt-id:Grant ;
           vt-id:target <did:webvh:example.org:agents:billing> ;
           vt-id:grantee ?grantee ;
           vt-id:event ?event .
    OPTIONAL { ?grant vt-id:expires ?until }
  }
}

Chart guards can still decide more finely on top of grants, for example accepting approve only from one DID.

Relay mailboxes: DIDComm and TSP

DIDComm v2 mediators and TSP endpoints serve as mailboxes and transports, so an actor can receive while deactivated or offline. Payloads are encrypted end to end: a relay sees routing data only, never the event name, payload, send id or origin.

• DIDComm v2. The sender authcrypts (or anoncrypts) to the recipient and wraps one forward message per routing key. The receiving node picks up in batches. A remote mediator is bound through Mediator Coordination 2.0 and Message Pickup 3.0.
• TSP. The actor's DID maps to a TSP identifier; messages go direct when the endpoint is reachable, or routed through intermediaries, nested so that only the final recipient opens them.
• Acknowledge after the append. The receiving mailbox acknowledges a message to the relay only after the turn that processed it is durable, or when it drops it as a duplicate. A crash causes redelivery, never loss.
• Transport choice. The relay outbox tries the DID document's services in preference order, TSP first and then DIDComm by default, with fallback. <send type="tsp"> or type="didcomm" restricts a send to one transport.
• Undeliverable. Unreachable recipients are retried with backoff (by default 5 attempts, starting at 1 s and doubling up to 60 s). After that the sending chart receives error.communication with the send id and the reason, without ever blocking.

Wiring a node, as the module documentation of crates/vertep-federation/src/federation.rs sketches it:

let fed = Federation::builder(resolver, network).protocol(didcomm).pickup(binding).build();
let node = Node::builder(storage, shell)
    .mailboxes_for_scheme("did", fed.mailboxes())     // local DID actors' relay mailboxes
    .transport(fed.transport(TransportKind::DidComm)) // <send type="didcomm">
    .build();
fed.attach(&node.handle());                         // did: resolver, wake-ups, error path
let did = fed.provision(&node.handle(), chart_type, key, &config, km)?;  // DID as address
loop { node.run_until_idle(); fed.pump(); }

The example crates/vertep-federation/examples/two_orgs.rs runs two organisations' nodes and an in-process test mediator. The buyer's <send> to the shop's did:peer DID is resolved to the mediator, authcrypted and forwarded; the shop's node picks it up, verifies the sender, checks the grant and journals the event before acknowledging it:

cargo run -p vertep-federation --example two_orgs --features didcomm,testkit

shop journaled order.placed from did:peer:2.Vz6MksFJPi77yVK2k5KeaVZQNqqFneW7JKC3r3S8pYjwyUHmx.Ez6LSiZjqpMXyzKGQPeDC4pCw8n3iovG3ibdtvHFABppNvQhW.SeyJpZCI6IiNkaWRjb21tIiwicyI6eyJhIjpbImRpZGNvbW0vdjIiXSwiciI6WyJkaWQ6cGVlcjoyLlZ6Nk1rcEJXYVdhcTR3REZCb29WeHFIY0xyaUxxcUM1SkVtR04zdVdWTWJ5V0hzMVkuRXo2TFNyUFpVREhoWnhOa0VRZjljdWtpcnJZVmUzeFJraU13eHhEeDVoRHFwcHd2ei5TZXlKcFpDSTZJaU5rYVdSamIyMXRJaXdpY3lJNmV5SjFjbWtpT2lKb2RIUndjem92TDIxbFpHbGhkRzl5TG1WNFlXMXdiR1V2Wkdsa1kyOXRiU0o5TENKMElqb2laRzBpZlEja2V5LTIiXSwidXJpIjoiaHR0cHM6Ly9tZWRpYXRvci5leGFtcGxlL2RpZGNvbW0ifSwidCI6ImRtIn0 via https://w3id.org/vertep/vt-id#DIDCommEventProcessor (send id Some("o-1"))
mediator still holds 0 message(s)

The DID differs on every run, because the example generates fresh keys. The event's origin is the buyer's verified DID, its origin type is the DIDComm Event I/O Processor, and the mediator holds nothing once the shop's turn is durable.

What each failure costs

Safety never depends on clocks or relays behaving well; only availability does.

The same core also runs in the browser, linked to a server node over a WebSocket transport. That is the subject of the next page, Browser and JavaScript. The design notes are in tg.md/views/distribution.md; the requirements in openspec/specs/cluster-membership, shard-placement, cluster-transport, worker-pools, did-identity and relay-mailboxes.