Active/Standby and Failover

Overview

Cluster mode lets you keep a second server ready to take over your node. People keep using the same Safe URL and credentials after you move service to the second server. You do not create a separate Safe on each server.

Server	What it does
Active	Serves your Safes and sends encrypted copies of its data to the standby.
Standby	Receives those copies and waits. It does not serve Safes until you explicitly promote it.

There is one active and one standby. This is not load balancing, automatic failover, or a way to combine the storage of several servers. If the active fails, you prevent it from coming back online, promote the standby, and point the permanent Safe URL at the replacement.

Warning

Copies are sent asynchronously: the active can save a change before the standby receives it. An emergency takeover can lose recent changes. A synchronized standby is not an independent backup, and it can receive accidental deletions too. Keep independent backups. See Backup and Recovery.

Should you use cluster mode?

One server is the standard installation. Start with one server. Consider a standby when you can manage two separate servers, HTTPS on both, private file transfers between them, and moving your node's public URL during a failure. A second container on the same server does not protect you from that server failing.

Pairing is performed with command-line maintenance commands. The standard run-node.sh launcher creates or starts an ordinary node and has no standby installation option. Running the ordinary installer on two hosts creates two separate nodes, not a cluster. The admin console verifies recovery and performs failover.

What to prepare

Two supported Linux servers with separate local storage. Do not share a Docker volume or database directory between them.
One permanent user URL, for example https://node.example.com.
A separate admin address for each server, for example https://active.node.example.com and https://standby.node.example.com. Each needs working DNS, HTTPS and a valid certificate.
The same passkey domain on both servers, node.example.com in this example. This is the WebAuthn RP ID. Include the permanent URL and both admin URLs in the exact allowed origins. See Configuration.
A working admin recovery security key and a verified offline backup of the active.
Control of DNS or the reverse proxy that directs the permanent URL, and a way to power off or disconnect the old active before takeover. Changing DNS alone does not stop it running or using its Tor identity.

Safe users always use the permanent URL. The per-server admin addresses let you reach each server even when the permanent URL points to the other one.

Use the same image on both servers

Both members must run the exact same image. A mutable tag such as latest can select different builds if a release is published between downloads.

On the active host, save its complete image reference:

sudo docker image inspect "$(sudo docker inspect unolock-node --format '{{.Image}}')" \
  --format '{{index .RepoDigests 0}}' > cluster-image.txt
cat cluster-image.txt

The file should contain one reference including @sha256:.

Transfer cluster-image.txt to the standby host through an authenticated channel, such as SSH/SFTP with the host identity checked.

On each host, from the folder containing that file, run:

export UNOLOCK_CLUSTER_IMAGE="$(cat cluster-image.txt)"
export UNOLOCK_CLUSTER_DIGEST="${UNOLOCK_CLUSTER_IMAGE##*@}"
sudo docker pull "$UNOLOCK_CLUSTER_IMAGE"

Use UNOLOCK_CLUSTER_IMAGE as the container image on both hosts. The CLI's --image-digest argument uses UNOLOCK_CLUSTER_DIGEST. Do not use Docker's local image ID as the cluster image digest: the local ID and registry digest identify different objects.

Pair the active and standby

The following commands use the unolock-node CLI. They are maintenance commands for a stopped node. For Docker, run them in a maintenance container using --entrypoint unolock-node, the selected UNOLOCK_CLUSTER_IMAGE, and the right host's data volume. File paths are paths inside that maintenance environment. Mount your protected transfer directory at /secure-transfer.

These commands do not configure a reverse proxy, DNS routing or a standby container for you.

1. Prepare the active and initialize both members

Before enabling cluster mode, set the active up as a standalone production admin-unlock node:

Register the first Administrator, unlock the node, and confirm the Administrator key can unwrap the node secrets.
Complete the Deploy, Client and Health checks at the permanent origin.
Register and test a separately stored Administrator recovery credential.
Create and verify a separately protected node backup.

A node with an active bootstrap code wrapper cannot be converted into an active cluster member.

Then stop the active and initialize its cluster identity once:

unolock-node cluster:active-init \
  --config /data/unolock-node/config.json \
  --management-url https://active.node.example.com \
  --image-digest "$UNOLOCK_CLUSTER_DIGEST"

On a different host with a new or completely empty private data directory:

unolock-node standby:init \
  --data-dir /data/unolock-node \
  --management-url https://standby.node.example.com \
  --image-digest "$UNOLOCK_CLUSTER_DIGEST"

2. Exchange and confirm the pairing files

Transfer standby-member-state.json to the stopped active through an authenticated channel, then create an offer:

unolock-node cluster:pair-offer \
  --config /data/unolock-node/config.json \
  --standby-state /secure-transfer/standby-member-state.json \
  --output /secure-transfer/pairing-offer.json

Transfer the offer to the standby. Read the one-time secret printed by standby:init at a prompt, so it does not go into shell history. The secret expires after 15 minutes:
```
read -r -s -p 'Standby pairing secret: ' UNOLOCK_PAIRING_SECRET
printf '\n'
```

Accept the offer on the standby:

unolock-node standby:accept \
  --data-dir /data/unolock-node \
  --pairing-secret "$UNOLOCK_PAIRING_SECRET" \
  --offer /secure-transfer/pairing-offer.json \
  --output /secure-transfer/accepted-transcript.json

The CLI takes the secret as a command-line argument, so use a trusted maintenance host. Unset UNOLOCK_PAIRING_SECRET after acceptance succeeds.

Compare the complete eight-word confirmation phrase on both hosts.

Transfer the accepted transcript to the active and complete pairing:

unolock-node cluster:pair-complete \
  --config /data/unolock-node/config.json \
  --transcript /secure-transfer/accepted-transcript.json \
  --confirm-phrase '<exact-eight-word-phrase>'

If the exchange is interrupted, revoke the pending offer on the stopped active with cluster:pair-cancel --confirm-destination-member-id <uuid>. Erase that standby before reusing it.

3. Start both members

Start the active through its configured startup path.
Start the paired standby with standby:start --data-dir /data/unolock-node --config <standby-startup-config>, or the start path with a correctly configured paired standby data root.

The standby's startup configuration must keep the pairing's permanent origin and RP ID. The standby serves its restricted admin and replication surface at its own management URL. Do not initialize it as an independent node.

The standard container's bundled Caddy configuration serves the single hostname selected by UNOLOCK_DOMAIN. It does not prepare both cluster hostnames or switch its certificate after promotion. Your cluster's HTTPS proxy and certificate configuration must handle the permanent name on the replacement host.

Check that the standby can take over

Open the standby management origin, for example https://standby.node.example.com/admin.
Sign in with an Administrator recovery key.
Choose Verify Standby Recovery.

The standby validates the newest recovery point in isolation and then stays locked. It does not start Safe, federation, Tor publication or background services.

Do not rely on the standby until recovery verification succeeds. Status must show:

Status	Meaning
promotion-safe recovery point	Database changes have been checked
archive-ready point	The files those changes need have arrived
verifier sequence	Admin credential state

Receiving bytes alone does not prove the standby can recover them.

Prepare DNS before you need it

Do this before an incident:

Confirm you can sign in to the account that manages your domain's DNS, and store its login and recovery method somewhere available when the active is down.
Record both servers' public IP addresses and the current DNS records, including IPv6 if you publish it.
Arrange HTTPS for the permanent Safe name on the replacement server or its front-end proxy, as well as HTTPS for its own admin name. A certificate for standby.node.example.com alone does not cover node.example.com.
For direct DNS, use a short TTL for the permanent name, such as 300 seconds if your provider supports it. Lower it at least one previous TTL in advance: if it was 3,600 seconds, allow at least an hour. A 300-second TTL does not guarantee every user switches within five minutes.
Test standby recovery through its admin address, and arrange how you will stop the old host and prevent automatic restart.

If you use automatic certificates, a fresh certificate requested through HTTP or TLS validation may need the public DNS switch first, which adds downtime. To test HTTPS before switching, prepare a valid certificate through your certificate-management process.

The admin names identify physical servers. In this example (the IP addresses are reserved for documentation):

DNS name	Purpose	Before takeover	After takeover
`node.example.com`	Permanent URL everyone uses for Safes	Old active: `198.51.100.10`	New active: `198.51.100.20`
`active.node.example.com`	Admin address for the original server	`198.51.100.10`	Keep `198.51.100.10`
`standby.node.example.com`	Admin address for the second server	`198.51.100.20`	Keep `198.51.100.20`

Planned failover

Use this when both servers are available, such as before host maintenance.

On the active, choose Prepare Planned Switchover.
Wait for the signed final cycle and zero-lag handoff to appear.
Fence the old active: stop it and prevent it from restarting or using the node's Tor identity. Power off or disconnect the host and remove it from routing. Changing DNS alone is not enough.
On the standby management origin, choose Fail Over to Standby.
Enter the fencing attestation and confirm the displayed target generation.
Wait for promotion to complete. Confirm the cluster status reports the new active role and check node health. If promotion fails, stop and keep the error. Do not send Safe traffic to an unpromoted standby.
Move the permanent URL and verify the new route.

Emergency failover

Use this if the active has failed and cannot finish a planned handoff. Recent changes not yet copied to the standby may be lost.

Fence the old active first, even if it appears unreachable. An unreachable page is not evidence that the host has stopped.
At the standby management origin, choose Fail Over to Standby.
Acknowledge that the unreplicated tail can be lost, enter the fencing attestation, and confirm the target generation.
Wait for promotion to complete, then move the permanent URL and verify the new route.

Promotion automatically uses the newest contiguous promotion-safe point. There is no row, WAL or timestamp selection and no merge of divergent history.

The promoted member keeps its existing administrator credentials. Restarting it still requires administrator sign-in and explicit unlock.

Move the permanent URL

Find out whether your permanent name points directly to a server or goes through a proxy or load balancer. Use the direct DNS steps or the proxy steps, not both, unless your deployment requires both. Neither promotion nor recovery verification edits your DNS for you.

Direct DNS

Sign in to your DNS provider and open the domain's DNS records.
Find the existing A record for the permanent Safe name. Check the full name the provider shows before saving.
Change its value from the old active's public IPv4 address to the promoted server's, for example 198.51.100.10 to 198.51.100.20. Save it.
If the permanent name has an AAAA record, update it to the new server's reachable IPv6 address. If the replacement has no working IPv6, remove the old AAAA record.
Check that no other A or AAAA records for the permanent name still point to the old host. Keep each server's own admin records. Leave unrelated records alone.
Record the change time and new values. Do not change nameservers, use a web redirect, or give users a new Safe URL.

If the permanent name uses a CNAME, edit its target instead, for example from active.node.example.com to standby.node.example.com. Do not add conflicting A or AAAA records alongside it. HTTPS must still cover node.example.com.

Reverse proxy or load balancer

If DNS points to a fixed proxy, the public DNS record can stay unchanged. In the proxy configuration:

Remove or disable the old active backend.
Select the promoted server as the only active backend for the permanent Safe hostname.
Keep HTTPS for the permanent hostname and the expected upstream hostname, certificate validation and Host-header settings. Do not disable TLS validation to make the switch work.
Apply the configuration and check the proxy's backend health result. Requests already in flight may fail and need to be retried.

If your DNS provider proxies traffic, update its origin or backend setting according to that provider's procedure. Keep the existing proxy mode during takeover.

Verify the new route and your Safe

Run these checks from a machine outside the server, using your own hostname.

Check both DNS record types:

dig +noall +answer node.example.com A
dig +noall +answer node.example.com AAAA

To compare with a public resolver (this sends your hostname to that resolver):

dig @1.1.1.1 +noall +answer node.example.com A
dig @1.1.1.1 +noall +answer node.example.com AAAA

Check HTTPS directly on the new server without waiting for DNS caches, replacing the example IP with its public IPv4 address:
```
curl --connect-timeout 10 --max-time 30 \
  --resolve node.example.com:443:198.51.100.20 \
  --head https://node.example.com/admin
```
It should complete without a certificate error and return HTTP 200. Do not add -k or bypass certificate warnings.
Open https://node.example.com/admin, sign in, confirm this is the promoted active with the expected cluster generation, and refresh Diagnostics.
Open https://node.example.com, and sign in to an existing Safe with its existing credential and PIN. Do not create a new Safe to work around a routing or authentication error.
Check a known note, upload and download a small test file, and test cross-node Vault Messaging if you use it.
Check from a second network if possible. Keep the old active fenced even if some users still have cached DNS answers.
Record promotion status, DNS or proxy change time, verification results and any acknowledged data-loss window.

After failover

The old member can never rejoin. Erase the former active before you reuse it as a new standby. Do not restart it with its old data, and do not point traffic back to it.

If the promoted server fails after accepting new changes, keep the old server isolated and diagnose or recover the new active. Reversing the DNS edit to the old data would create conflicting histories.

Remove or replace an unavailable standby

On the unlocked active, choose Remove Unavailable Standby.
Type the complete member ID and confirm.
Restart the active when prompted.
Erase the removed host, then repeat the standby pairing to replace it.

The active waits for existing changes and replication to finish, retires the old replication state, and keeps hosted-client assets, signing trust and administrator credentials.

Upgrading a cluster

Do not apply the standalone upgrade steps independently to both members. Pairing and replication records contain the exact image digest, so downloading latest on each host does not update those records or prove compatibility.

Select the new release's exact image once and use it for both members. Follow the release's cluster upgrade procedure, including backup, stopping services, any required re-pairing, recovery verification and a failover test before relying on the updated pair. There is no cluster:upgrade command.

Restoring a clustered backup

A restored database that contains cluster history but lacks its matching signed host authority fails startup with CLUSTER_RECOVERY_ISOLATED. It cannot silently start as a standalone node.

Keep the restored host off the permanent user origin and keep the original backup unchanged.
Fence both former members: power them off or disconnect them from user, federation, DNS, proxy and Tor traffic.
Start the restored node only on its HTTPS management origin.
Sign in with an Administrator recovery key and try Unlock Node. The node stays locked and shows Recover Cluster Backup once it confirms the isolated cluster history.
Enter the exact management origin and immutable image digest, describe how every former member was fenced, and accept possible loss of changes made after the backup.
The node creates a fresh member identity and revokes persisted admin sessions before any public or background service starts.
Verify the new unpaired active status. Move the permanent origin only after validation. Erase every former member.

To add a standby afterwards, do not run cluster:active-init again. Start with standby:init on a fresh, empty standby and continue the pairing exchange above.

Troubleshooting failover

Symptom	Next check
DNS still returns the old address	Confirm the edit was saved at the provider that actually hosts your DNS. Check the previous TTL and other caches. Editing the record again does not clear cached answers.
Some users connect and others cannot	Check AAAA as well as A, leftover old addresses, CNAME targets and client or network caches.
Certificate warning at the permanent URL	Check which server or proxy answered and whether its certificate covers the permanent name. A working standby admin certificate is not enough.
DNS resolves but the connection times out	Check the provider firewall, host firewall, port forwarding, proxy routing and TCP 80/443 on the new host.
Admin loads but the Safe does not	Check that promotion completed, the node is unlocked, Diagnostics, and hosted-client signing status.
Passkeys fail after the switch	Confirm the original permanent hostname, RP ID, allowed origins and replicated admin and Safe state. Do not register replacement credentials as a routing fix.

FAQs

Does the standby take over automatically?

No. Failover is manual. You fence the old active, choose Fail Over to Standby on the standby, and move the permanent URL yourself.

Do users need new passkeys or a new URL after failover?

No. Users keep the same URL and credentials. The permanent URL and passkey domain stay the same.

Is a standby a replacement for backups?

No. Replication is asynchronous, so recent changes can be lost in an emergency, and the standby receives accidental deletions too. Keep independent backups.

Can I reuse the old active as the new standby?

Only after erasing it. The old member can never rejoin with its old data. Pair it as a fresh standby.

Can I run the active and standby on the same server?

No. A second container on the same server does not protect you from that server failing, and the two members must not share a Docker volume or database directory.

Back to Self-Hosted Overview