High Availability

To start an ArcadeDB server with HA enabled, at minimum you need to:

Example starting a 3-node cluster on a single host (different ports):

The Raft gRPC port is 2434 by default and is configurable via arcadedb.ha.raftPort. The HTTP port in each entry is required so that replicas can forward non-idempotent requests to the leader over HTTP.

Each comma-separated entry in arcadedb.ha.serverList can be written in either of two interchangeable forms. The two forms may be freely mixed in the same list, and both accept the optional name@ prefix (see Named Peers).

The following two entries are equivalent:

A client protocol’s routing view tells a driver which node to send writes to and which nodes it may read from, so it deals in the ports clients dial - never the Raft port the cluster uses to talk to itself, and not necessarily the HTTP port either.

Declare bolt and grpc whenever your nodes do not all listen on the same port for that protocol. With the field omitted, a peer’s address for that protocol is derived as that peer’s host plus this node’s own port. That is correct only for a homogeneous deployment where every node listens on the same port - a Kubernetes StatefulSet, for example - and a one-time WARNING per protocol is logged whenever the fallback is used.

On a cluster like that one, leaving the ports undeclared makes every peer derive to the same address - and an address two peers both claim identifies neither of them, since two listening sockets cannot share one host:port. Rather than advertise a leader address that is really the address of the node answering (which would send a self-redirecting client straight back to the node that just refused it), ArcadeDB advertises nothing for that protocol: BOLT then offers this node as READ and ROUTE but never as writer, and a refused gRPC call falls back to naming the leader’s HTTP address. A WARNING naming the field to declare is logged once per protocol. (Since v26.9.1)

The same derive-from-this-node’s-port fallback applies to the endpoints the nodes use to reach each other: a follower downloading a snapshot from the leader, a cluster verify comparing checksums with a peer, and the database presence matrix (GET /api/v1/cluster?presence=true). When two peers resolve to one host:port, that address identifies at most one of them and nothing can say which, so ArcadeDB withholds it rather than guessing: the operation is refused and reported as refused - a peer comes back ERROR / "not verified" from a verify, and unreachable from a presence matrix - instead of quietly answering for the wrong node. (Since v26.9.1)

The refusal is visible in three places:

One request is exempt: the capability check, in which the leader asks each peer which replication formats its build understands so it can use a compact one only when every node can read it. That call changes nothing on the peer and its reply says who answered, so ArcadeDB dials the shared address anyway and credits the answer to the node that identified itself. A cluster that has never declared its http ports therefore still gets the compact formats, instead of falling back to the verbose ones forever with nothing to explain why. (Since v26.10.1)

On the common single-host layout where every node shares an address and the Raft and HTTP ports move in step (2434/2480, 2435/2481, …​), the capability check derives each peer’s HTTP port from that same offset, so it reaches the peers the shared-address fallback cannot name. The derived port is used for the capability check only - never for a snapshot download, a cluster verify or a forwarded write, which keep refusing a shared address. (Since v26.10.1)

A peer the leader still could not get an answer for carries capabilitiesUnknownReason next to its entry in GET /api/v1/cluster, saying whether it is unreachable, running an older build, or not identified by any address of its own - the last being the one you fix by declaring its http port. (Since v26.10.1)

By default, each node identifies its slot in arcadedb.ha.serverList from the numeric suffix of arcadedb.server.name (for example, ArcadeDB_0 maps to the first entry, ArcadeDB_1 to the second, and so on). Display names shown in logs and Studio are then synthesized from the local node’s prefix. This positional convention works well for Kubernetes StatefulSets but is awkward when nodes have human-readable names such as frankfurt, london, nyc.

You can add an optional name@ prefix to any entry in arcadedb.ha.serverList to give each peer a stable, human-readable name:

When peer names are configured, each node identifies its slot by matching arcadedb.server.name against the configured peer names first; if no match is found, it falls back to the legacy prefix_N / prefix-N suffix resolution. This means:

The Raft peer ID is still derived from the address (e.g. 10.0.0.1_2434), so changing or omitting peer names does not affect Raft identity stability.

ArcadeDB uses a leader/replica model with Raft consensus. At any time one server holds leadership and accepts writes; the others are replicas that serve reads and stand by for failover.

Each server persists its own Raft log segments under <rootPath>/raft-storage/. The log is used for recovery after restart and to replicate state to peers that fall behind.

Any read (query) can execute on any server in the cluster. All writes must go through the leader: a replica that receives a write transparently forwards it to the current leader via HTTP.

Internally, every committed write follows a replicate-first, commit-after three-phase commit:

If Phase 2 times out or fails, Phase 3 never runs: no local writes, no divergence. Multiple concurrent transactions are batched into fewer Raft round-trips via group commit (HA_RAFT_GROUP_COMMIT_BATCH_SIZE).

Phase 3 runs on the Raft apply thread of every node, the leader included, when the entry reaches its position in the log: the leader publishes the pages its transaction prepared, a replica applies the entry’s WAL bytes. Every node therefore writes its pages in log order, and the leader’s own commit can never overtake or trail the apply of a neighbouring entry.

Two transactions validated on different nodes against the same version of a page (for example two documents that share a page, each written by a different node) are both shipped to the leader with the same next version. The leader validates every entry, page by page, against the versions the log has already assigned, at the point that decides the log order, and refuses the second one before appending it: the originator receives a retryable ConcurrentModificationException, exactly as it would for a conflict on a single node, re-reads the page and retries. An entry is never merged onto another one region by region, so a node can neither lose an acknowledged write nor drift from its peers. While such an entry is in flight, a transaction on the leader that touches one of its pages fails its own Phase 1 with the same retryable error, without a round trip.

A quorum is the number of peers that must acknowledge a write before it commits. Configure it via arcadedb.ha.quorum:

If the configured quorum is not met within arcadedb.ha.quorumTimeout milliseconds, the transaction is rolled back and an error is returned to the client.

When a read runs on a replica, ArcadeDB offers three consistency levels, configurable per-server via arcadedb.ha.readConsistency:

Clients communicate the consistency contract through three HTTP headers:

The bookmark is a lower bound: it says the server had applied at least that index when it answered. That holds on a refused request too, so a client can keep its read-your-writes barrier after an error instead of falling back to a stale one.

The one endpoint where it is not a header is the streaming bulk load, POST /api/v1/batch/{database} with Accept: application/x-ndjson. There the response has already started by the time the commit index is known, and a header set at that point would be dropped silently, so the value travels in the body instead, as commitIndex on the terminal summary or error line. The Java driver handles this transparently: RemoteDatabase.getLastCommitIndex() advances the same way whichever encoding was used.

The Raft group-commit queue is bounded (arcadedb.ha.groupCommitQueueSize, default 10000 pending transactions). When a write arrives but the queue is full, the server waits up to arcadedb.ha.groupCommitOfferTimeout ms (default 100) for a slot, and if still full throws ReplicationQueueFullException — a NeedRetryException that clients automatically retry with backoff. This protects the server from OOM under sustained write overload and lets clients back off gracefully.

Set arcadedb.ha.serverRole=replica to pin a node as a permanent replica. The node’s Raft priority is set to 0 at startup so it is never elected leader, while still receiving all writes and serving reads. Useful for read-scale deployments or for cross-DC witnesses that should not take over as leader.

If the leader becomes unreachable, replicas start a new Raft election. A replica with an up-to-date log is elected as the new leader and the cluster resumes serving writes. Pre-vote prevents partitioned nodes from triggering disruptive elections.

Common causes of leader unavailability include:

When a replica rejoins after being offline, it catches up via the Raft log automatically; if it has fallen behind past the log purge boundary, the leader streams a full database snapshot over HTTP and the replica installs it atomically.

The leader also auto-recovers a follower’s replication channel that is wedged on a stale peer address — for example after a Kubernetes pod is rescheduled onto a new IP: it re-resolves the peer and re-establishes the channel on its own, instead of leaving the follower silently disconnected and the cluster at bare quorum until an operator forces a leadership transfer. See Replication-Channel Self-Healing.

A follower can be up, healthy, and reachable by every other node, and still receive nothing from the leader. The leader keeps one long-lived gRPC channel per follower. If that channel is wedged on a peer address that is no longer valid (the classic case is a Kubernetes pod restarting onto a new IP inside the JVM’s positive DNS cache window), the leader’s appender for that follower keeps writing into a socket that goes nowhere. The follower’s matchIndex stops advancing, or never advances at all (matchIndex=-1), while the rest of the cluster commits normally. Quorum is unaffected as long as a majority is still healthy, so the cluster keeps serving writes at reduced redundancy and nothing fails loudly.

ArcadeDB detects and repairs this on its own, in three escalating steps.

1. Detect: the follower is reported unreachable. The leader’s cluster monitor tracks the time since the last successful RPC to each follower. Once that exceeds arcadedb.ha.peerUnreachableThreshold (default 10s), the follower is reported unreachable. This is a diagnostic signal only: it does not change Raft membership, quorum, or the write path. It is, however, the input every step below depends on, so setting it to 0 disables the whole self-healing sequence.

2. Reset: rebuild that one follower’s channel. Once a follower has stayed continuously unreachable for arcadedb.ha.peerChannelResetDuration (default 60s), the leader closes that follower’s replication channel so the next send re-resolves DNS and reconnects. Only the unreachable peer’s channel is touched; the other followers, and leadership itself, are left alone, so there is no flapping risk. A follower that reconnects inside the interval clears the streak and is never reset, so an in-progress reconnection is not disrupted.

A single reset does not always stick, because the rebuilt channel may still resolve the stale address if the JVM DNS cache has not expired. The reset is therefore retried once per interval, up to 5 attempts. The counter re-arms as soon as the follower becomes reachable again. Set the duration to 0 to disable the automatic reset and keep only the manual leadership transfer.

3. Escalate: transfer leadership to rebuild the appender. The retry budget alone leaves a gap: the streak only re-arms when the follower becomes reachable, so a permanently wedged channel would sit in the given-up state until an operator restarted the leader process. arcadedb.ha.peerChannelResetEscalation (default true) closes it. When the budget is exhausted and the channel is still dead, the leader transfers leadership to a healthy peer, so the new leader builds a brand-new appender to the stuck follower.

The target is selected with the same rules as a manual step-down and is never the wedged follower itself. If no healthy peer is eligible, the leader logs at SEVERE and leaves the follower for the operator.

The escalation is bounded, not perpetual. Leadership transfer is exactly what makes a new leader inherit the problem, so each healthy peer escalates a given follower at most once per 30-minute cooldown. If the follower is unreachable for a reason a fresh appender cannot fix (it is genuinely down, or partitioned away), the cluster passes leadership around at most once per peer and then settles on the operator-intervention path instead of churning. Set the flag to false to keep the detection and logging but never move leadership automatically.

With ephemeral Raft storage, wiping the log on restart could leave the cluster in a broken state:

Durable storage lets a restarted node rejoin by replaying its own persisted log rather than always requiring a full snapshot resync from the leader.

Migration: the per-node directory is named raft-storage-<nodeName> dynamically, which container runtimes and Kubernetes cannot mount as a volume (a wildcard like raft-storage- is created as a literal directory, not expanded). Set arcadedb.ha.raftStorageDirectory to a *static path (e.g. /home/arcadedb/raft-storage) and mount that path on durable storage before upgrading. On Kubernetes, back it with a PersistentVolumeClaim — see Kubernetes. On a plain filesystem you can instead copy the existing raft-storage-<nodeName> directory across the upgrade. A throwaway or test cluster that intentionally wants ephemeral behavior can opt out with arcadedb.ha.raftPersistStorage=false.

When a write is forwarded from a replica to the leader (or issued directly to the leader), it is possible for the transaction to be committed cluster-wide by quorum and then fail to apply on the node that is answering the client. This is a distinct outcome from an ordinary failure: the data is durably committed, so retrying would create a duplicate.

This case has its own contract. The server raises a TransactionCommittedRemotelyException, mapped over HTTP to status 409 Conflict. A client that receives it must not retry the transaction — the write already succeeded cluster-wide; it should instead re-read to observe the committed state. This is different from a transient conflict (a genuine write-write conflict or DeadlockException), which remains safe to retry.

By default, when an HA cluster forms for the first time and one or more peers already hold a non-empty database on local disk, the peers do not download the database over HTTP from a single source. Instead, every peer reports a (fingerprint, lastTxId) tuple per database, the cluster elects the peer with the highest lastTxId as the source via leadership transfer, and any peer whose fingerprint matches the source bootstraps locally in seconds with zero bytes transferred.

This is the recommended path for scaling a single-instance ArcadeDB deployment up to a multi-node HA cluster after a large dataset has already been imported. Operator workflow:

Time-to-cluster goes from "minutes-to-hours of HTTP snapshot transfer per replica" to "seconds, because everyone already has the bytes". For the Kubernetes recipe, see Pre-staging the database on every pod.

The bootstrap protocol handles the following cases automatically:

The committed bootstrap baseline (fingerprint and lastTxId per database) is visible in the response to GET /api/v1/cluster and in the Studio cluster dashboard.

Cluster membership and leadership can be changed at runtime through dedicated REST endpoints. All endpoints require authentication as the root user.

Example: add a new peer at runtime.

Adding a peer with a human-readable name:

POST /api/v1/cluster/verify/{database} reports one status per peer and rolls them up into a single result.overallStatus:

ALL_CONSISTENT therefore means what it says: it is not reachable while any peer is unverified. An unverified peer is deliberately not rolled up as an observed divergence, which would send an operator hunting for something nobody saw.

ArcadeDB Studio exposes a Cluster tab (visible when HA is enabled) that displays the current leader, peer roles, Raft term and commit index, per-follower replication lag, and provides buttons for leadership transfer, peer add/remove, and database verification. See Studio.

When arcadedb.ssl.enabled=true, the server binds an HTTPS listener on arcadedb.server.httpsIncomingPort (default 2490) in addition to the plain HTTP listener on arcadedb.server.httpIncomingPort (default 2480). The two listeners use different ports.

Peer-to-peer transfers such as snapshot download and cross-node verify travel over HTTP. To send them encrypted, declare each peer’s HTTPS port in arcadedb.ha.serverList using the https field (object form) or the fifth positional field:

The peer certificate is validated against the configured trust store (arcadedb.ssl.trustStore).

If the https port is omitted on a homogeneous cluster (every node listens on the same HTTPS port, e.g. a Kubernetes StatefulSet), each peer’s HTTPS endpoint is derived automatically from the peer host plus this node’s local HTTPS port, so no extra configuration is needed. Only when no HTTPS endpoint can be resolved does the transfer fall back to plain HTTP, logging a one-time warning.

The previous section covers the HTTP side channels. This one covers the Raft gRPC port (arcadedb.ha.raftPort, default 2434), which carries AppendEntries, RequestVote, InstallSnapshot and the Ratis admin/client calls.

The two are independent axes. arcadedb.ssl.* does nothing for the gRPC port, and arcadedb.ha.tls.* does nothing for the HTTP listeners; a hardened cluster wants both. Nor does gRPC TLS replace the X-ArcadeDB-Cluster-Token check described in Security — that check still authenticates the HTTP forwarding and snapshot-download traffic.

TLS on the Raft port is off by default, so a development or test cluster still starts with no configuration at all. Turn it on with four settings:

One setting covers all three gRPC endpoints (admin, client, and server-to-server), so there is nothing further to switch on per message type. Mutual authentication is on by default; see Server-only TLS to relax it.

When arcadedb.ha.tls.enabled=true, all three file settings must name a readable PEM file or the node refuses to start, with a ConfigurationException naming the offending setting:

The paths are validated at startup rather than left to the TLS layer, so a typo or an unmounted Secret volume fails once and clearly, instead of surfacing later as an opaque handshake error on every peer connection.

The private key must be PKCS#8 — the PEM whose armor labels it simply PRIVATE KEY. A PKCS#1 key, which the armor labels RSA PRIVATE KEY instead and which several toolchains emit by default, is not accepted. Convert one with openssl pkcs8 -topk8 -nocrypt.

For Kubernetes deployments using the official Helm chart — including the StatefulSet + headless Service pattern, auto-join on scale-up, and the preStop auto-leave hook — see High Availability on Kubernetes in the Kubernetes guide.

The following settings control HA behavior. A complete list of all HA parameters is available in Server Settings.