Cluster¶

Status: Production-Ready Last Updated: November 11, 2026

Overview¶

MaxIOFS provides complete multi-node cluster support for high availability (HA) and automatic failover. Multiple MaxIOFS instances work together as a unified storage cluster: every node holds every bucket and serves any request, with automatic health monitoring and failover. In v0.9.0-beta, cluster sync was extended to 6 entity types with tombstone-based deletion sync to prevent entity resurrection.

Key Features¶

✅ Multi-node cluster support: any node serves any request
✅ Dedicated cluster port 8082 — inter-node traffic is fully separated from S3 (8080) and console (8081)
✅ Inter-node TLS encryption — automatic, zero-configuration (v0.9.1)
✅ HMAC-authenticated node-to-node replication
✅ Synchronous replication (factor 1–3): a factor of 2 is a mirror that keeps writing with its peer down; a factor of 3 needs one of its two peers
✅ Parity (Reed-Solomon erasure coding across nodes): each object is cut in data and parity shards, one per node; up to 4 nodes it survives 1 node down, from 5 nodes 2; 3 nodes of 1 TB hold 2 TB
✅ Storage classes: groups of nodes keep the data of a storage class (GLACIER, DEEP_ARCHIVE…), each class with its own factor or parity; lifecycle transitions move the objects to a colder class
✅ A node that comes back is caught up at once with the writes and deletes it missed
✅ Ciphertext replication with a cluster-shared KEK; retention and legal hold travel with the object
✅ Elected coordinator for configuration changes; a surviving node takes over
✅ Automatic synchronization for 6 entity types (users, tenants, access keys, bucket permissions, IDPs, group mappings)
✅ Tombstone-based deletion sync (prevents entity resurrection)
✅ Health monitoring (30-second intervals)
✅ Every node holds every bucket; each object's copies go to the least full nodes, whatever the factor
✅ Partitions and long absences — writes and deletions are ordered by time; a deletion is kept until every node has it
✅ Web-based cluster management dashboard

Use Cases¶

High Availability - Automatic failover if primary node fails
Geographic Distribution - Nodes in different regions for low latency
Disaster Recovery - Replicate data to backup nodes
Load Balancing - Distribute requests across healthy nodes
Zero-Downtime Maintenance - Update nodes without service interruption

What happens when a node fails¶

A cluster has two planes, and it is worth knowing how each one behaves.

The data plane (S3 API, port 8080) needs no coordinator at all. Every node serves reads and writes on its own. A write's data is copied to the factor's nodes and its entry to every other healthy node before it is acknowledged (see Objects), and half of the factor's copies, rounded up and counting the local one, must hold the data. A factor of 2 is therefore a mirror, like RAID 1: with its peer down, the survivor keeps accepting writes on its own copy. A factor of 3 needs one of its two peers, so it keeps writing with one node down and answers 503 ServiceUnavailable (Retry-After: 30) only with both down.

A factor of 1 is like RAID 0. Each object has one copy: on the node that took the write, or on the least full node when that node is under storage pressure. The space of every node is used before any node runs out. While a node is down, reads of its objects answer 503 ServiceUnavailable with Retry-After; when it is back they are served again. An object whose node is lost is lost. For redundancy, use a factor of 2 or 3, or parity.

Parity is like RAID 5, and from five nodes like RAID 6. Each object is cut in data shards and parity shards, each on its own node: 2+1 on three nodes, 3+1 on four, 3+2 on five, 4+2 on six, up to 8+2. Any of its shards as many as its data shards read it: up to four nodes an object survives one node down, from five nodes two. Three nodes of 1 TB hold 2 TB of objects, against 1.5 TB with a factor of 2 and 1 TB with a factor of 3. A read takes the data shards; a shard that does not answer is replaced by a parity shard and its bytes are computed. A write needs another node that takes data: with nodes down it is cut in fewer data shards, and short of three nodes it is kept in whole copies, its parity shards plus one. The copy repair cuts them in shards once the nodes are back, at its next pass: after each full anti-entropy cycle, and when a node or the protection changes. With no other node taking data it is answered 503. An empty object is kept in whole copies.

A node that missed writes is caught up as soon as it is back. The node that accepted them records, per peer, the time of the earliest write that peer missed. When a health check finds the peer healthy again, it sends the peer every bucket (see Buckets), compares every key modified since then with the peer and sends what differs — the entry of every version and delete marker the peer lacks, the data of those the peer holds, and the current object when it differs, lock state included — then sends the deletes the peer missed. A delete of a whole key is sent only if the peer's copy is not newer than the delete, so a key written on the other side of a partition is kept. What fails is recorded again and retried at the next health check. The catch-up runs even with the periodic scrub disabled.

Metadata-only changes — tags, ACLs, retention, legal hold, user metadata, restore status — reach every live node before the request returns, so a client's successive changes arrive in order. A node that is down, fails, or still has changes waiting gets the change queued behind them, in SQLite, and the queue is delivered in order when the node is caught up. A change the node refuses (the object is gone, or its lock state no longer allows it) is dropped and logged; any other failure keeps the queue for the next attempt. With the periodic scrub enabled (the default), a node that restarts also runs a full anti-entropy cycle within an hour of starting; it compares keys the same way. A version or delete marker a node deleted is not stored again when a node that missed the delete sends it, or gives it, a copy.

The control plane (web console, port 8081) elects a coordinator, so that two nodes cannot edit the same entity at the same instant and quietly disagree about the result. Configuration — users, access keys, policies, roles, tenants, bucket permissions — is written on the elected node; a change made on any other node is forwarded to it, which is invisible to whoever is using the console. A bucket's own settings and objects are on every node: console requests about them are served by the node they reach and do not need the coordinator.

When the coordinator dies, the nodes that are still alive hold an election and one of them takes over. The majority is counted over the nodes that answer, not over every node the cluster has ever known, so a node that is gone does not get a vote in whether the cluster may continue. In a two-node cluster the survivor elects itself within about twenty seconds — one lease — and from then on it is the coordinator: every configuration change works normally on it.

When the failed node comes back, it comes back as a member. It finds a live lease under a newer term, follows it, and the synchronisation managers bring it up to date with everything that changed while it was away. Nothing needs to be done by hand, and nothing has to be removed or re-joined. If you do want to decommission it permanently, remove it from the console in the ordinary way — the surviving node is a coordinator, so that works too.

	2 nodes, factor 2	3 nodes, factor 3	3 nodes, parity
Object reads with a node down	✅ Continue	✅ Continue	✅ Continue
Object writes with a node down	✅ Continue	✅ Continue	✅ Continue, in two whole copies
Data redundancy	✅ A whole copy on each node	✅ A whole copy on each node	✅ A shard on each node, 1.5× the data
Configuration changes with a node down	✅ Continue on the survivor	✅ Continue	✅ Continue
Failed node returns	✅ Rejoins and syncs	✅ Rejoins and syncs	✅ Rejoins and syncs

The trade this makes, stated plainly. If both nodes are alive but the network between them fails, each sees only itself and each becomes the coordinator of what it can see, so both accept configuration changes. When the link is restored, the synchronisation reconciles them the way it reconciles everything else: newest write wins, entity by entity, by updated_at. Object data is never affected, since it does not go through the coordinator at all. The alternative — freezing administration until a strict majority is available — was rejected on purpose: it means a two-node cluster locks its operator out the moment either node fails, including out of the very operations needed to recover.

Architecture¶

Port Layout¶

Each MaxIOFS node exposes three independent ports:

Port	Purpose	Exposed to
8080	S3 API — object storage operations	S3 clients, load balancer
8081	Web Console — admin UI and console REST API	Operators, load balancer
8082	Cluster inter-node — coordination, sync, CSR signing	Other cluster nodes only (firewall off from public)

Every node of a cluster listens on these default ports. Add Node registers the new node's S3 API on port 8080 and its cluster port on the cluster port of the node that adds it: a node on other ports is registered at the wrong addresses, and its health checks, synchronization and the requests forwarded to it fail.

Cluster Components¶

┌──────────────────────────────────────────────────────┐
│         Load Balancer (HAProxy/Nginx)                │
│  :8080 → S3 API     :8081 → Web Console              │
└──────────────────────┬───────────────────────────────┘
                       │
          ┌────────────┴────────────┐
          │                         │
   ┌──────▼──────┐           ┌──────▼──────┐
   │   Node 1    │◄──:8082──►│   Node 2    │
   │  10.0.1.10  │  cluster   │  10.0.1.20  │
   │  S3  :8080  │  (HMAC+    │  S3  :8080  │
   │  UI  :8081  │   TLS)     │  UI  :8081  │
   │  CL  :8082  │            │  CL  :8082  │
   └─────────────┘            └─────────────┘

Note: Port 8082 should never be exposed to end users or the public internet. Restrict it at the firewall to cluster node IPs only.

Core Components¶

1. Cluster Manager - Manages cluster configuration and state - Handles node registration/removal - Tracks nodes in SQLite database (cluster_config, cluster_nodes)

2. Request handling - Every node holds every bucket and serves any request about it; an object whose data another node holds is read from that node - A request about a multipart upload goes to the node the upload was started on

3. Health Checker - Monitors all nodes every 30 seconds - Measures network latency - Updates status: healthy (<1s), degraded (1-5s), unavailable (>5s)

Quick Start¶

Prerequisites¶

2+ MaxIOFS instances on different servers
Network connectivity on all three ports (8080, 8081, 8082) between all nodes
Firewall: port 8082 open between node IPs only (not to the public internet)
Admin access to all nodes

Containers: a multi-node cluster inside a single Docker host provides no real HA — every "node" shares the same disk, power supply and kernel, so replication multiplies storage cost without adding durability. If you run nodes as containers (Docker/Swarm/Kubernetes), place one node per physical host, pin each container to its host with a local volume (a rescheduled container without its volume loses the node's identity and data), and make port 8082 reachable between hosts. Container multi-host orchestration is not part of the officially tested matrix.

Setup Steps¶

1. Start both nodes

# Node 1 (IP: 10.0.1.10)
./maxiofs --data-dir /data/node1

# Node 2 (IP: 10.0.1.20)
./maxiofs --data-dir /data/node2

Both nodes start on their default ports: S3 on :8080, Console on :8081, Cluster on :8082.

2. Initialize the cluster on Node 1

# Open http://10.0.1.10:8081 → Cluster → Initialize Cluster
# Fill in:
#   Node Name: node-1
#   Region: us-east-1 (optional)
# → Cluster token is displayed — COPY AND SAVE IT

3. Open a join code on Node 2

# Open http://10.0.1.20:8081 → Cluster → Join code
# → A join code is displayed, good once for 15 minutes

4. Add Node 2 from Node 1's console

# Cluster → Nodes → Add Node
# Fill in:
#   Node IP Address: 10.0.1.20
#   Join Code:       <the code Node 2 shows>

Node 1 sends the join package to Node 2's console (port 8081) sealed with the join code: AES-256-GCM, with a key derived from the code. Only Node 2 opens it, over plain HTTP too, and Node 2 seals its answer with the same key, so Node 1 registers Node 2 only once Node 2 read the package. A node joins only this way: from the console of a node of the cluster, with the join code its administrator opened.

5. Verify cluster

Check the Cluster page on either node: - Total Nodes: 2 - Healthy Nodes: 2 - Both nodes showing green status with latency < 10 ms

5. Configure replication (optional — required for HA)

Navigate to Cluster → Bucket Replication: - Select bucket - Choose destination node - Set sync interval: 60 s for real-time HA, 300 s for near-real-time, 3600 s for hourly - Enable "Replicate deletes" and "Replicate metadata"

Cluster Setup¶

Production Deployment¶

                ┌──────────────────┐
                │  Load Balancer   │
                │  192.168.1.100   │
                └────────┬─────────┘
                         │
            ┌────────────┴────────────┐
            │                         │
     ┌──────▼──────┐           ┌──────▼──────┐
     │   Node 1    │           │   Node 2    │
     │  10.0.1.10  │◄─────────►│  10.0.1.20  │
     └─────────────┘           └─────────────┘

HAProxy Configuration¶

Port 8082 is internal cluster coordination — do not route it through the load balancer. Each node must be able to reach the other nodes' port 8082 directly.

# /etc/haproxy/haproxy.cfg
global
    maxconn 4096
    daemon

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# S3 API (Port 8080) — client-facing
frontend s3_frontend
    bind *:8080
    default_backend s3_backend

backend s3_backend
    balance roundrobin
    option httpchk GET /health
    server node1 10.0.1.10:8080 check inter 10s fall 3 rise 2
    server node2 10.0.1.20:8080 check inter 10s fall 3 rise 2

# Web Console (Port 8081) — operator-facing
frontend console_frontend
    bind *:8081
    default_backend console_backend

backend console_backend
    balance roundrobin
    option httpchk GET /health
    server node1 10.0.1.10:8081 check inter 10s fall 3 rise 2
    server node2 10.0.1.20:8081 check inter 10s fall 3 rise 2

# Port 8082 — cluster inter-node communication
# DO NOT expose via HAProxy — nodes reach each other directly

Network Configuration¶

Firewall Rules:

# Allow S3 and Console API from anywhere (or your client CIDR)
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
iptables -A INPUT -p tcp --dport 8081 -j ACCEPT

# Allow cluster inter-node communication only between cluster node IPs
iptables -A INPUT -s 10.0.1.10 -p tcp --dport 8082 -j ACCEPT  # From Node 1
iptables -A INPUT -s 10.0.1.20 -p tcp --dport 8082 -j ACCEPT  # From Node 2
iptables -A INPUT -p tcp --dport 8082 -j DROP                   # Block all others

DNS Configuration:

# /etc/hosts
10.0.1.10  node1
10.0.1.20  node2
192.168.1.100  maxiofs-cluster

Configuration¶

Server Config Options¶

Config key	Default	Description
`cluster_listen`	`:8082`	Bind address for the cluster inter-node server. Keep port 8082 on a node of a cluster; the address can be restricted to one interface.

# config.yaml — cluster port (all other cluster config is managed via the web console)
cluster_listen: ":8082"

Environment variable equivalent: MAXIOFS_CLUSTER_LISTEN=:8082

⚠️ A node of a cluster listens on the default ports: S3 API 8080, console 8081, cluster 8082.

When you add a node, the primary registers the new node's S3 API on port 8080 and its cluster port on the primary's own cluster_listen port; it does not ask the new node for them. A node on other ports is registered at the wrong addresses: health checks fail, the node never votes, and requests forwarded to it fail.

Cluster Initialization Parameters¶

Parameter	Type	Required	Description
`node_name`	string	Yes	Human-readable name (e.g., "node-east-1")
`region`	string	No	Geographic region (e.g., "us-east-1")
`local_endpoint`	string	No	Override S3 API endpoint other nodes should use to reach this node. Leave empty to use `public_api_url`.

Add Node Parameters (from the primary node's console)¶

Field	Default port	Description
Node IP Address	8081	IP address of the remote node. The join package is sent to its console on port 8081; the cluster then reaches it on port 8082.
Join Code	—	The code opened on the remote node: Cluster → Join code. Good once, for 15 minutes; five wrong codes close it

Health Check Configuration¶

Interval: 30 seconds (hardcoded)
Timeout: 5 seconds
To modify: Edit internal/cluster/manager.go → healthCheckInterval

Cluster Replication¶

Overview¶

Cluster replication enables node-to-node replication for HA. This is separate from user replication (external S3 backup).

Key Differences:

Feature	Cluster Replication	User Replication
Purpose	HA between MaxIOFS nodes	Backup to external S3
Authentication	HMAC with node_token	S3 access key + secret
Credentials	None required	AWS credentials required
Tenant Sync	Automatic (30s)	N/A

How It Works¶

Object PUT on Node 1
Sign with HMAC-SHA256
Send the stored ciphertext as-is to Node 2 (plus the object's metadata and encryption sidecar — the primary's modification time and client checksums travel with it)
Node 2 verifies the HMAC signature
Node 2 stores a byte-identical copy and decrypts only on read

The object's retention and legal hold travel with it, in this transfer and in the decrypt/re-encrypt fallback below, so a replica is protected from its first write. The fallback also carries what its headers cannot: the ETag (a multipart ETag included), tags, ACL and restore state. The receiving node refuses bytes that do not hash to that ETag.

The same holds for every other copy between nodes: the initial sync of a new replica, and the anti-entropy push and pull.

The initial sync of a new replica copies every version of every key, oldest first and with its version ID — delete markers as delete markers, with their ID and time, including keys a delete marker hides. A copy that arrives after a newer version of the same key (an initial sync racing live writes) is stored as an older version and does not replace the latest; a version the replica already holds is replaced, not counted twice. In a bucket whose versioning was suspended, the versions kept from before are copied as versions and the current object without a version ID last; a copy is stored as the version it copies, whatever the receiving bucket's versioning status is. An object the sync fails to send is left to the catch-up: the job says how many, and the replica is recorded as having missed writes from the oldest of them. A node that joins a cluster starts its initial sync at once.

When fewer peers confirm than the replication factor needs (a factor of 3 with both peers failing), Node 1 answers 503 ServiceUnavailable (Retry-After: 30) and removes the version it just wrote, even under the retention or legal hold that write set: the client was told the write failed. An unversioned write is kept locally, since deleting it would lose the object it replaced. A peer that failed is recorded as having missed the write and caught up when it is back.

Encryption Keys: all cluster nodes share a cluster-wide KEK, distributed inside the join package and re-synced on every key rotation. Because every node can unwrap any object's DEK, replication never decrypts in transit. Objects wrapped with a node-local (pre-join) key transparently fall back to the legacy decrypt-on-source / re-encrypt-on-destination path until the background worker re-wraps them to the shared key.

Encryption secret: the secret identity provider secrets, replication destination keys and share link keys are encrypted with is the cluster's on every node. A node that joins takes it from the join package; it holds no credentials yet (see Membership). Every minute the coordinator compares each healthy node's fingerprint of the secret with its own and gives its secret to a node that differs; a node takes it only from the coordinator it knows. The Nodes page marks a node that holds another secret.

Objects¶

Every node holds the entry of every object — its versions, delete markers, tags, ACL, retention and legal hold — and the factor's nodes hold its data. With a factor of 1 an object has one copy.

A write's data is held by the node that takes it and the least full healthy nodes, up to the factor: what each holds of its room, held and free, as the last copy repair counted it, or of its disk before a count. Free space is what each node's last health check reported, and a node under storage pressure takes no new data. The entry names the nodes that hold the data and reaches every other healthy node before the request returns.
Listings, HEAD, versions, tags and ACLs are answered from the entry on any node. A GET on a node that does not hold the data reads it from a node that does, healthy nodes first, a range alone; a node that holds another write of the key is passed over. With no node holding the data answering, the read is answered 503 with Retry-After.
A delete reaches every healthy node; the nodes that held the data remove it. An overwrite without versioning is held where it is placed, and a node that held the earlier data and does not hold the new one removes it.
Of two writes of one key the later one wins on every node, whichever order the copies arrive in. A node that deleted a key does not store a copy of a write made before the delete.
A node under storage pressure that takes a write passes its copy to the least full node among those holding the entry only, before the request returns; it keeps the copy when no node takes it.
The anti-entropy, a catch-up and the first copy to a new node send each node the entries it lacks and the data of the writes it holds. A node that holds a write and lost its data is sent it again.
The copies follow the factor. When a node holding copies is removed or marked dead, when the factor changes, and after each full anti-entropy cycle, a write with fewer copies than the factor is copied to the least full healthy nodes; one with more copies keeps the copy of the node that repairs it and those of the least full nodes, and the others remove theirs. Each write is repaired by the first of its living holders by node ID.
The copies and shards of each class are spread over its nodes by the room of each: what a node holds plus its free space. When a node is more than 10 points fuller or emptier than its class, as after a join or a return, the repair moves copies and shards to the emptier nodes, each new one made before the old one goes. Nothing moves while a node of the class is out of service or drained, nor while a node reported its free space before the copies last moved: the next health check, within 30 seconds, counts them. Every node repairs its writes at the next pass of the dead-node reconciler after the nodes of a class change. ha.rebalance_enabled (true) turns it off.
Each change of where a write's copies are is numbered. Every node keeps the newest, whichever order the changes, entries and copies arrive in: an entry or a copy with older locations does not change them, and a copy is stored only on a node they name. The anti-entropy compares the numbers too and sends a node with older locations the newer entry.
Every node acts on a change of the factor, made on it or received from another node; the node where it is made tells the others at once. A higher factor adds the copies once the first syncs end; a lower one removes the copies it does not need.
After an update from a version that kept each bucket on one node, each node with a factor of 1 gives the objects it holds itself as their location, once, at start. It then sends the other nodes its buckets and their entries, without their data. Until a node has them all, it answers NoSuchBucket for a bucket it lacks and may not list every object of one it has: update every node, and send traffic once the HA page shows the sync jobs done.
An object written before the cluster kept where its data is names no node. The copy repair asks every other healthy node which of these it holds; the first of the holders by node ID gives each its holders, trimmed to the factor, and makes the copies it lacks. Without the answer of every healthy node it waits for the next repair.
The capacity Veeam reads for a bucket (SOSAPI capacity.xml) is its quota, else its tenant's quota less what the tenant uses on every node, else the room of STANDARD: what its nodes but the dead ones hold with each byte on factor different nodes, or with parity each shard of a byte on its own node, as their last health checks found their disks.

Objects with parity¶

With parity on (PUT /api/v1/cluster/ha with {"parity":true}, or the HA page) every node holds the entry of each write, and the nodes it names hold one shard each.

The layout follows the nodes that hold data, those neither dead nor drained: one parity shard up to four of them, two from five, and data shards for the others, eight at most. Each entry keeps the layout it was cut with.
The node that takes a write stores it whole, cuts it, sends each node its shard, then names them in its entry and sends the entry to every other node; the copy it stored goes. One shard more than the data shards is enough for the write; a node that missed its shard is given it when it is caught up. Short of that the write is kept in whole copies.
Each node encrypts its shard with a key of its own. Shards travel decrypted over the cluster's TLS connection. A shard is a file of its own beside the object's file, and carries the write it is of: a node never serves a shard of another write.
A write in shards is read from any node: from the data shards, or from a parity shard in place of one that does not answer, a range alone. With fewer shards answering than the data shards, the read is answered 503 with Retry-After; HEAD and listings still answer.
The copy repair follows the protection. Turning parity on cuts the writes kept in whole copies in shards; turning it off joins them in whole copies. The shard of a node removed, dead or drained is made again from the others on a node that holds none of the write, at the same index; a drained node's shard is copied from it while it answers. A write whose shards cannot stay on as many different nodes, with fewer parity shards than the cluster asks (a fifth node), or with fewer data shards than its nodes hold (a node back, a sixth node), is cut again; no node is given the index it already holds a shard of, so the shards in use stay until the new placement lands. When the layout changes, every node repairs its writes at the next pass of the dead-node reconciler (ha.redistribution_check_interval_minutes, 5) once every node that holds data takes it.
The anti-entropy sends the entries of writes in shards. A shard a node is to hold and lacks is made again by the first of the write's other holders in service.
The integrity scan of a node checks the shard it holds of each write in shards: it must decrypt and match its digest. KEK rotation re-wraps the key of each shard.
A node that leaves the cluster drops its writes in shards: its shard alone cannot serve them. maxiofs recover rebuilds no entry from a shard; the cluster sends the node the entries when it is back.
Every node runs the same version before parity is turned on.

Storage classes¶

A storage class with an S3 name (REDUCED_REDUNDANCY, STANDARD_IA, INTELLIGENT_TIERING, ONEZONE_IA, GLACIER_IR, GLACIER, DEEP_ARCHIVE) can be kept on nodes of its own, with a protection of its own: whole copies (a factor of 1 to 3) or parity. STANDARD is every node in no other class; its protection is the one set on the HA page. The classes and the class of each node are set on the Storage classes page (Cluster → Storage classes).

An object is of the class of its write: the x-amz-storage-class of its PUT, multipart upload or copy (a copy without one is STANDARD, as in S3), or the class a lifecycle transition gives it. An object of a class the cluster does not keep is kept in STANDARD and keeps its class: HEAD and listings give it.
The data of a write goes to the least full nodes of its class; a node that takes a write of another class passes its copy on, as a node under storage pressure does. A class with fewer nodes taking data than its protection spreads a write over keeps its writes on nodes of other classes as well, so that each write has its copies or shards; the copy repair moves them to the nodes of the class once it has them. The cluster is then degraded, with the class named.
Every node reads every object, whatever its class: there is no restore to wait for (RestoreObject answers at once). The AWS CLI's aws s3 cp downloads an object of GLACIER or DEEP_ARCHIVE only with --force-glacier-transfer, or once aws s3api restore-object has answered; aws s3api get-object and the SDKs read it as any other.
A node put in another class copies the writes of its old class to the other nodes of that class, then drops them, and takes the writes of its new one. A node does not leave a class whose other nodes taking data would be fewer than its protection needs (409); a class is emptied by deleting it, which puts its nodes back in STANDARD. Its objects keep their class and are kept in STANDARD.
Lifecycle Transition (days or a date) and NoncurrentVersionTransition rules move objects to a colder class: STANDARD_IA, INTELLIGENT_TIERING, ONEZONE_IA, GLACIER_IR, GLACIER, DEEP_ARCHIVE, from the warmest; never to a warmer one. Of the transitions due, the coldest wins; an object due to expire is not moved. Every node runs the rules and moves the writes it is the first living holder of: it gives each its class on every node, with a change of its placement, then copies it to the nodes of the class, or cuts it in shards there, before the other nodes drop theirs. A move that does not end is ended by the next repair.
When the protection of a class changes, or the nodes of a class, every node repairs its writes at the next pass of the dead-node reconciler; the node where the change is made at once.
Drains and dead nodes count every node that takes data against the most demanding class, as a class short of nodes keeps its writes on others.
The room Veeam reads (SOSAPI capacity.xml), and the room and protection on the HA page, are STANDARD's.
Outside a cluster a transition gives the object its class only.
maxiofs recover gives an object the class of its write; the next lifecycle pass moves it again.
Every node runs the same version before a class is kept.

Buckets¶

Every node holds every bucket, whatever the factor.

Creating a bucket, changing its configuration or ACL, and deleting it reach every other node before the request returns. Usage (object count, size) is each node's own and is not sent.
A node that is down or does not take the change is recorded as having missed a write. When it is caught up, it is sent every bucket and every deletion before the objects. A new replica, and every peer at the start of an anti-entropy cycle and when a node starts, is sent them too. A cluster whose buckets exist only on the node that created them converges this way after the upgrade.
Two versions of a bucket are ordered by the time of the change; a deletion by the time it was made, to the nanosecond, so a bucket created again right after its deletion exists again.
A node keeps its copy of a bucket deleted elsewhere when the copy changed after the deletion or holds an object under retention or a legal hold; such a copy is kept and logged. A copy holding objects written after the deletion is kept with those objects only, and sent to the other nodes as created again then. Otherwise the copy is removed with its objects.
A bucket whose name another tenant holds on a node is refused there and logged.
A deletion is kept at least 7 days, and until every member of the cluster has been sent it.
Lifecycle: the coordinator expires objects, and its deletes reach the other nodes. Every node aborts the incomplete multipart uploads started on it, and moves to a colder storage class the writes it is the first holder of (see Storage classes).
Multipart uploads: an upload is held by the node it was started on, and its ID names that node. Its parts, part listing, completion and abort are sent to that node from whichever node they reach, so a load balancer may spread them. A node that does not answer is answered 503 with Retry-After; a node no longer in the cluster took its uploads with it (NoSuchUpload). Listing the uploads of a bucket asks every healthy node. An upload started before the upgrade has an ID that names no node and is made on the node a request reaches.
Inventory: the coordinator writes the reports. The report file, its row and the configuration's last run reach the other nodes.
Scheduled replication rules run on the coordinator. Real-time rules queue each write on the node that took it.

Bucket rows¶

Every node also holds the rows a bucket keeps in the node's database: shares, replication rules, inventory configurations and inventory reports.

A change is sent to every other node before the request returns. A node that is down or does not take it is recorded as having missed a write; it is sent every row, after the buckets, when it is caught up, at the start of an anti-entropy cycle and when a node starts.
Each row carries the time of its last change, to the nanosecond. A later change wins; at the same time a deletion wins. Rows from before the upgrade carry no time and reach every node.
Two rows that may not coexist (two shares of one object, two inventory configurations of a tenant's bucket) end as the later one on every node.
A report whose configuration was deleted is not stored. A row a node cannot store (unknown column, missing tenant) is refused and logged by the sending node.
Expired shares are not sent: every node removes them.
A deletion is kept at least 7 days, and until every member of the cluster has been sent it.
Stored credentials in these rows are encrypted with the cluster's encryption secret, which every node holds.

HMAC Authentication¶

Message Format:

HMAC-SHA256(node_token, METHOD + PATH + TIMESTAMP + NONCE + BODY)

Request Headers:

X-MaxIOFS-Node-ID: sender-node-id
X-MaxIOFS-Timestamp: <unix-timestamp>
X-MaxIOFS-Nonce: <random-uuid>
X-MaxIOFS-Signature: <hex-encoded-hmac>

Validation: - Retrieves node_token from database - Computes expected signature - Compares with provided signature (constant-time) - Checks timestamp skew (max 5 minutes)

Automatic Entity Synchronization¶

What the cluster shares is automatically synchronized across all cluster nodes every 30 seconds, and at once after a change:

Entity Type	Endpoint	What Gets Synced
Users	`/api/internal/cluster/user-sync`	Credentials, roles, tenant, preferences
Tenants	`/api/internal/cluster/tenant-sync`	Quotas, settings, status
Access Keys	`/api/internal/cluster/access-key-sync`	Key ID, secret, user association
Bucket Permissions	`/api/internal/cluster/bucket-permission-sync`	ACLs, policies
IDP Providers	`/api/internal/cluster/idp-provider-sync`	LDAP/OAuth config (encrypted secrets)
Group Mappings	`/api/internal/cluster/group-mapping-sync`	IDP group → MaxIOFS role mappings
Groups	`/api/internal/cluster/group-sync`	Groups and their members
IAM	`/api/internal/cluster/iam-sync`	Managed policies with their versions, roles, inline policies and attachments
Console Sessions	`/api/internal/cluster/console-session-sync`	Console sessions: any node renews or ends one
STS Sessions	`/api/internal/cluster/sts-session-sync`	Temporary credentials
Settings and Logging Targets	`/api/internal/cluster/settings-sync`	Every setting and logging target; the newer change wins
Cluster Configuration	`/api/internal/cluster/global-config-sync`, `node-list-sync`, `kek-sync`	The cluster's configuration, its list of nodes and the cluster-shared encryption keys, every 60 seconds

How it works: - Users, tenants, access keys, bucket permissions, identity providers and group mappings are sent only when their SHA256 checksum changes - HMAC-authenticated node-to-node communication - 30-second sync interval per entity type

Result: - Admin password is identical across all nodes - Users created on one node are immediately available on all nodes - User sessions work correctly after node failover - IDP/SSO configurations available on all nodes

Membership¶

Every node holds the list of nodes. A change to a node (name, region, priority, drained) is stamped with its time and the newest copy wins on every node. The list is sent every 60 seconds and right after a change made in the console.

Remove (console, any other node): every node removes it, and no node that still lists it adds it back: a node joins under a new ID every time, so a removal is final. The removed node is answered 410 Gone the next time it contacts the cluster, and leaves. Its data stays on its disk.
Leave (console, the node itself): the node tells every node it reaches to remove it, then leaves. The others pass the removal on. A node that leaves, or is removed, drops the entries of the objects the other nodes hold and keeps those it holds; an entry naming no node is held where its file is.
Drain (console of any node, the drained one included): planned decommissioning, with any factor.
The node takes no new copy of data. A write it takes is passed, before the request returns, to the least full node; multipart uploads started on it finish there and pass the object on.
It serves reads and is sent every entry, delete and change while its objects are moved: each copy it holds is made on the least full nodes that take data, then dropped from it. The copy repair does this when the drain starts and at every pass of the dead-node reconciler (ha.redistribution_check_interval_minutes, 5).
The HA page shows the move: the objects and bytes the node held when it started, what it moved, and the expected end at the pace so far. The drained node counts them, and the other nodes take them from its health check.
Once a pass finds it holding no data, it answers every S3 request with 503, /health included, so load balancers stop sending it requests; the console still answers. Every node then marks it dead (drained). It can be removed from the cluster. Multipart uploads still open on it are lost when it is removed.
A drained node that does not answer is marked dead at once when its copies are held elsewhere (factor 2 or 3, or parity); with a factor of 1 it is waited for, and its objects are moved when it is back.
Refused with 409 when the other nodes that take data would be fewer than the replication factor, or than three with parity. A drained node stays out of service even when it answers.
Dead: a node unreachable for longer than ha.dead_node_threshold_hours (24 hours) is marked dead by each node from its own health checks. When it answers again it is back in service and is compared on every write: the copies it held were made again elsewhere meanwhile, and their new locations reach it.
Slow: a node whose health check takes over a second is shown as degraded. It stays in service and takes data like a healthy one.
Storage pressure: a node whose disk reaches ha.storage_pressure_threshold_percent (90 %) is under storage pressure until it drops below ha.storage_pressure_release_percent (85 %), and the console is alerted both ways. It stays in service: its buckets are found and listed from every node, it is sent every entry, delete and change, and it counts as alive. It takes no new copy of data: writes, repairs and moves place copies on the other nodes.
Join: only a node without data joins a cluster. A node that holds a bucket, a tenant, a user besides its first administrator, an access key, a group, an identity provider, an IAM policy or role created on it, a share or a replication rule is refused with 409, which names what it holds, and stays as it was. Its first administrator, settings and keys are not data.
A node that joins a cluster takes the cluster's list of nodes. What it held of a cluster it was in before is dropped.

Deletions¶

A deletion is recorded in cluster_deletion_log so a node that still holds the entity does not bring it back. It covers users, tenants, access keys, bucket permissions, identity providers, group mappings, groups, STS sessions, IAM policies, roles, attachments and inline policies, objects, and nodes removed from the cluster.

A deletion keeps the time it was made on the node that made it, dated after the last change that node knew of what it deleted. Two records of the same deletion keep the later time.
An entity and a deletion of it are ordered by time on every node: a copy changed after the deletion is taken, one changed before it is refused, and a deletion removes a copy only if the copy did not change after it. At the same second the entity is kept. Access keys, bucket permissions and STS sessions keep no change time: any deletion of them wins.
An object deleted once and written again is kept and sent to the nodes that lack it; a copy written in the second of the deletion counts as written after it.
Deleting a user, a tenant or a group records the deletion of the IAM policies they held.
Each node sends every other node the deletions it has not taken yet, in the order they were recorded, every 30 seconds (POST /api/internal/cluster/deletion-log-sync). A new node is sent the last 7 days.
A deletion received by any channel removes the node's copy: the deletion log, the IAM and STS payloads, and each entity's own synchronization.
A deletion is kept at least 7 days, and until every member of the cluster has been sent it and no catch-up that replays it is pending. A member away, offline or cut off and serving clients, for however long, is sent them when it is back. A node removed from the cluster no longer holds them back.
Deletions are numbered in a sequence that never goes back.
Object times are whole seconds. Each copy of an object also carries the time it was written to the nanosecond, which orders two writes of one second, and a write and a deletion of one second. Objects written before the upgrade carry none: within the second, a deletion keeps them and two of them are ordered by ETag. S3 LastModified is unchanged.

A bucket deleted by a client takes with it its shares, replication rules, inventory configurations, permissions and the policies naming it; a bucket created by a client starts without what a former bucket of the name left. Every node that removes a bucket, for any reason, drops its notification configuration and integrity scans.

Dashboard UI¶

Accessing Cluster Dashboard¶

Login to web console (http://localhost:8081)
Click "Cluster" icon in sidebar (requires global admin)

Cluster Overview¶

Status Cards: - Total/Healthy/Degraded/Unavailable Nodes, Total Buckets

Nodes Table Columns: - Name, Endpoint, Health Status (🟢/🟡/🔴/⚪) - Latency (ms), Capacity (used/total) - Priority, Last Seen, Actions (Edit/Remove)

Dialogs¶

Initialize Cluster: - Node Name, Region (optional), Local S3 API Endpoint (optional — leave empty to use public_api_url) - Shows the cluster token: the nodes authenticate to each other with it. Other nodes are added with Add Node

Join code (on a node in no cluster): - Opens a join code, good once for 15 minutes; opening another closes it. Five packages it does not open close it too

Add Node (initiated from the primary node): - Node IP Address — IP of the remote node; its console is reached on port 8081 - Join Code — the code opened on the remote node - The primary node handles the full join handshake automatically

Edit Node: - Editable: Name, Region, Priority, Metadata - Read-only: Endpoint, Node ID (cannot change after join; remove and re-add to change)

API Reference¶

Base URL: http://localhost:8081/api/v1 Authentication: JWT token required in Authorization: Bearer <token> header

This document explains cluster behavior and operations. The canonical endpoint list lives in API.md; keep endpoint additions there to avoid drift.

Primary endpoint groups:

Cluster management: initialize, join, leave, status, nodes, health, drain.
Replication: factor, sync jobs, anti-entropy runs, degraded state, a bucket's replicas.
Storage classes: the classes kept, the protection of each, the class of each node.

Security¶

HMAC-SHA256 Authentication¶

Purpose: Secure node-to-node communication without S3 credentials

Algorithm: HMAC-SHA256 Secret Key: node_token (generated during cluster initialization)

Signing Process: 1. Compute message: METHOD + PATH + TIMESTAMP + NONCE + BODY 2. Compute HMAC: HMAC-SHA256(node_token, message) 3. Hex-encode signature 4. Add headers: X-MaxIOFS-Node-ID, X-MaxIOFS-Timestamp, X-MaxIOFS-Nonce, X-MaxIOFS-Signature

Validation: 1. Extract headers from request 2. Retrieve node_token from database 3. Compute expected signature 4. Compare with provided signature (constant-time) 5. Verify timestamp within ±5 minutes 6. Reject if validation fails (HTTP 401)

Node Token Security Best Practices¶

Generate Strong Tokens: 256 bits of entropy minimum (openssl rand -hex 32)
Rotate Regularly: Every 90 days recommended
Store Securely: Encrypted in SQLite, never log in plaintext
Network Security: Use TLS/HTTPS, restrict ports to cluster network only

Firewall Configuration¶

Nodes reach each other on port 8082 for everything between nodes, health checks included; on port 8080 to send a multipart upload request to the node the upload was started on; and on port 8081 when Add Node sends the join package to the new node's console. Clients need ports 8080 and 8081 only.

# S3 and console: clients, the load balancer and the other nodes
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
iptables -A INPUT -p tcp --dport 8081 -j ACCEPT

# Cluster port: the cluster's nodes only
iptables -A INPUT -s 10.0.1.10 -p tcp --dport 8082 -j ACCEPT  # From Node 1
iptables -A INPUT -s 10.0.1.20 -p tcp --dport 8082 -j ACCEPT  # From Node 2
iptables -A INPUT -p tcp --dport 8082 -j DROP                   # Block all others

Monitoring & Health¶

Health Check System¶

Automatic Checks: - Interval: 30 seconds - Measures network latency - Updates status: healthy (<1s), degraded (1-5s), unavailable (>5s), unknown (not checked)

Health Endpoint:

curl http://localhost:8080/health
# Returns: status, timestamp, version, uptime, cluster_enabled, node_id, node_name

Troubleshooting¶

0. "The cluster is choosing a coordinator" on every configuration change¶

Symptoms: The console answers 503 with "the cluster is choosing a coordinator for configuration changes", or "this node is no longer the coordinator", on configuration changes. S3 traffic is unaffected — objects upload and download normally, which is the clue that this is the control plane and not the cluster as a whole. A bucket's settings and objects in the console are unaffected too.

Diagnosis:

# Who holds the lease, and in which term?
sqlite3 /data/node1/db/maxiofs.db "SELECT leader_id, term, expires_at FROM cluster_leader;"

# The election says why it failed, on every node
grep '"component":"leader"' /var/log/maxiofs/maxiofs.log | tail -20

The log names the peer, its answer and its term:

reason: unreachable — the peer's cluster port is not answering. Check the port is open between nodes and that every node listens on the default ports (see Configuration).
reason: voted no: lease still held — normal for a few seconds after a leader changes. Persisting means clocks are far apart, or a node was restored from a backup of another node's database.
got 1 of 2 votes needed among 2 responding nodes — both nodes are alive and disagree, which is normal for a few seconds during an election. Persisting means the two are campaigning against each other; check the clocks and that neither database was restored from a copy of the other's.

Note: in builds before 1.6.0 a two-node cluster could never elect anybody at all — the term climbed without bound and configuration changes were refused for ever, whether or not both nodes were up. Upgrade rather than chasing it.

1. Node Shows as "Unavailable"¶

Symptoms: Node appears red in dashboard

Diagnosis:

# Test connectivity: health checks go to the cluster port, over TLS with the cluster's own CA
ping -c 4 10.0.1.20
curl -k https://10.0.1.20:8082/health

# Check if MaxIOFS is running
ssh node2 "systemctl status maxiofs"

# Verify endpoint URL in database
sqlite3 /data/node1/db/maxiofs.db "SELECT id, name, endpoint FROM cluster_nodes;"

Fixes: - Check firewall rules (port 8082 open between the nodes) - Start MaxIOFS service if down - Update endpoint URL if incorrect

2. HMAC Authentication Errors¶

Symptoms: "Invalid HMAC signature" errors, 401 Unauthorized

Diagnosis:

# Verify cluster tokens match
sqlite3 /data/node1/db/maxiofs.db "SELECT cluster_token FROM cluster_config;"
sqlite3 /data/node2/db/maxiofs.db "SELECT cluster_token FROM cluster_config;"

# Check timestamp skew (clocks must be synchronized)
ssh node1 "date -u"
ssh node2 "date -u"

Fixes: - Ensure both nodes have same cluster token - Use NTP to synchronize clocks (max 5 minutes skew allowed)

3. Dashboard Not Loading¶

Symptoms: Loading spinner forever, console errors

Diagnosis: - Check browser console (F12 → Console tab) - Verify API endpoint responds:

curl -X GET "http://localhost:8081/api/v1/cluster/health" \
  -H "Authorization: Bearer $TOKEN"

Fixes: - Clear browser cache - Check network tab for failed API calls - Verify JWT token is valid

Debug Mode¶

# Enable debug logging
./maxiofs --data-dir /data/node1 --log-level debug

# Cluster-specific debug output:
# [DEBUG] Cluster Manager: Initialized with node_id=...
# [DEBUG] Health Checker: Node uuid-5678 is healthy (latency=15ms)
# [DEBUG] HMAC Auth: Signature valid for node uuid-5678

Log Locations¶

# systemd
journalctl -u maxiofs -f

# Docker
docker logs -f maxiofs-node1

# Standalone
./maxiofs --data-dir /data 2>&1 | tee maxiofs.log

For cluster test coverage and commands, see TESTING.md. The live SQLite schema is maintained by internal/db/migrations; avoid duplicating full schema definitions in this guide.

Version: 1.10.0 Last Updated: November 11, 2026 Documentation Status: Complete

For questions or issues, see README.md.