Changelog¶

All notable changes to MaxIOFS will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]¶

Added¶

Documentation site at https://docs.maxiofs.com/: installation, deployment, every console page with screenshots, S3 tools, the maxiofs CLI, integrations and operations.

Changed¶

Disabling your own 2FA and regenerating your backup codes ask for your password. A global administrator still disables another user's 2FA without it.
GET /config of the console gives the data directory, storage root and backend, ports, log level and metrics settings only to a global administrator.
Changing a password ends the console sessions of its account, the one that changed it included: the user signs in again with the new password. Access keys are not affected.
The parts of multipart uploads in progress count against the bucket and tenant quotas: a part is refused with QuotaExceeded once what is stored and what is in progress reach the quota. Before, only the completion was checked.
The permission picker offers Tenant administrator only for a user of a tenant, and Super administrator only for a user of no tenant.

Fixed¶

A user of no tenant with the tenant administrator role or permission was taken for an administrator, and could set a global administrator's password. It administers nothing.
A presigned GET whose query the router and the handler read differently was allowed by its signature alone, without its signer's permissions: any access key could read any object.
A presigned URL dated ahead of the clock outlasted the 7 days X-Amz-Expires allows: one dated more than 15 minutes ahead is refused.
A POST form upload was not held to all of its policy: eq conditions and conditions on fields other than the bucket, key and content type were ignored, and fields the policy does not name were accepted. They are enforced as in S3, and a field outside the policy is refused.
An identity provider set to Inactive still signed users in: by password for LDAP, by its sign-in URL for OAuth, and for STS federation. It signs nobody in.
STS federation (AssumeRoleWithWebIdentity, AssumeRoleWithLDAPIdentity and their JSON endpoints) did not apply the provider's groups to a user that follows them: one in no mapped group still got temporary credentials, with its old role. It is held to them as at a console sign-in.
AssumeRoleWithWebIdentity took a token to the provider named in ProviderId whatever its status: a provider that is not Active is refused, as on the JSON endpoint.
The last global administrator that can sign in could be deleted or deactivated when an inactive one existed, and deleted through the IAM API in any case, leaving the deployment with nobody to administer it. It is kept.
A cluster request with a wrong signature wrote the signature expected to the log.
Webhooks, the HTTP log target and replication reached addresses in the shared address space (100.64.0.0/10), where some clouds serve their metadata: they are refused as the private ranges are.
A public WRITE grant let an anonymous request remove a version of an object: it deletes the object only, adding a delete marker in a versioned bucket.
Enabling 2FA on an account that already has it replaced its secret and backup codes: it answers 409, and 2FA is disabled first.
A mistyped code when enabling 2FA answered 500: it answers 400.
The console dashboard named buckets the user cannot list.
Any console user could end the share of an object of a bucket they cannot read.
A stop waited for the integrity scan and the lifecycle pass in progress to end: both stop with the server.

[1.10.0] - 2026-11-11¶

Added¶

Auto Sync reads Google Workspace (with a service account key) and Microsoft tenants (named by the Issuer URL) every hour, as LDAP directories and Keycloak realms.

Changed¶

Integrity scans cover every version of a bucket and objects written in parts, whose composite ETag is recomputed.
ListObjectVersions markers name the last entry returned, and key-marker alone starts after that key, as in S3.
Writes stop at 99.5% of the disk of the objects or of the metadata, and a full disk answers 507 InsufficientStorage, not 500: Pebble and SQLite keep room, deletes still work, and SOSAPI reports only what can be written.
maxiofs recover says every 30 seconds how far it is: objects done of the total, percentage, time left and when it should end. It waits for the disk once per thousand files, not on each object.
A bucket's marker keeps its settings as JSON (owner, versioning, Object Lock and its default retention, policy, lifecycle, CORS, encryption, tags, quota…) and maxiofs recover gives them back. A bucket of an earlier release gets them at the first statistics pass.
An upload is encrypted as it is received: its temporary file never holds it in clear, and a PUT of a large object is about 25% faster.
/metrics answers only with the token of Settings → Metrics (metrics.token) as a bearer token, and 401 while none is set: it named every bucket of every tenant, with its size, to anyone. Prometheus sends it with authorization in its scrape config.
trust_private_networks is false by default. Behind a proxy or WAF, list in trusted_proxies the address it reaches MaxIOFS from, or logs, rate limits and aws:SourceIp see the proxy.

Fixed¶

Google and Microsoft sign-ins had no groups, so a group mapping refused them; with the directory set they read them from it.
ListObjectVersions read every version of the bucket for each page: a page now costs what it returns.
Under load nearly every request opened a new database connection, which parses the whole schema: S3 writes are about 5 times faster.
A byte changed on disk, or a lost sidecar or key, was reported as an error nobody was told of: integrity scans now report it as corruption, with their audit event and alert, and the alert says why.
DeleteObjects answered InternalError for what Object Lock protects: it answers AccessDenied, and takes x-amz-bypass-governance-retention.
A delete marker could be given a retention, which hid its object until the date: it answers 405, and one locked before is removed.
PutObjectRetention and PutObjectLegalHold were taken on a bucket without Object Lock.
Deletes of versions no longer hold the bucket through their fsync.
SlowDown counted as a server error: it is the server asking to send less, and is counted with the refusals.
A start removes the temporary files of uploads a crash cut short.
A stop during the reconciliation after a crash marked the shutdown clean, and it was never finished: the next start runs it again.
A start decoded every object of every bucket before opening its ports: with millions of objects it took a minute, and now seconds.
A stop waited for the bucket statistics pass to finish, minutes with millions of objects.
Behind a trusted proxy, the S3 request log, bucket access logs and the audit of setting and bucket changes named the proxy, not the client; anonymous S3 requests were limited per connection, not per client.
The request and latency charts of the Performance tab were always at zero, and those of the API tab showed the average since the start: both show what happens at each moment.
A refused S3 signature was logged whole; the log keeps its access key only.
Setting the SMTP password logged it.
With storage.root on another disk than data_dir, the console, the cluster, disk alerts and SOSAPI showed the space of the data_dir disk, not that of the objects.
maxiofs recover read the objects only from <data-dir>/objects: it takes --storage-root, as maxiofs reconcile does.
A node joining a cluster empties everything the cluster replicates and takes it from the cluster: its own policies, administrator and settings, newer than the cluster's, replaced them on every node.
Settings and logging targets are the same on every node of a cluster: each node kept its own.
An IAM policy received from another node is stored with its versions or not at all, and one without its default version is not taken.
A node that stops while sending its changes no longer makes the other nodes drop them and log context canceled.
In Docker, stopping the container killed MaxIOFS 2 seconds after the stop signal, before it finished shutting down, and the next start reported an unclean shutdown.
A part whose body was cut short was stored with what had arrived, without an error.
maxiofs recover left every bucket without its Object Lock setting, so no retention could be set or extended on it: a bucket with a locked version gets it back.
A forced bucket delete checked COMPLIANCE retention and legal holds only on a bucket with the Object Lock setting: it checks the objects always.
The background encryption pass took the bytes of an object whose sidecar was lost for plaintext, encrypted them again and made up a sidecar for them: bytes it cannot identify are left to the integrity scan.
A plaintext object without its sidecar that the encryption pass encrypted could not be read after.
A bucket a global administrator created naming a user as its owner gave that user no access to it.

Security¶

In the console, a bucket owner had more permissions than its own bucket. Its policy grants its own bucket only; existing owner policies are rewritten at start.
In the console, reading a bucket's settings needs its permission on the bucket, as changing them does.

[1.9.0] - 2026-11-05¶

Added¶

Parity: Reed-Solomon erasure coding as an alternative to the replication factor, from 3 nodes (HA page, PUT /api/v1/cluster/ha).
Storage classes: groups of nodes with their own factor or parity (Cluster → Storage classes); lifecycle transitions move objects to them.
Lifecycle filters by tags and size, and NewerNoncurrentVersions.
The copy repair spreads copies and shards by the free room of each node (ha.rebalance_enabled).
The Nodes page shows the coordinator.
Hadoop and Spark through S3A, verified.
maxiofs recover resumes an interrupted run.
Sign-out ends the console session on every node.
SSO: a session follows its provider; a user deleted or disabled there is signed out. OpenID Connect back-channel logout.
SSO: users made by a group mapping follow the provider's groups.
Auto Sync reads LDAP directories and Keycloak realms every hour: it creates, deactivates and reactivates users.
A node joins a cluster with a one-time join code opened on it (Cluster → Join code).
SSE-C: keys sent by the client, over HTTPS; blocked per bucket by default, as in AWS (PutBucketEncryption BlockedEncryptionTypes, Bucket → Security).
trust_private_networks (default true): false takes X-Forwarded-For and X-Forwarded-Proto from the proxies of trusted_proxies only.

Changed¶

Test Connection checks each step against the provider.
ETags are sent between double quotes.
4xx answers are counted apart from errors (maxiofs_s3_refused_total).
Keys and bucket names follow S3's rules.
The console's object search is GET /api/v1/buckets/{bucket}/search.
New writes and repair copies go to the least full nodes.
The S3 request log line gives received bytes, version, ETag and the reason of a 403.
POST /api/v1/cluster/nodes takes join_code instead of username and password.

Security¶

Add Node sent the cluster's keys (CA key, KEK, JWT secret) over unverified TLS; they are now sealed with the join code.
OAuth signed in users whose email the provider had not verified.
An OAuth provider could sign in users of another provider or tenant; an email now belongs to one user.
A tenant administrator could use another tenant's identity providers.
Making or changing an identity provider answered its secrets in clear.
Signed chunks and the digests of a write (Content-MD5, checksums, x-amz-content-sha256) were not checked.
A COMPLIANCE retention could be turned into GOVERNANCE and bypassed.
A folder downloaded as a ZIP could hold entries outside the folder.
A write with SSE-C headers was stored with the server's key and read without the client's.
The parts of a multipart upload in progress were stored unencrypted.
The debug log of request headers and the signature log gave the STS session token.
Behind a trusted proxy a client could choose its address: the leftmost X-Forwarded-For entry was taken (limits per address, aws:SourceIp, audit).
Built with Go 1.27.3: eleven standard-library advisories of Go 1.26.6 closed (HTTP/2, Range headers, MIME headers, TLS, x509).
The console's Content-Security-Policy runs no inline script but its own.
New data, metadata and metrics directories are created 0750, not 0755.
The 404 page of a bucket website ran script put in the link's path.

Fixed¶

Multipart: x-amz-version-id and errors of CompleteMultipartUpload, retried completions and aborts, If-Match/If-None-Match, headers and tags kept, EntityTooSmall.
Reads: partNumber, response-* overrides, ranges no longer sent the object's checksum, x-amz-tagging-count, Expires.
S3 error codes as AWS answers them (NoSuchBucket, InvalidBucketName, InvalidTag, KeyTooLongError, MetadataTooLarge, InvalidStorageClass, SignatureDoesNotMatch).
SigV4 with a header sent twice or runs of spaces; aws-chunked with another Content-Encoding; ACL xsi:type; PutObjectLockConfiguration without xmlns.
Versioning: objects written before it was enabled, or while suspended, are kept as version null.
Lifecycle rules that deleted or expired the wrong objects; the console no longer replaces rules set through S3.
Inventory reports gave NONE as the encryption status of every object.
Cluster: a request forwarded to another node lost its client's address and aws:SecureTransport.
PutObject with If-Match or If-None-Match under concurrent writes.
A PUT no longer holds its key's lock while receiving data; signed requests are about 10 times faster.
Websites: the root index, and an S3 host under the website's host.
Console: keys ending like a route (x/tags), presigned URLs of keys with special characters, Sync All results.
A delete of x no longer removes the folder marker x/.
LDAP: imported or synced users could not sign in; groups named by member or uniqueMember; Test Connection with many users.
LDAP and OAuth: a provider that does not answer no longer hangs a sign-in or locks accounts (503 after 10–15 s).
The login limit per IP counts failed sign-ins only, and the security settings apply to sign-in.
Settings were not read from their MAXIOFS_ environment variables (cluster_listen, auth.jwt_secret, …).
Cluster: user changes, two-factor and the recovery bundle state reach every node at once.
Cluster: reads with nodes down, joins, changes within one second, KEK rotation and the degraded banner.
Cluster: the copy repair could move too many copies to a node right after another node's moves.
A stop with the console open no longer hangs.
maxiofs recover object counts in versioned buckets.

[1.8.0] - 2026-10-13¶

Changed¶

Draining a node moves its objects to the other nodes before it leaves service, with any replication factor. It takes no new data, a write it takes is passed to the node with the most free space, and it serves reads while its objects move. Once it holds none it answers no S3 request (503, /health included) and every node marks it dead; it can then be removed. A drain marked the node dead at once: with a factor of 1 its objects were unreadable. The drained node itself can now be drained from its own console.
A replication factor of 1 keeps one copy of each object: on the node that takes the write, or on the node with the most free space when that node is under storage pressure. Every node holds every bucket and every object's entry, and lists and reads every object. Each bucket was on one node, which took all its writes: a bucket could not hold more than that node's disk. As with RAID 0, the objects of a node that is down are unreadable until it is back; a factor of 2 or 3 keeps copies on other nodes.
After the update, each node of a cluster with a factor of 1 gives the objects it holds itself as their location, once, at start, then sends the other nodes its buckets and their entries, without their data.
The replication factor goes back to 1 with several nodes: the copies it does not need are removed.
Console: the replication factor descriptions say how many copies of each object are kept and where.
Multipart completion streams parts directly into one encrypted storage write.
Removed two full-object reads and two full-object writes from multipart completion.
Multipart completion keeps only one part reader open at a time.
Bucket and tenant quotas reserve room for each write in flight: concurrent PUTs and multipart completions on a node cannot pass against the same free space.
Invalid Object Lock headers on PUT are rejected with InvalidRequest; the object is not stored.
DELETE and batch delete on a missing bucket return 404 (NoSuchBucket in the S3 API).
A node that comes back after missing writes is caught up at once: the node that accepted them compares the objects modified since the first missed write and sends the deletes the peer missed.
In a cluster, the coordinator expires objects by lifecycle rules and its deletes reach the other nodes; every node aborts the incomplete multipart uploads started on it. Expiration ran on the node the rule was set on, and its deletes did not reach the others.
A bucket's UpdatedAt dates its configuration: object writes and statistics recounts no longer change it.
The secret stored credentials are encrypted with (identity provider secrets, replication destination keys, share link keys) lives in the database, like the KEK. The first start stores auth.encryption_secret, or else an explicit auth.jwt_secret, or else a generated one; the configuration is not read again. The JWT secret no longer encrypts anything. Migration 21.
In a cluster every node holds the cluster's encryption secret: a node that joins takes it; the coordinator gives it to a node that holds another, which re-encrypts what it stores. The Nodes page marks a node that holds another secret.
At startup the log names the stored credentials the encryption secret does not decrypt.
In a cluster, only the coordinator writes inventory reports and runs scheduled replication rules.
Only a node without data joins a cluster. A node holding a bucket, a tenant, a user besides its first administrator, an access key, a group, an identity provider, an IAM policy or role created on it, a share or a replication rule is refused (409, naming what it holds) and stays as it was. Its data was mixed into the cluster's, and its global users were deleted.
A copy of a write without a version made before the current one does not replace it on the node that receives it, whichever order the copies arrive in.
A node that deleted a key does not store a copy of a write made before the delete. After a partition, the catch-up of the node that missed the delete could send the object back, and it stayed on both nodes.
Add Node no longer fails with Failed to determine local cluster address on a node with no route to 8.8.8.8 and no cluster_advertise_address. The address it worked out for itself was sent to the new node and never read.
A node of a cluster listens on the default ports: S3 API 8080, console 8081, cluster 8082. Add Node registers the new node at those; the documentation allowed another cluster port on every node.
Debian and RPM upgrades no longer change the owner of every file under /var/lib/maxiofs and /var/log/maxiofs, which took long with many objects. A new installation gives the two directories to the maxiofs user; the service writes as that user.

Fixed¶

Prometheus gets every metric MaxIOFS declares. S3 operations and errors by S3 error code, storage backend operations, object operations and sizes, sign-ins (S3 signatures, presigned URLs, form uploads, console password, 2FA and OAuth), bucket and Object Lock changes, objects under retention, each bucket's usage, events sent to the console, background task runs and the metadata cache were declared and never fed.
Prometheus: S3 request metrics are labelled by S3 operation instead of path. Each object key was a new series, and Prometheus memory grew with the number of objects. A refused request does not name its bucket.
Prometheus: refused S3 requests are counted. The metrics middleware ran after authentication.
Grafana: the dashboard shows every exported metric, by node and bucket. Its Buckets, Objects and Storage panels read metrics that did not exist.
Prometheus alert SuspiciousHighOperationRate applied rate() to a gauge; it reads the request counter.
In a cluster, the code of an OAuth sign-in is exchanged on any node: it names the node the provider's redirect reached, which the other nodes ask for it. Behind a load balancer the exchange could reach another node and fail with "invalid or expired code".
In a cluster, a change to an identity provider takes effect on every node at the next sign-in: each node rebuilds the provider when its stored row changed. A node kept the provider it first loaded until restart, so a new client secret or LDAP server, or a disabled or deleted provider, took effect only on the node where the change was made.
OAuth sign-in of a user with two-factor authentication asks for the code: the console exchanges the single-use code for the second step's token. The browser was sent to the login page with the user ID, where nothing asked for the code, and the user could not sign in.
Docs: docker/README.md listed dashboards and alert rules that do not exist.
Docs: CLUSTER.md's firewall example opened ports 8080 and 8081 only to the node subnet and left out 8082, which every node needs from the others; its unavailable-node checks tested port 8080 where health checks use 8082. Removed the cluster replication API, rules, queue and Prometheus metrics it described, which no longer exist. Prometheus metrics are on the S3 port only, not on both. The console health check is /api/v1/health; /health on the console port answers the console page.
A tenant's bandwidth cap paces browser form uploads (S3 POST) as it paces a PUT. They were read at full speed.
Console: the Security page says what is in place. It said the nodes use mutual TLS (each request is signed with HMAC-SHA256 over TLS), that cluster copies are encrypted again per destination (an object encrypted with the cluster key is copied as stored), that idle sessions end after the session timeout (the console signs out after 15 minutes idle; the timeout is the refresh-token lifetime), and listed a self-replication check that no longer exists. Durations show in the console's language.
S3: PutBucketReplication answers that replication rules are set in the console or with /api/v1/buckets/{bucket}/replication/rules. It named a cluster replication API that no longer exists.
Console and packages: the Security page, the About page and the Debian package description no longer claim GDPR, SOC 2, HIPAA, ISO 27001 or PCI DSS support. Nothing in MaxIOFS backs those claims.
Console: the tenant bandwidth hint says what the cap covers: S3 uploads and downloads to the tenant's buckets, on each node; console transfers are not capped. It said every upload and download of the tenant.
A tenant's storage goes down when an object is overwritten by a smaller one, on the node that writes it and on the nodes that receive the copy. It went down only at the next statistics pass, up to 15 minutes later; until then the tenant was charged the old size and writes that fit were refused.
With versioning suspended, a write without a version ID over a version kept from before is charged its whole size against the bucket and tenant quotas, as usage counts it. It was charged the difference and admitted past the limit.
In a cluster any node serves a sign-in: login, refresh, logout, two-factor verification and the start of an OAuth sign-in. They went to the coordinator: no one signed in during an election or on a node cut off from it. Each node counts the failed sign-ins it serves and locks the account on that node; an administrator's unlock reaches every node.
A change forwarded to the coordinator keeps the user's address and whether they connected over TLS. The coordinator saw every forwarded change come from 192.0.2.1: the audit log recorded that address, and the sign-in limit per address applied to all STS federation sign-ins through the console together.
A sign-in with two-factor authentication is recorded in the audit log.
The initial sync to a node asks which writes it holds and sends it only the entry of those it has. A sync running while the copies of a raised factor were made sent each copy back to the node it came from.
A node is marked dead while another component writes to the database: the change waits for it. It failed with SQLITE_BUSY and the node stayed unavailable until the next check.
A repair of the copies asked for while one runs runs once that one ends. A factor changed during a repair waited for the next anti-entropy cycle, hours later.
An anonymous GET of an object in a bucket whose ACL grants public read returns the object without a share, unless Block Public Access ignores public ACLs or restricts public buckets. It answered AccessDenied (Object is not shared) when the object had no share link (#9).
Bucket policies decide again, with the caller's IAM policies as in AWS: a Deny in either refuses, an Allow in either grants a user of the bucket's tenant. A statement naming "*" grants or denies requests without credentials, over the bucket ACL. The console decides the same way. Since 1.6.0 they were stored and never evaluated.
Console: editing an existing bucket policy loaded the API response envelope instead of the policy; saving it again failed.
A bucket policy cannot deny its own management: s3:GetBucketPolicy, s3:PutBucketPolicy and s3:DeleteBucketPolicy are decided by the caller's permissions.
PutBucketPolicy (S3 and console) refuses with MalformedPolicy what the evaluator would not enforce as written: NotPrincipal, NotAction, NotResource, other principal types and ARNs, actions outside S3, resources outside the bucket, condition keys other than aws:SourceIp and aws:SecureTransport. Such a policy was stored and read differently from what it said.
BlockPublicPolicy refuses a public bucket policy (403 AccessDenied) and RestrictPublicBuckets voids what a public policy grants to requests without credentials, as in AWS. RestrictPublicBuckets no longer voids public ACLs; IgnorePublicAcls does.
BlockPublicAcls refuses a request carrying an ACL that grants anything to everyone or to every authenticated user, with 403 AccessDenied, before storing anything: PutBucketAcl, PutObjectAcl, x-amz-acl on PutObject, CopyObject and CreateMultipartUpload, and the console ACL endpoints. It was stored and not applied. ACLs already set are kept.
maxiofs recover and maxiofs reconcile restore each version's Object Lock retention and legal hold. The object's sidecar keeps them, and a metadata update cannot change them there; a version stored earlier gets them at the encryption worker's first pass. They came back unlocked.
A node installed after a cluster setting changed takes the change, and the defaults it sends replace no setting of the other nodes. A default no one changed was dated by the start of the node that stored it: a node that joined later kept its replication factor of 1, sent it to the other nodes, and they took it and removed copies.
The Docker image keeps its data in the directory MAXIOFS_DATA_DIR names, where it is mounted. It always used /data, so data written with the directory mounted elsewhere was lost when the container was recreated.
A success or error dialog of the console given markup shows it laid out, sanitized. The temporary credentials just issued, the policy of a temporary session and other dialogs showed their HTML as text.
Failed raw replica overwrites restore the previous data and sidecar, including existing versions.
Raw replica rollback retains recoverable backups when restoration fails.
Multipart completion waits for encryption to stop on failure or cancellation.
Destination metadata lookup errors abort multipart completion before overwriting data.
PUT stores the retention and legal hold from Object Lock headers, or the bucket default retention, in the same metadata write as the object. A failure after the write no longer leaves the object unprotected.
Bucket metadata read errors fail PUT, DELETE and quota checks instead of treating the bucket as unversioned or without quota.
PUT and raw replica writes fail when the existing object's metadata cannot be read, instead of overwriting it without a backup.
Every copy between nodes carries retention and legal hold with the object: replication (raw or not), the initial sync of a new replica, the anti-entropy push and pull, and the stale-node catch-up.
Metadata-only changes (tags, ACLs, retention, legal hold, user metadata, restore status) reach the other nodes before the request returns, in order. A node that is down or fails gets them queued and delivered in order when it is back; they were sent in the background to healthy nodes only and lost after three attempts.
The initial sync of a new replica copies every version of every key with its version ID, including delete markers and keys they hide; it copied only the latest version of visible keys, under new version IDs.
A version copied from another node never replaces a newer latest version, and one already present is not counted twice in bucket and tenant usage.
Two nodes with a replication factor of 2 keep accepting writes while one of them is down, as a mirror. Writes answered 503 until the peer returned.
A tenant quota refusal inside a write answers 403 QuotaExceeded instead of 500 (S3 PUT, CopyObject, POST upload, console upload, rename and version restore). CopyObject and POST uploads also answer bucket quota and Object Lock errors with their S3 codes.
A multipart upload keeps user metadata apart from its storage fields and state: x-amz-meta-content-type no longer replaces the object's Content-Type, x-amz-meta-x-amz-acl no longer applies a canned ACL, and user metadata no longer enters the object's sidecar. An upload created before this release completes with its user metadata; a canned ACL it recorded is not applied, and the object is private.
Object Lock headers on a write need the permission to set them, as in AWS S3: s3:PutObjectRetention for a mode or retain-until date, s3:PutObjectLegalHold for a legal hold (S3 PUT, CreateMultipartUpload, CopyObject and console upload). s3:PutObject alone was enough. The bucket default retention needs neither.
CopyObject applies the Object Lock headers of the request to the copy; they were ignored.
A console upload to a tenant without a storage limit (MaxStorageBytes 0) is no longer refused as over quota.
CreateMultipartUpload applies x-amz-object-lock-mode, x-amz-object-lock-retain-until-date and x-amz-object-lock-legal-hold: validated when the upload is created, applied when it completes. They were ignored.
A write that misses HA quorum removes its local version even under the retention or legal hold it set; the client was told the write failed.
Encryption migration keeps a crash-recoverable backup of each object and restores it when the rewrite or its verification fails.
docs/API.md listed console routes that do not exist and missed many that do; its route tables now match the router.
Copies between nodes that re-encrypt the data (objects under a node's own key) carry tags, ACL, restore state and the multipart ETag; they were lost. Bytes that do not match the version's ETag are refused.
A copy of a version into a bucket whose versioning was suspended since is stored as that version; it replaced the current object. A copied delete marker stays a delete marker; it deleted the current object.
The initial sync of a new replica copies the current object without a version ID of a suspended bucket whose key also has older versions; it was skipped.
Changing the tags, ACL, retention or legal hold of a restored object keeps its restore status; it was erased.
The performance collector is created once per process. Each server construction replaced it while its throughput ticker was reading it, and left another ticker running.
In a bucket whose versioning is suspended, a write without a version ID over a version kept from before adds its size to bucket and tenant usage; it subtracted the kept version's size. Over a delete marker it counts the key again. The bucket statistics recount includes such an object.
In a bucket whose versioning is suspended, DELETE without a version ID adds a delete marker and keeps the versions from before, as in AWS S3; only a current object without a version ID is removed. It deleted the current version permanently, a version from before included, and added no marker.
Creating a bucket whose name has a removal still pending finishes that removal first. The startup sweep deleted the new bucket's directory; it now keeps the directory of a bucket that exists.
An object ACL belongs to the version it was set on. ACLs were also kept per key, so the next object at the key, by overwrite, new version or after a delete, inherited the previous object's ACL. ACLs kept per key by earlier releases are moved into the object at startup.
Tenant synchronisation read created_at and updated_at, stored as unix seconds, as timestamps and failed for every tenant; tenants did not synchronise between nodes.
Tenant synchronisation no longer copies storage usage and bucket count: each node counts what it stores. A synchronised tenant took the sending node's counters.
Storage usage changes no longer update the tenant's updated_at, which orders configuration between nodes: a write on one node made that node's older configuration win.
A tenant's bucket limit counts the tenant's buckets on every node, each once; the stored counter was per node and drifted. In the console a limit of 0 is no limit; it refused every bucket.
The console shows each tenant's bucket count and storage for the whole cluster. The tenant list computed them from the first 10,000 objects of each bucket on the local node.
The bucket statistics pass sets each tenant's storage on the node to what its buckets hold.
In a cluster, buckets reach every node. Creating a bucket, changing its configuration or ACL, and deleting it are sent to the other nodes before the request returns; a node that missed them is sent every bucket before the objects when it is caught up, when it becomes a replica, at every anti-entropy cycle and when a node starts. A bucket existed only on the node that created it: the other nodes refused its objects, so each write was kept on one node, and its configuration applied on one node. A cluster in that state converges after the upgrade.
A tenant deletion received from another node no longer removes a tenant changed here after the deletion, with its users and access keys. The tenants table keeps change times as unix seconds and they were read as timestamps, so the deletion always won.
The reconciliation of a node back from a partition read the tenants' change times the same way and failed whenever a tenant existed.
Stored credentials are decrypted after a restart when neither auth.jwt_secret nor auth.encryption_secret is configured. They were encrypted with a JWT secret generated anew each start, so a restart made share links, replication destination keys and identity provider secrets unreadable. Those stored before this release have to be entered again; the log names them.
The settings API no longer returns the JWT signing secret; the console showed it under Security.
In a cluster, shares, replication rules, inventory configurations and inventory reports reach every node. A change is sent to the other nodes before the request returns; a node that missed it is sent every row after the buckets. They existed only on the node where they were created, and worked only through it. Rows created before the upgrade reach every node at the next synchronization.
The sync of a replication rule's existing objects, and each scheduled run, queues every object of the bucket; it queued the first 10,000.
Replication of a tenant's bucket reads and lists its objects under the tenant. It looked for a bucket of that name without tenant.
An inventory report lists every object of the bucket; it listed the first 10,000.
An inventory report is stored as a client write is: encrypted, counted in bucket and tenant usage and quotas, and sent to the other nodes in a cluster. It was written directly to storage and metadata.
A completed inventory report keeps the path of its file; the path was not stored.
A share created for an object whose share expired but was not yet removed takes the new share's ID. The row kept the expired share's ID, so the ID returned did not exist.
A deletion sent to another node keeps the time it was made. Every node dated the deletions it received with the time of reception, again every 30 seconds, so deletions never expired and each looked newer than any later change.
An IAM policy, role, attachment or inline policy deleted and created again under the same name is no longer deleted on every node. A bucket created again lost its owner's policy; a bucket permission granted again to the same user was lost.
An object deleted and written again is no longer removed when a node that missed the new write is caught up; it is sent to that node. A copy written in the second of a deletion is kept.
Deletions and changes are ordered by time on every node: a copy changed after a deletion is taken, one changed before it is refused. A deletion is dated after the last change of what it removes.
Deleting a user, a tenant or a group records the deletion of the IAM policies they held; the other nodes sent them back.
Each node sends another the deletions it has not taken yet, in order; it sent its whole deletion log every 30 seconds.
Deleting a bucket deletes its shares, replication rules, inventory configurations and permissions, and the policies naming it also when the delete is forced. A share link of the deleted bucket served the object of the same key in a new bucket of the name without authentication, and replication rules sent the new bucket's objects to the former destination.
A node drops the notification configuration and integrity scans of a bucket it no longer holds. A new bucket of the name sent its events to the former webhooks. Those earlier releases left are dropped at start; the rows they left are dropped when a bucket of the name is created.
The deletion log of an earlier release holds no deletion times: at the first start its object deletions, and its deletions of IAM entities the node holds again, are dropped.
The systemd unit of the Debian and RPM packages no longer limits writes to /var/lib/maxiofs and /var/log/maxiofs: a data directory set elsewhere in config.yaml works without editing the unit, which every upgrade replaced. /usr, /boot, /efi and /etc stay read-only.
A fresh Debian install sets data_dir to /var/lib/maxiofs, as the RPM does. It kept ./data, under /opt/maxiofs, which the service could not write.
A node removed from the cluster is removed on every node. It was removed only on the node where it was removed, and the next node list from any other node added it back. The removal is recorded like a deletion and is final, as a node joins under a new ID every time; the removed node is answered 410 Gone and leaves the cluster.
A node that leaves the cluster is removed from the others. They kept it and kept sending it changes.
A change to a node (name, region, priority) is no longer reverted by the next node list of another node: the newest copy wins.
A drain reaches every node; it was taken only on the node where it was made, and the others kept writing to the drained node. A drain the other nodes cannot hold the copies of is refused (409); it answered success and did nothing.
A node marked dead after being unreachable is back in service when it answers again, and is caught up from its first missed write. It stayed dead for good. A drained node stays dead.
A cluster created or joined while the server runs replicates object writes and deletes. They were replicated only after a restart.
A node that boots with the cluster active encrypts its existing plaintext objects; the encryption worker did not start on it.
A node that joins a cluster drops the nodes of a cluster it was in before, its own former entry included.
A node away from the cluster for longer than 7 days, offline or cut off and serving clients, no longer brings back what the other nodes deleted meanwhile, and gets those deletions when it is back. A deletion is kept until every member of the cluster has been sent it and no catch-up that replays it is pending; deletions of buckets and of their shares, replication rules and inventory configurations too. A node removed from the cluster no longer holds them back.
A deletion that arrives by the deletion log removes the node's copy of the entity. It was only recorded; the copy was removed by the entity's own synchronization, which can be turned off.
A bucket deleted on one side of a partition and written to on the other is kept on every node with the objects written after its deletion. The node that wrote kept it with everything it held, and the other node never took it back.
Deletions are numbered in a sequence that never goes back. A number freed when the newest deletions were forgotten was given again, and a node already sent up to it skipped the deletion; a deletion received older than 7 days freed its number within the hour. After the upgrade every node is sent the whole log once.
Two writes of an object made in the same second on different nodes, or a rewrite while a node was away, are ordered by the time each was written. The anti-entropy took either by ETag order and could replace the last write with the one before it on every node. Each copy carries the time its write was made to the nanosecond; S3 LastModified is unchanged. Objects written before the upgrade are ordered by second.
A bucket deleted in the second of its last write is deleted on the other nodes; they kept it.
A delete marker reaches the other nodes with its time. Each node gave it the time it received it.
A group deletion received from another node keeps the time it was made; a revoked STS session too. Each node dated the revocations it received with the time of reception, again on every cycle, so they were never forgotten.
In a cluster, every node holds the entry of every object and the factor's nodes hold its data: the node that takes the write and the healthy nodes with the most free space. A node that does not hold the data reads it from one that does, a range alone; HEAD, listings, versions, tags and ACLs are answered from the entry. A write's data went to the first nodes by priority, and a node without it answered 404, listed only what it held, and kept what another node deleted or overwrote.
In a cluster, a delete reaches every node.
A read whose data no node holding it serves is answered 503 with Retry-After (S3 ServiceUnavailable), in the console too.
HEAD, GetObjectAttributes and RestoreObject with versionId, the download token and the presigned URL check the object's entry and do not open its data.
In a cluster, the parts, completion and abort of a multipart upload are made on the node the upload was started on, whichever node they reach, and listing a bucket's uploads covers every node. A load balancer that spread them answered NoSuchUpload. The upload ID names its node; an upload started before the upgrade is made where its requests arrive, as before.
In a cluster, the anti-entropy, the catch-up of a node that was away and the first copy to a new node send each node the entries it lacks and the data of the writes it holds, and compare every pair of healthy nodes. They compared a node with the first nodes by priority only and sent them every object's data. A node that holds a write and lost its data is sent it again.
In a cluster, the copies of a write follow the factor. A write with fewer copies than the factor, after a node holding one is removed or dead or the factor is raised, is copied to the healthy nodes with the most free space; one with more copies keeps the copy of the node that repairs it and those of the nodes with the most free space, and the other nodes remove theirs. This runs when a node is removed or marked dead, when the factor changes and after every full anti-entropy cycle. Each change of where a write's copies are is numbered, and every node keeps the newest, whichever order the changes, entries and copies arrive in.
A node under storage pressure that takes a write passes its copy to the node with the most free space among those holding the entry only, before the request returns. It keeps the copy when no node takes it.
Every node acts on a change of the replication factor, made on it or received from another node; the other nodes are told at once. They took the change within a minute and did nothing: only the node where it was made sent its buckets and repaired its copies.
A replication factor raised from 1 makes the copies the factor needs, one per object. Every object was copied to every node.
Objects written before the cluster kept where their data is are given, by the copy repair, the nodes that hold them: the first of those nodes keeps the factor's copies, the others remove theirs, and missing copies are made. They were copied to every node.
The copy repair started by a change of membership or factor runs once the first copies to new nodes end.
A slow node, one whose health check takes over a second, is in service and takes data like a healthy one. It was treated as down. A node whose disk is past the threshold is under storage pressure even when slow.
A node under storage pressure stays in service and only takes no new copy of data. It was treated as down: with a factor of 1 the other nodes answered that its buckets did not exist and left them out of their lists, and with a factor above 1 it missed every entry, delete and change until the pressure ended.
A node that was dead and answers again is compared on every write, and the anti-entropy sends a node with older locations of a write the newer entry. A node back from the dead kept the locations it had and the copies made again elsewhere.
A node that leaves the cluster, or is removed, drops the entries of the objects the other nodes hold and keeps those it holds. It kept every entry and listed objects it could not read.
In a cluster, the capacity Veeam reads (SOSAPI capacity.xml) for a bucket without a quota of its own, whose tenant has one, subtracts what the tenant uses on every node. It subtracted what the tenant used on the node that answered. The tenant is the bucket's: a user without a tenant was given the disk.
In a cluster, the capacity Veeam reads for a bucket without a quota is the room of the cluster, each byte on factor different nodes; dead nodes do not count. It was the disk of the node that answered.
A change of the replication factor in the console starts the sync jobs after the request is answered. The jobs were cancelled with the request.
In a cluster, a bucket is listed once in the console and in S3 ListBuckets. It was listed once per node holding it.
The degraded-cluster message is each node's own and says whether writes are kept on fewer nodes or refused. A node showed the message of the last node that sent it its settings.
A node that fails to apply a delete from another node answers so, and is caught up from it when it is back; it answered success. A node that keeps the object under a retention or a legal hold answers that it refused the delete, and is not marked unavailable for it.
A node that fails a replicated write is unavailable from that failure on: the dead-node threshold counts from it. The failure also turned a dead node back to unavailable.
An initial copy to a new replica that fails to send some objects says how many in its job, on the HA page too, and the replica is caught up from the oldest of them. They were missing on the replica until the daily anti-entropy.
A node that joins a cluster is sent the buckets and objects at once. It got them from the anti-entropy, 5 to 60 minutes after start-up.
The catch-up of a node that missed writes, and the anti-entropy, compare every version and delete marker of a key and send those the other node lacks. Only the current version was compared and sent: a node away while a key was written more than once never got the versions before the last.
A version or delete marker a node deleted is not stored again when a node that missed the delete sends it, or gives it, a copy.
A copy the anti-entropy sends keeps the object's tags, ACL and ETag, and is sent encrypted when the other node can decrypt it. It lost them.
Console: the language menu shows its flags on Windows. They were emoji, which Windows renders as two letters in Edge and Chrome. The language list in user preferences shows the language name only.
Console: the search icon of the bucket, user, access key, tenant, group and audit log lists is shown; the input covered it.
Console: the login page animation costs a quarter of the CPU without a GPU (remote sessions), from 2.7 cores to 0.6, and the page keeps 60 frames per second. The waves moved by repainting three full-screen gradients each frame and the card and icons blurred the moving background; the waves now move by transform, 20 times a second, along the same path, and nothing blurs it.
Console, dark theme: the light-blue metric icons (Total Objects, Admin Users) and the drop zone of the bucket page no longer show a light background; the dark shades they used were not defined. Checkboxes, date pickers, select lists and autofilled fields follow the theme.

Removed¶

internal/metrics collector: nothing used it, and it reported invented events. The gauges maxiofs_storage_bytes_total and maxiofs_storage_objects_total, never fed: maxiofs_bucket_bytes_total and maxiofs_bucket_objects_total carry each bucket's usage.
The forwarding of a request about a bucket to the node that held it, with its bucket location cache, GET /api/v1/cluster/cache/stats and POST /api/v1/cluster/cache/invalidate; the bucket and tenant-usage aggregation across nodes; the internal endpoints /bucket-exists/{name}, /buckets and /tenant/{id}/storage. Every node holds every bucket and every object's entry.
Console: the Replicated Buckets and Local Buckets cards of the cluster overview, and the Buckets column of the nodes page.
Bucket migration between nodes: the console Migrations page, POST /api/v1/cluster/buckets/{bucket}/migrate, GET /api/v1/cluster/migrations, GET /api/v1/cluster/migrations/{id}, GET /api/v1/cluster/buckets and the internal /migration/* endpoints. Every node holds every bucket, and the copies of each object go to the nodes with the most free space.
Console: the Node column of the bucket list and the node choice when creating a bucket.
Console: the Join Existing Cluster form of a standalone node. It sent the cluster token and an address to the endpoint that takes the join package of an existing node, and always failed. A node joins from the console of a node of the cluster: Manage Nodes, Add Node.
The stale-node reconciler and its endpoints /api/internal/cluster/state-snapshot and /api/internal/cluster/ha/objects/changed-since. It never ran: no node ever marked itself stale. A node away for long is sent what it missed by keeping deletions until every node has them.
The replica-first S3 read: each read was first sent to another node's cluster port, which serves no S3 request and answered 404, then served by the node that received it.
Internal endpoints used only by the previous bucket migration: /api/internal/cluster/objects/{tenant}/{bucket}/{key} (PUT, DELETE, HEAD), /bucket-permissions, /bucket-acl, /bucket-config and /bucket-inventory.
Console: the Make bucket public option when creating a bucket, and the bucket's is_public field. It was stored and never read; a bucket is made public with its ACL.
Console: the Authenticated Read option of the bucket ACL dialog. It granted nothing: requests with credentials are decided by permissions and the bucket policy. A bucket that has it through the S3 API still shows it.

Added¶

Prometheus alerts: MaxIOFSDown, S3ServerErrors, StorageFailures, DiskAlmostFull, SignInFailureBurst, BackgroundTaskFailing.
Prometheus tests: every metric family is exported once the server does what it counts; no label carries an object key; a refused request names no bucket; every background task is observed; every S3 route has an operation name. OAuth sign-in tests with and without two-factor authentication, server and console page; a code exchanged on another node; an identity provider changed through the database is rebuilt.
Console: a bucket policy is edited as rules instead of JSON: allow or deny; everyone or users of the bucket's tenant; actions from the permission catalogue; the whole bucket or objects under a prefix; source addresses and TLS. Common rules are added in one click. A policy set through the S3 API that rules cannot show is shown read-only and can be replaced.
Console: each card of a bucket's Security & Access tab says what it decides: Block Public Access what it voids, the bucket policy whom it grants or denies, the ACL that it applies only to requests without credentials. The policy dialog says when public policies are blocked.
Console: the Security & Access tab of a bucket edits its Block Public Access, with the same fields as bucket creation. The ACL card says when Block Public Access ignores the bucket's public ACL; the ACL dialog offers no public ACL while public ACLs are blocked. It could be set only when creating the bucket.
Tests for anonymous reads of a public-read bucket, with and without a share link and Block Public Access, and for BlockPublicAcls on every request that sets an ACL.
Bucket policy tests: the decision with IAM policies, the tenant boundary, requests without credentials, its own management, RestrictPublicBuckets, BlockPublicPolicy, conditions on source address and TLS, and what is refused when saved, through S3 and the console.
Object Lock recovery tests: a rebuilt store gets back the retention a write set, an extended retention, a legal hold on and one turned off; the sidecar keeps the state through multipart completion, a metadata update, a raw copy between nodes and the encryption worker's pass.
Console: the HA page shows the progress of a drain: the objects and bytes moved, and the expected end at the pace so far.
Drain tests between three complete nodes, factors 1 and 2: every object moved and readable, the drained node out of service for S3 and on every node, later writes on the others; a node being drained takes no copy and passes on the writes it takes.
Factor-1 tests between three complete nodes: one copy where the write is made, read and deleted from every node; a write moved off a node under storage pressure; an update from buckets kept on one node; the factor lowered to 1.
Cluster setting tests: a node installed after a change takes it; its defaults replace no change.
Write-path I/O accounting tests for PUT, overwrites, versioning and multipart uploads.
Multipart streaming regression tests for cancellation, read failures, rollback and retry.
Quota tests for concurrent writers, slow tenant checks, writes finishing or refused during a check, and deletes over the limit.
Object Lock tests for PUT headers, HA replica transfers and quorum rollback of protected versions.
Rollback tests for raw replica overwrites and failed encryption migration restores.
HA tests for mirror writes with a node down, the catch-up of a returning node (window, deletes, retries) and quota error codes.
Tests for suspended DELETE, bucket creation over a pending removal, object ACL ownership and migration, and tenant usage and synchronisation between nodes.
HA bucket tests between two complete nodes: every bucket change reaches the other node, in both directions; a node that was down is sent its buckets, then their objects; a new replica is sent the buckets first; deletions keep newer data; refusals. Lifecycle expiration on the coordinator only.
Encryption secret tests: stored once and kept across restarts; adoption re-encrypts every stored credential in one transaction; each component's credentials re-encrypted; a real join between two nodes; the coordinator gives its secret and no other node can; the settings never show the JWT secret.
Test for PutBucketReplication and GetBucketReplication through the S3 API.
Bandwidth tests through the S3 API: a tenant's cap paces a PUT, a multipart part, a GET, a range GET and a form upload.
Quota tests through the S3 API: a bucket or tenant over its quota refuses writes that grow usage and allows deletes, batch deletes and overwrites that shrink an object, then admits a write that fits once usage is under the limit. Tests for the tenant's storage after overwrites and for a suspended write over a kept version.
Two-factor sign-in tests: a code alone, the token of the first step used as a session, a session token used for the first step, a user disabled between the steps, lockout after the configured number of wrong codes and the limit per address. An unlock reaches the node where the account locked. Tests for the sign-in paths any node serves and for the user's address and TLS on a forwarded change.

Security¶

A two-factor sign-in needs the password step: login returns a two_factor_token valid for five minutes, and /api/v1/auth/2fa/verify takes it with the code. It took a user ID and a code: an account with two-factor authentication was reached without its password, with unlimited attempts, locked or not. A wrong code counts as a failed sign-in, the limit per address applies, and a locked account is refused. The login response no longer carries user_id.
Only an access token is a console credential; a token of any other type is refused. The check refused the refresh and download types only.

[1.7.0] - 2026-09-18¶

Added¶

MAXIOFS_BOOTSTRAP_ACCESS_KEY and MAXIOFS_BOOTSTRAP_SECRET_KEY give the administrator that key pair on a deployment that has none, so a compose stack comes up with credentials it already knows instead of having to call the API for them. Applied once: a deployment that already has a key ignores them, and changing them later does nothing. A pair nothing could sign with — too short, or carrying a slash or a space — stops the server rather than being ignored. Environment only: a credential does not belong in a file that gets committed. (#8, internal/auth/bootstrap_key.go)
maxiofs reconcile restores the metadata entry of any object that is on disk but missing from the index, reading its identity from the sidecar. The server already did this after an unclean shutdown; there was no way to ask for it. (cmd/maxiofs/reconcile.go)
maxiofs repair-pointers --drop-orphans removes version entries that name no bucket. They were reported as failures, which made a successful repair exit non-zero; nothing can reach them and no pointer can be rebuilt from them. (internal/recovery/repair_pointers.go)
scripts/s3-battery.sh — an S3 acceptance battery over a running server: keys that collide on a filesystem, versioning under shared prefixes, multipart integrity and metadata round-trips. It creates its own bucket and access key and removes both afterwards.

Changed¶

Object and version metadata writes release the bucket mutation lock before waiting for WAL persistence, allowing concurrent writes to share Pebble's group commit without weakening acknowledgements. Startup reconciliation uses the configured storage root; offline reconciliation accepts --storage-root.
Storage layout v2. Objects are stored one file per object under <bucket>/<aa>/<bb>/<sha256 of the key>, with no directories built from key components and no folder markers on disk. Buckets sit at the storage root; the tenant-qualified path is recorded in .maxiofs-bucket and in every sidecar. The root carries a .maxiofs-layout marker and the server refuses to start on a layout it does not read. (internal/storage/filesystem_layout.go)
The listing filters that hid those invented folders are gone with them, along with the bucket-deletion rule that treated a bucket holding only invented folders as empty. A folder-marker key is now an object like any other. (internal/object/manager.go, internal/metadata/)
The migration removes the folder objects earlier releases invented for parent prefixes, instead of giving each one a file of its own. It runs on an already-migrated installation too, so those entries do not outlive the release that stopped writing them. (internal/layout/migrate.go)
Replicated writes no longer create an object for the parent prefixes of the key either. The direct write path stopped doing it; the replication path still did, so a replica grew folders the primary did not have. (internal/object/raw_replication.go)
No object is created for the parent prefixes of an uploaded key. Every upload wrote a folder object per level into the index, which ListObjectsV2 hid but ListObjectVersions returned, so clients saw folders they never created. Entries from earlier releases are kept out of both listings. (internal/object/manager.go, internal/metadata/pebble_objects.go)
User metadata keys are returned as stored, in lower case. Header.Set canonicalised them, so x-amz-meta-proyecto came back as Proyecto. (pkg/s3compat/handler.go)
Object keys ending in .metadata or .metadata-staging are accepted. They were refused because the key was the file name and would have collided with a sidecar; the file name is a digest now. (internal/object/manager.go)
New setting storage.abort_incomplete_multipart_days: aborts multipart uploads left incomplete for that long in buckets with no AbortIncompleteMultipartUpload rule of their own. Default 0, which keeps the AWS behaviour of never aborting on its own. (internal/lifecycle/worker.go)
RemoveDirectory retries like single-file removal already did. On Windows a file another process holds open makes a single attempt fail, which is how a deleted bucket's objects were left on disk. (internal/storage/filesystem.go)
Multipart cleanup no longer drops the upload record when the part files could not be removed. Discarding that error left parts on disk that nothing could reach, since every cleanup path — lifecycle rules included — lists uploads from the index. The upload directory is removed once the parts are gone, and parts whose record is already lost are discarded at startup. (internal/object/manager.go, internal/server/multipart_sweep.go)
Bucket deletion records the removal in the same atomic write that drops the index entry, and clears it once the directory is gone. A removal that fails is finished at the next start instead of leaving the objects on disk with no marker, invisible to the index. The objects go only because someone ordered the bucket gone, never because the index happens not to mention them. (internal/metadata/pebble_store.go, internal/server/bucket_removal_sweep.go)
Storage layout migration. An existing installation is moved to the new layout on the first start of this release, bucket by bucket, and maxiofs migrate-layout runs it by hand with --dry-run. Objects are hard-linked into place and verified before the original is removed, an interrupted run resumes, and it refuses to start when two buckets share a name across tenants or a bucket is named after a tenant. Objects the index names but disk no longer holds are reported, not deleted, and so are directories that still hold data for a bucket the index does not know. (internal/layout/)
storage.Backend takes an object reference (bucket, key, versionID) instead of a joined path string, with dedicated calls for bucket lifecycle and for multipart parts, which live outside any bucket. The on-disk layout is unchanged. (internal/storage/backend.go)
The object sidecar records which object it belongs to (maxiofs-bucket, maxiofs-key, maxiofs-version), and maxiofs recover and reconcile take the identity from there instead of inferring it from the path. Sidecars without those fields still fall back to the path. (internal/storage/filesystem_layout.go, internal/recovery/)
The packaged systemd unit allows 10 minutes for a clean stop instead of the 90 seconds systemd defaults to. A server with a large index was killed mid-shutdown, which costs a WAL replay on the next start. (debian/maxiofs.service, rpm/maxiofs.service)
Vitest 4 -> 5 for the frontend test suite, with @vitest/ui and @vitest/coverage-v8 on the same major. engines.node now excludes Node 25, which the toolchain does not support.

Fixed¶

Interrupted overwrites are recovered from the oldest retained copy matching the current index entry. Obsolete copies no longer hide newer recoverable backups. Copies without an index entry remain available for operator review without resurrecting deleted objects, and rollback errors prevent startup. (internal/rollback/, internal/server/interrupted_writes.go)
Object, version and multipart-part commits fsync before returning, before the writer can discard its rollback copy. This closes the window where an acknowledged part survived on disk with stale metadata after a process kill. (internal/metadata/)
Reconciliation shares object-key locks with normal operations and re-reads sidecars inside the lock. It no longer overwrites a concurrent PUT's metadata and detects equal-size overwrites within the same second. (internal/recovery/reconcile.go, internal/storage/object_lock.go)
A multipart completion that fails after publishing the assembled object restores the previous one. The hook that puts it back was armed after the assembly, so a storage error raised once the bytes were already in place — ENOSPC on sync, a device error after the rename — returned early, deleted the retained copy and left the new content under the old object's metadata. (internal/object/manager.go)
Replacing a multipart part no longer destroys the part it replaces when the new one cannot be recorded. The part file was deleted on a failed metadata commit while the committed row still advertised the previous ETag, so ListParts promised bytes that were gone and the upload could only fail at completion. The stored part is retained before the replacement lands and put back if the commit fails. (internal/object/manager.go)
The sidecar of an object assembled from parts records the multipart ETag. It is the MD5 of the part digests and cannot be recomputed from the assembled bytes, so an entry rebuilt from disk was given a whole-object MD5 the client had never seen. (internal/object/manager.go, internal/recovery/recovery.go)
A session policy now covers the operations whose resource is not named in the URL. It was evaluated against the request URL alone, so a copy authorized its source with the base user's permissions: a session denied s3:GetObject on a bucket could still read it by copying the object somewhere it was allowed to write, and the same held for the source of an UploadPartCopy. The policy travels inside the resolved permission set now, so every authorization decision consults it, including the ones a handler makes about a second resource. A multi-object delete authorizes each key, and a request proxied between nodes carries the session in a signed header the receiving node revalidates. (internal/auth/sts.go, internal/auth/policy_set.go)
A GET concurrent with an overwrite could return one write's metadata with another's bytes. The index was read and the file opened afterwards with nothing holding the two together; a response declared 3 bytes and delivered 14, and the ETag belonged to neither. Reads and writes serialize per key, and a reader keeps the generation it opened while a writer replaces the name — on Windows too, where replacing an open file needs POSIX rename semantics. (internal/object/manager.go, internal/storage/filesystem.go)
A global administrator can no longer set a quota on another tenant's bucket. The route asked for a generic capability and honoured ?tenantId=, so the bucket's owner never entered the decision. It evaluates maxiofs:PutBucketQuota against the bucket's tenant now. Reading a quota stays available to a global administrator — it is the audit view — and a tenant administrator holds both actions for its own buckets. (internal/server/bucket_quota_handlers.go)
A request for a bucket subresource maps to its own action. GET /bucket?encryption and a dozen others were authorized as ListBucket, so a policy granting exactly s3:GetEncryptionConfiguration was refused; the Object Lock configuration had no mapping at all. (internal/auth/s3auth.go)
Every setting can be overridden by an environment variable, not just the top-level ones. MAXIOFS_STORAGE_ROOT and the rest of the nested keys were read as MAXIOFS_STORAGE.ROOT, which is not a name a shell can export, so they were silently ignored. (internal/config/config.go)
The docker run line in the README mounted the volume at /var/lib/maxiofs, which the image does not use: everything written went to the container's own layer and was lost with it. The image stores at /data. (README.md)
Console login works with auth.enable_auth: false. Credential validation returned a synthetic "anonymous" user that exists in no table, and the next step of the login read its 2FA status back from the database — user not found, 500. Both the console and the S3 path now answer with the real admin user. (#7, internal/auth/manager.go)
config.example.yaml describes what the server reads and nothing else: 677 lines down to 101. storage.enable_encryption and auth.users_file are gone, since nothing read them — the first was already logging "deprecated and ignored" at startup while the example explained how to use it. storage.encryption_key now says what it does: it seeds the master key on the first start of an empty deployment and is ignored afterwards, because the key lives in the database, which is what has to be backed up.
auth.access_key removed: nothing ever read it. Access keys are created once the server is running. (#8)
auth.secret_key renamed to auth.encryption_secret. It was documented as a default credential and is not one: it encrypts the stored secrets of identity providers, replication targets and share links. The old name is still read, with a warning, because a deployment that set it has data encrypted with that value.
/debug/pprof/* answers a global administrator instead of refusing everyone. The routes hang off the root router, outside the subrouter that authenticates, so no user ever reached the check and every request was a 401. (internal/server/profiling.go)
The S3 rate limit answers SlowDown in an XML error document instead of a plain 429. A client that hits the limit backs off and retries; a bare 429 is not an S3 error and aborted the transfer. security.ratelimit_api_per_second default raised 100 -> 1000 per access key, and 0 now means no limit rather than falling back to the default. Existing installations keep the value they have. (internal/auth/api_rate_limiter.go)
Administration granted by a policy is recognised as administration. IsAdminUser read the role alone, so a user holding maxiofs:SuperAdmin without it was refused across the console. maxiofs:TenantAdmin is not read there: every holder of the tenant-admin role is given it when policies resolve. (internal/auth/manager.go)
A global administrator no longer changes a tenant's bucket from the console. Twenty-one mutation routes — lifecycle, tagging, CORS, ACL, policy, versioning, object lock, notifications, encryption, public access block, website and inventory — asked only for the bucket:configure capability, which says nothing about whose bucket it is, and then honoured ?tenantId=. They resolve the bucket's owner and evaluate the S3 action now, the way the read routes already did. A bucket quota is the exception and stays writable: it is the ceiling the operator imposes on the tenant. (internal/server/capability_guard.go, internal/server/console_api.go)
IAM roles are confined to the tenant that owns them. GetRole, ListRoles, DeleteRole, UpdateAssumeRolePolicy and the policy actions on a role resolved by name alone, so an administrator of one tenant read, changed and deleted another tenant's roles. A deployment-wide role stays readable by every tenant — it can be assumed — and changeable only by a global administrator. Assuming a role across tenants was already refused. (internal/server/iam_aws_api.go)
IAM managed policies belong to a tenant. iam_policies gained tenant_id: a tenant sees and changes its own, reads the deployment-wide ones to attach them, and reaches neither another tenant's nor the built-in documents. Every policy that exists stays deployment-wide, since nothing recorded who created it; the ones to re-own are named in the log at startup. Migration 20. (internal/auth/iam_store.go, internal/server/iam_aws_api.go)
An IAM action on a group resolves the caller's own group. GroupName was looked up among the deployment-wide groups whoever asked, so a tenant could not manage its own groups by name and could attach a policy to a global one. (internal/server/iam_aws_api.go)
Cluster note: upgrade every node. A 1.6.0 node that re-broadcasts a policy writes the row without tenant_id and turns it deployment-wide. Roles are unaffected. (internal/cluster/iam_sync.go)
Force-deleting a bucket records the removal like an ordinary delete does. It dropped the index entry outright, so a kill before the directory was removed left the objects on disk with nothing to finish the job. (internal/metadata/pebble_store.go)
The share dialogs read in dark mode. Their bodies are raw HTML, so they carried light-only classes — a white panel and near-black text on a dark page. They use the theme tokens now. (web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/components/ObjectVersionsModal.tsx)
Stop sharing is an item of the actions menu, both for a selected object in the bucket view and in the object detail view, shown only for objects that are actually shared. Revoking a link was only reachable as a secondary button inside the dialog that opens from Share public link, where nobody looks for it. (web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/components/ObjectDetailsView.tsx)
The object actions menu and the bucket actions menu are rendered outside the page's scroll container. Both were positioned inside it, so the menu was clipped by it and could not open over the bar above — leaving actions such as revoking a share unreachable for rows near the top. (web/frontend/src/components/ui/DropdownPanel.tsx)
Restoring a delete marker is refused when no earlier version is behind it. Removing the marker left no object at all, which read as the restore having deleted it. (internal/server/object_extra_handlers.go)
Data loss: a folder marker destroyed a same-named object. PUT key/ deleted any file occupying a component of that path, along with its metadata sidecar and the wrapped DEK, then created a directory there. Listings and GET kept reporting the original size and ETag while the body read empty. Non-versioned buckets only.
Data loss on Windows and macOS: two keys differing only in case shared one file. Report and report are distinct S3 keys, but the key was the filename, so the second write replaced the first and a GET returned the other key's bytes under the original's Content-Length.
Both collisions are gone with layout v2: the filename is a digest of the key, so 2026 and 2026/, Report and report, and a key ending in .metadata are all distinct files. Key length and Windows path limits stop applying too.
Test setup assigned globalThis.localStorage and globalThis.sessionStorage directly; under Vitest 5 those are getter-only and every suite failed to load. Installed with Object.defineProperty. (web/frontend/src/test/setup.ts)
Setting descriptions, types and categories are refreshed on startup instead of being written only on first boot, so an upgraded install no longer shows the text of the release it was first installed with. Values are untouched. (internal/settings/manager.go)
security.iam_api_enabled description said Veeam creates its own identities and credentials, and that disabling it stops the advertisement "to Veeam". Veeam signs with the key it is given, and system.xml is served on the object key with no User-Agent check.
A failed overwrite replaced the stored object anyway. The bytes were written before the index entry, so a metadata failure returned an error while a later GET served the new content under the previous ETag and size. The object that was there is restored. (internal/object/manager.go)
A part could be replaced while the completion was assembling it, producing an object built from bytes its validated ETag never described. UploadPart, completion and abort now exclude each other per upload. (internal/object/manager.go)
A delete authorized for one key could remove another. The manager rewrote folder to the folder/ marker after the permission check, so an explicit Deny on the marker was bypassed. Callers resolve the key before authorizing. (internal/object/manager.go, pkg/s3compat/handler.go, pkg/s3compat/batch.go, internal/server/console_api.go)
The encryption worker's cancel could be overwritten by a second start, leaving the first pass with no way to stop it — it outlived the metadata store and wrote to a closed Pebble. Shutdown now cancels as it closes the gate. (internal/server/encryption_worker.go, internal/server/server.go)

[1.6.0] - 2026-08-26¶

Added¶

STS temporary S3 credentials — short-lived credentials (ASIA… key, secret, session token) that project an existing user instead of creating a new identity. Sessions are revocable rows in the auth database, replicated across the cluster and swept hourly. Console UI under Access Keys; security.sts_max_session_duration (default 12 h). (internal/auth/sts.go, internal/server/sts_handlers.go)
Session policies — an optional policy attached at issuance, evaluated as an intersection, so it can only narrow what the user already has. Constructs that cannot be enforced exactly as written (Principal, Condition, oversized documents) are rejected at issuance rather than ignored. (internal/auth/sts_policy.go)
STS federation for headless clients — POST /sts/ldap-identity and /sts/web-identity trade LDAP credentials or an OAuth token for S3 credentials, so scripts and backup jobs need no permanent keys. Disabled by default via security.sts_federation_enabled. (internal/server/sts_federation.go)
AWS STS protocol on POST / of the S3 port — GetSessionToken, AssumeRole, AssumeRoleWithWebIdentity, AssumeRoleWithLDAPIdentity, so aws sts … --endpoint-url <s3-endpoint> works with no MaxIOFS-specific client code. A temporary credential cannot mint another. (internal/server/sts_aws_api.go)
AWS IAM protocol on the same endpoint — users, access keys, managed policies with versions, inline policies, attachments and roles. Requires permanent credentials and the new iam:manage capability; gated by security.iam_api_enabled. (internal/server/iam_aws_api.go)
IAM is the authorization model — roles, bucket permissions and attached policies are all IAM policies now, evaluated AWS-style (default deny, explicit Deny wins). An installation's existing permissions are converted once, on the boot that upgrades it; the old capability and bucket-permission tables are no longer read on the request path. Migration 18. (internal/auth/policy_migration.go)
Roles are IAM roles — admin, user, readonly and guest are rows in iam_roles with an optional trust policy deciding who may assume them. A role session carries the role's permissions, resolved live so edits and deletions reach sessions already issued. The separate Role Capabilities screen is gone. (internal/auth/iam_role.go)
Permissions are picked from a catalogue, per bucket — every assignable permission has a group and a readable label and can be granted on individual buckets. Nobody edits a policy document; the document is what the selection is stored as. The only JSON left in the console is a role's optional trust policy. Console page under Users → IAM, in all 9 languages. (internal/auth/iam_catalog.go, web/frontend/src/pages/iam/)
IAM entities replicate across the cluster on the access-key schedule with an immediate push after each change; deletions travel as tombstones so a lagging node cannot resurrect them. (internal/cluster/iam_sync.go)
Large downloads and folder ZIPs stream to disk rather than being assembled in the browser tab, so a download is no longer limited by tab memory. They navigate with a two-minute token naming one object or prefix, accepted only on the download routes. (internal/server/download_token.go)
Transfers are bounded by progress instead of elapsed time — a request is cancelled once its body has moved no bytes for 60 s, so a large object is never cut off merely for taking long. (internal/transfer)
s3:GetBucketObjectLockConfiguration and s3:PutBucketObjectLockConfiguration — a bucket's Object Lock settings had no permission of their own, separate from the per-object retention permissions.
Forced password change — an administrator can require a new password at the next sign-in. The API refuses everything but the password form until it is set, and only setting a password lifts it. Migration 19.
Read-only notice on tenant buckets — a global administrator browsing a tenant's bucket is told why the write controls are disabled.

Security¶

Anyone could act as any user on the S3 port by replaying three headers from an observed inter-node request — the proxy signature covered neither the forwarded identity nor the request, so the caller chose who to be. It now covers the method, the full URI, the body digest, the identity and a nonce that is remembered for as long as the signature is accepted. (internal/cluster/proxy.go)
Header-signed S3 requests never checked their timestamp, so a captured signature stayed valid indefinitely. Requests outside a 15-minute window are refused, as on AWS. (internal/auth/manager.go)
Suspending a user did not revoke their S3 access keys — the console locked them out while their keys kept full permissions. Both signature versions now check the user's and the key's status.
A node accepted the internal cluster headers from anyone — sending X-MaxIOFS-Proxied: true was enough to be treated as an already-routed request, so a write landed on whichever node received it instead of the one owning the bucket. A request that has not proved it comes from a peer has those headers removed before any handler reads them. (internal/server/server.go)
The inter-node signature did not cover the query string, where the internal API carries the bucket, the key and the version — so a signature was valid for a request pointing somewhere else. Signing and verifying now share one definition of what is covered, instead of building the payload separately at each end. (internal/clusterauth/)
Inter-node requests could be replayed within a 60-second window — the nonce was generated and signed but never recorded. It is now remembered for as long as the signature is valid. (internal/middleware/cluster_auth.go)
Streamed object replication left the body unauthenticated, so a peer's bytes could be substituted in flight. The sender now signs the digest it already holds and the receiver verifies while writing, without buffering. (internal/middleware/body_digest.go)
A client-supplied chunk size was passed straight to an allocation — one small aws-chunked request could panic the process or exhaust its memory. Sizes are bounded and negatives rejected. (pkg/s3compat/aws_chunked.go)
No XML or JSON endpoint limited its request body, so a single authenticated request could drive the node out of memory. All of them now cap it. (pkg/s3compat/body_limits.go)
A global administrator could write to any tenant's buckets from the console — the guard granted them everything without consulting policies. Access across a tenant boundary is now read-only, matching the S3 API. Deleting or configuring a tenant's bucket is now the tenant administrator's to do. (internal/server/capability_guard.go)
A presigned URL whose signer could not be resolved was granted full access instead of being denied — the lookup errors were discarded, leaving the request on the unauthenticated path where a valid signature alone stood in for IAM. (pkg/s3compat/presigned.go)
Veeam's SOSAPI documents (capacity.xml, system.xml) were readable by any member of the bucket's tenant, regardless of permissions on that bucket. They now require s3:GetObject like any other object. (pkg/s3compat/handler.go)
Five endpoints performed no authorization beyond requiring a session — SelectObjectContent (which also crossed tenants), handleDownloadZip, HandlePresignedPost, handleUpdateBucketOwner and the replication endpoints, which additionally returned destination credentials in plaintext. All now ask for the permission the operation needs, and the destination secret is write-only.
A tenant administrator could take over the global administrator account — handleChangePassword matched the admin role without comparing tenants and skipped the current-password requirement for other users, and handleCreateUser validated no roles. Granting either administrator role is now a global administrator's decision.
A multipart upload could be operated on through a key the caller held rather than the one it belonged to — the permission check read the URL while every operation afterwards went by uploadId alone. All four entry points now assert the upload belongs to the authorized object, answering NoSuchUpload on a mismatch.
Ten bucket subresources were guarded by an unrelated permission — encryption by CORS, logging/website/inventory by lifecycle, notification and public-access-block by bucket-policy, ownership by ACL. Each has its own catalogued action now, including s3:PutBucketReplication.
ACLs could overrule the policies for an authenticated caller — every permission check fell through a widening cascade on denial, so an explicit IAM Deny was overridden and a bucket ever set to authenticated-read-write handed DELETE to every credential in the deployment. The cascade is gone; ACLs and public access still decide anonymous requests. (pkg/s3compat/handler.go)
A credential without a tenant reached every tenant's buckets — the boundary was enforced in one direction only, and because an ARN names a bucket by bare name, a grant on the global backups matched every tenant's backups. It is required in both directions now; only maxiofs:SuperAdmin crosses it, read-only. (pkg/s3compat/tenant_boundary_test.go)
Naming a specific version was not a permission — object subresources checked only the current-object action while reading ?versionId, so s3:GetObject reached the tags, attributes and retention of any historical version. CopyObject and UploadPartCopy read the destination's version instead of the source's.
Anyone signed in could set a bucket's default Object Lock retention — the guard compared the caller's tenant against itself, a condition that can never be true. A compliance-mode default of ten years would have locked every subsequent upload permanently, for everyone. (pkg/s3compat/handler.go)
Overriding governance-mode retention was granted by holding the admin role, so the catalogued s3:BypassGovernanceRetention granted nothing while every administrator could bypass protection regardless of their policies. It is asked of the caller's policies now.
A revoked permission came back on its own in a cluster — removing a bucket grant recorded no tombstone, so the next sync round restored it, and a deleted bucket's owner-<bucket> policy returned to hand full access to whoever recreated the name. Revocations and bucket deletions record tombstones now.
Turning off access-key synchronisation also stopped authorization from replicating, so revocations never reached the other nodes, which went on granting what had been taken away. The flag is separate now; the interval is still shared.
A tenant administrator could read and rewrite another tenant's users' permissions, handleShareObject published objects out of any tenant named in the query string, the cluster bucket inventory listed every tenant's buckets to any caller, and handleGeneratePresignedURL signed a URL for any object while gating its cross-tenant override on a role that exists nowhere in the repository.

Fixed¶

Shutdown closed the stores while background workers were still writing, panicking Pebble. All 23 components are stopped and waited for first. (internal/bgwork/, internal/server/server.go, internal/server/components.go, internal/metadata/pebble_store.go)
A bucket that ever held a key with / could not be deleted over S3 — folder markers are hidden from listings but counted as content. They are now removed with the bucket. (internal/metadata/pebble_store.go)
Flat listings returned short pages — hidden entries (folder markers, delete markers, internal files) consumed the MaxKeys budget, so a page could come back with fewer objects than existed. (internal/object/manager.go)
S3 jobs that outlive the request were not part of the shutdown wait: realtime replication on PUT and DELETE, and multipart completion. They now run through the server's worker. (pkg/s3compat/handler.go, pkg/s3compat/multipart.go)
max-keys=0 was not honoured — it now returns an empty listing with KeyCount 0, as on AWS. (pkg/s3compat/handler.go)
An unparseable Range header was silently ignored and the whole object returned; it now answers InvalidRange with Content-Range. (pkg/s3compat/handler.go)
Deleting a version that was itself a delete marker reported x-amz-delete-marker: false. (pkg/s3compat/handler.go)
The cluster TLS listener was replaced while shutdown was reading it, an unsynchronised field write, and its serve goroutine called logrus.Fatal — exiting the process with no shutdown at all. (internal/server/server.go)
Deleting the last administrator of a tenant left it with no operator. Deleting the tenant itself is the way to remove it. (internal/server/console_api.go, internal/server/console_idp.go)
Four background goroutines wrote to the stores untracked: bucket quota alerts (on every object write), tenant quota alerts (on every upload), scheduled replication sync, and dead-node catch-up sync. (internal/bucket/manager_impl.go, internal/auth/manager.go, internal/replication/manager.go, internal/cluster/dead_node_reconciler.go)
Remote log targets (HTTP, syslog) were never flushed on shutdown — buffered entries were discarded. (internal/server/server.go)
The inter-node object endpoints answered 200 {"success": true} without storing the object on their no-object-manager fallback. Removed; they refuse the transfer. (internal/server/cluster_object_handlers.go)
Storage-pressure events were emitted from an untracked goroutine and could arrive out of order with the health check. (internal/cluster/health.go)
An alert that fired before the browser connected was never shown — on connect the server reported only that the disk was back to normal, never an alert still active, and it ignored the critical threshold. It now syncs the current state, and active disk alerts go to global administrators only. (internal/server/sse_notifications.go)
Repeated alerts for one condition crowded out every other notification — each was a new entry in a list of three. A notification now replaces the one it supersedes, matched by condition and by the node, bucket or tenant it concerns. (web/frontend/src/hooks/useNotifications.ts)
Successful deletions were styled as failures in the audit log — removing a user, a tenant or an access key was painted red with an error icon by a hardcoded event list. Only a failed status is. (web/frontend/src/pages/audit-logs/index.tsx)
Writes forwarded to the node that owns a bucket escaped the tenant's quota — one of the two proxy paths dropped the content length, so the body arrived chunked and the quota check read the missing length as nothing to account for. A tenant could exceed its limit indefinitely by writing through any other node. (internal/cluster/proxy.go)
Paging through object versions lost some and returned others twice — versions are ordered newest-first within a key, but resuming compared version ids as text, which is a different order. A client walking the markers saw neither every version nor each one once. (pkg/s3compat/versioning.go)
Every delete subtracted from the bucket twice — the object manager and the S3 handler each adjusted the size, the object count and the tenant's quota. Removing one version of an object took away two versions' worth and hid the object from the bucket's count while it was still there. (pkg/s3compat/handler.go)
Deleting a folder destroyed every object under it — the folder marker and its prefix are the same path on disk, and the delete removed the whole directory while leaving the objects in the metadata store. They stayed listed, with their sizes, and every read of them failed. (internal/storage/filesystem.go)
A failed CompleteMultipartUpload destroyed the object already stored under that key — the parts are combined onto the live path, and the error path deleted it. The previous object is now restored. (internal/object/manager.go)
The cluster brought deleted objects back — the repair scrubber treated "the peer does not have it" as "the peer is missing it", with nothing to distinguish a deletion. It now consults the deletion log and removes its own stale copy instead. (internal/cluster/anti_entropy.go)
If-Match and If-None-Match rejected the list form — a header naming several entity tags was compared as one opaque string, so a client offering any of the versions it holds was refused. * and weak validators are understood too, and a failed precondition now answers with the XML error an SDK can map instead of a bare status. (pkg/s3compat/handler.go)
A Range header the server could not parse was an error — RFC 7233 says an unparseable range is ignored and the whole object returned; a proxy or downloader that sent a range in any other unit got a refusal where AWS delivers the file. A well-formed but unsatisfiable range still answers 416. (pkg/s3compat/handler.go)
CompleteMultipartUpload answered 200 OK with a NoSuchUpload error inside it — nothing has been written at that point, so it is an ordinary 404. A client checking the status line reported an object as stored that does not exist. (pkg/s3compat/multipart.go)
ListMultipartUploads ignored prefix and never echoed Prefix or Delimiter, so a cleanup tool asking for its own namespace received every upload in the bucket — and aborting those destroys other workloads' transfers. (pkg/s3compat/multipart.go)
NextKeyMarker was url-encoded twice when encoding-type=url was requested, so a client resuming a version listing asked for a key that does not exist. (pkg/s3compat/versioning.go)
Every miss scanned the whole bucket's versions — a lookup for one object listed and deserialised every version entry in the bucket and filtered afterwards. Backup tools probe with HEAD before each write, so a job writing new objects into a large bucket stalled the node. It reads only that key's versions now. (pkg/s3compat/handler.go)
DELETE ?tagging&versionId= deleted the version instead of its tags, so a metadata-only call from any SDK destroyed data. (internal/api/handler.go)
Objects whose metadata sidecar was lost could be served as a successful download of unreadable bytes — the check refused only on evidence of a mismatch, so bytes it could not identify passed. It now requires positive proof that the stored bytes are the recorded object. (internal/object/storage_identity.go)
Concurrent writes to one object could leave its bytes and its metadata describing different things — eight operations that read, modify and write the same object did so without serializing, including tagging, ACLs, retention, legal hold and version deletes. (internal/object/manager.go, internal/storage/filesystem.go)
Bucket size and object counts drifted apart from what a recount produced — the running counters and the full recount disagreed on whether versions and folder markers count. Size now means every version, which is what occupies disk; the count means the keys you can see. Deleting a version that was not the latest returned nothing to the bucket or the tenant's quota, and deleting the latest with another version underneath returned nothing either. (internal/metadata/pebble_store.go, internal/object/manager.go)
Tag-filtered search lost one object per page — the same marker mistake as the object listings, in the path that fix did not cover. Where a listing stops and which marker it returns are now one decision. (internal/metadata/pager.go)
Cluster-wide settings never propagated between nodes — every entry synced with a timestamp of zero and was skipped as stale, so a node could show a replication factor its peers did not have. (internal/cluster/sqlite_time.go)
A replication factor of 2 confirmed writes that reached no replica, and a factor of 3 refused writes when any one node was down. Both now require a majority of the copies, counting the local one. (internal/cluster/ha_object_manager.go)
The repair scrubber pushed every object to every healthy node, ignoring the replica set and consuming several times the provisioned disk.
Syncing a replication rule while it was being deleted could panic the process, because the lock was released by looking it up again rather than by holding it. (internal/replication/manager.go)
Dialogs did not hold the keyboard — Tab walked out of the modal into the page behind the overlay, where every control was still reachable and could be activated while the user believed they were answering the dialog. Focus is trapped, returned to whatever opened it, and the dialog announces itself to assistive technology. (web/frontend/src/components/ui/Modal.tsx)
The bulk-delete dialog said one number and deleted another — filtering the view with the search box left the selection intact but narrowed what the action resolved, so "10 selected" quietly deleted 2. It also read in English inside an otherwise translated console, which is the worst place for it: twelve strings across the confirmation, the progress bars and four notices are now translated in all nine languages.
Group operations reported nothing when they failed — creating, renaming, deleting a group and adding or removing a member all discarded the error, leaving the form open and the user pressing the button again.
Dates were formatted in US English regardless of the interface language, including relative times like "3 hours ago", and a missing translation key rendered its own name to the user.
The bucket list re-fetched every bucket's scan history every 30 seconds and discarded the answers — 120 requests a minute on an installation with 60 buckets.
The login page's progress and welcome messages went nowhere, because it renders outside the layout that hosts them, and an untranslated SSO error code displayed the lookup key instead of the code.
Deleting a bucket from the console could delete another tenant's — the tenant was resolved by bucket name, and names repeat across tenants. (web/frontend/src/pages/buckets/index.tsx)
The console object browser showed only the first 1,000 objects with no indication there were more, so search and select-all silently operated on that page alone. (web/frontend/src/pages/buckets/[bucket]/index.tsx)
Every quota-checked overwrite leaked a file handle, exhausting the limit under a backup job.
A two-node cluster could never elect a coordinator, so no configuration change could be made at all — a candidate wrote its own vote into the lease table before the election was decided, and each node then defended its hopeful lease against the other. Votes are separate from leases now, with a pre-vote round and a randomised campaign delay. (internal/cluster/leader.go)
A cluster that loses a node now re-elects among the nodes still alive — the majority was counted over every node ever known, so the death of one of two left the survivor unable to elect anybody, permanently, including to remove the dead node. It is counted over the nodes that answer the round; the returning node rejoins as an ordinary member and syncs. Two nodes that cannot see each other will each lead what they can see, and entity sync reconciles them by updated_at on reconnect. (internal/cluster/leader.go)
Every immediate cross-node push after a configuration change was cancelled by its own HTTP response — TriggerSync ran on the request's context inside a goroutine that outlives the request, so changes took up to a minute to propagate instead of a moment. Seven of the nine sync managers were affected. (internal/cluster/sync_trigger.go)
Two writers of the same bucket document lost each other's work — UpdateBucketMetrics locked and UpdateBucket did not, so quotas, versioning and Object Lock configuration silently rolled back. They share the mutex now, with the counters owned by the metrics writer and everything else by the configuration writer.
Deleting one version of an object left the main entry describing data that no longer existed — listings advertised objects whose every GET failed, and a hard kill during the repair made it permanent. The delete and the promotion of the next version are one batch now. (internal/metadata/pebble_objects.go)
An object whose metadata sidecar was lost had its raw ciphertext served as the object with HTTP 200 — nothing signalled a problem, so a backup tool would have restored garbage. Derived metadata now says it is derived, and bytes that cannot be identified are refused. (internal/storage/filesystem.go)
The storage backend never called fsync — not on the data file, the sidecar carrying the wrapped DEK, or the directory — while the metadata store has fsynced its WAL every second since 1.5.2, so after a power cut Pebble reported objects whose bytes were still only in the page cache. Cost of the added durability: +16% on a 10 KB object, +23% on 1 MB, +12% on 10 MB. (internal/storage/filesystem.go)
A tenant using versioning was slowly locked out of its own quota — versioned deletes never returned the freed bytes, and tenant storage does not clamp at zero, so it drifted in the direction that denies service until a manual recount.
The delete paths took no per-key lock, so two concurrent deletes each debited the same size and deletePermanently debited twice for one removal.
An upload that failed mid-write leaked a goroutine — FilesystemBackend.Put returned without draining the reader, leaving the encryption goroutine blocked forever holding the object's DEK. One per failed upload, during exactly the disk-full episodes with no headroom to spare.
Empty objects never had their data file deleted — the versioned delete meant to skip delete markers and skipped legitimately empty objects too, orphaning their files permanently.
A write to a bucket owned by another node could answer 200 having stored nothing — a failed forward fell through to local handling, answering from a node that does not hold the data and applying mutations in the wrong place. A failed forward is refused with ServiceUnavailable now. (pkg/s3compat/handler.go)
Total HTTP deadlines on the paths carrying object data were size limits in disguise — the cluster proxy (60 s), console proxy (30 s) and replication client (120 s) each expressed a maximum object size as a maximum duration, and replication failed silently and permanently since every retry died at the same point. Bounded by progress now; two of the three proxy entry points had no bound at all. (internal/transfer, internal/cluster/proxy.go)
Console requests proxied to the owning node buffered the whole body in memory, so proxying a large object cost as much memory as the object; a failed forward then stored an empty object in its place. Streamed now, answering 502 on failure. (internal/server/console_api.go)
The bucket listings did not consult the policies, so a bucket granted purely through IAM was invisible while every object in it was accessible, and a revoked grant left its bucket on screen. Both listings ask the same policy set as every other request; s3:ListAllMyBuckets is enforced for the first time. (internal/server/console_api.go, pkg/s3compat/handler.go)
A caller holding only an IAM policy could not read an object — a second gate demanded an explicit ACL after the policy check had already approved the request, so s3:GetObject returned 403 for anyone but the owner, and CopyObject failed on its source.
A tenant user could no longer reach a bucket outside every tenant — a bucket in the global namespace does not belong to another tenant. It is decided by the policies now; reaching into a tenant from outside stays closed.
The console refused a global administrator every write, everywhere — the read-only rule was applied to all global administrators rather than only when crossing into a tenant, so creating a bucket in the global namespace answered 403. (internal/server/capability_guard.go)
Editing a role had no effect on any user with a tenant — the bounded document substituted for tenant administrators discarded every other document the role carried, so attaching or revoking a policy on a role was silently a no-op for them. It also omitted both Object Lock configuration actions.
Identities without a stored row lost every permission attached to their tenant — role sessions and federated logins carry a tenant without a users row, and one of the two permission views re-read the tenant from that table. (internal/auth/policy_translate.go)
Saving a user's permissions screen made their role's permissions permanent — the screen shows effective permissions and saved what it showed, turning everything inherited into a direct grant that outlived the role. Only what is not already inherited is stored now.
Transferring a bucket's ownership moved no permission — the owner fields were written and the owner policy was not, so the new owner got nothing and the previous one kept everything.
Temporary credentials replicated between nodes lost their role identity — role_arn and policy_mode were absent from the sync payload, so a credential narrowed to a role came back wider on every other node. (internal/cluster/sts_session_sync.go)
Presigned URLs signed with temporary credentials ran with no user in the request context, so a valid presigned URL made with an ASIA key was denied downstream. (pkg/s3compat/presigned.go)
A rejected credential answered 401 on the S3 API where AWS answers 403 — the authentication middleware's error writer had drifted from the handler's. It now also emits the XML declaration, HostId and AWSAccessKeyId, and sends no body on HEAD.
STS errors carried an empty <RequestId> while the response header had the real one.
Three databases were opened with connection parameters that did nothing — the audit log, share store and metrics history used the wrong DSN syntax for modernc.org/sqlite, which applies none of it without complaint, so they ran in rollback-journal mode with a zero busy timeout and dropped audit entries under load. All three set WAL and a ten-second wait now.
A download token was accepted on four sibling routes when the object key was literally acl, tags, versions or legal-hold. Both download routes are matched by route identity now.
Downloading a specific object version from the console served the current object — in the versions dialog and, identically, on the bucket page.
FilesystemBackend.List returned work in progress as objects and discarded the walk's errors, so an unreadable subtree produced a short listing in silence. Its only consumer deletes what it is handed.
Two tenants could exceed a bucket quota at once — the count was read, compared and then incremented. It is enforced in the increment now, and UpdateTenant no longer writes usage counters back from a stale snapshot.
coordinatorExemptPath matched substrings rather than path segments, so a bucket or user named download skipped the coordinator gate.
A data race on the cluster TLS configuration and its HTTP client — both were replaced as plain pointers from the initialize and join handlers while every server was serving, so a request in flight during a join could be sent with a half-published config. Both are atomics now.
Ten Start() methods re-created the cluster proxy client without synchronisation while other goroutines were reading it; the reassignment bought nothing and is gone. LeaderManager.Stop() panicked if called twice.
The cluster health prober built and leaked an http.Transport per probe, GetAllLatencyStats released its lock mid-iteration, the bucket-location cache cleanup had no stop path, the integrity scan history was a racy read-modify-write, and transfer.Transport left GetBody unwrapped so a replayed request stopped stamping the progress clock.
A refused vote was invisible in the logs when a cluster could not elect anybody, and a failed election was logged at debug level. A peer that stays down is now reported once, not every five seconds — a node left down over a weekend wrote a hundred thousand identical lines.
CI ran go test with no -timeout, so Go's ten-minute default applied to packages taking about three minutes locally on a runner several times slower. The likeliest reason CI failed while every command passed locally.
The Vite and Vitest configs used __dirname, which breaks under Vite's forthcoming native config loader. (web/frontend/vite.config.ts, web/frontend/vitest.config.ts)
Removed the replication of user_capability_overrides, which wrote rows an operator would reasonably believe were in force, and APIClient.fetchObjectBlob, which had no callers and dropped versionId while its comment advertised reading a specific version.

Changed¶

The tenant boundary is enforced by the evaluation engine, not by its callers — a tenant is an account, and an S3 ARN cannot name one, so arn:aws:s3:::backups names a bucket of that name in every tenant at once. PolicySet.Allows now takes the resource's owning tenant and the old form no longer compiles, so no path can forget it. The owner is a type whose zero value means "nobody resolved this" and is refused, because as a string an unset field and the global namespace were the same empty value. (internal/auth/policy_set.go)
Both APIs authorize per S3 action rather than per capability — bucket:configure used to cover versioning, lifecycle, CORS, tagging and ACLs together, and several paths accepted sharing a tenant with the bucket as sufficient. (pkg/s3compat/bucket_ops.go, internal/server/capability_guard.go)
S3 authentication failures are no longer all 401 InvalidAccessKeyId — an expired session returns 403 ExpiredToken (SDKs refresh and retry), a session-policy denial returns 403 AccessDenied (do not retry).
SOSAPI advertises IAMSTS=true with both endpoints, served on POST / of the S3 endpoint so a single URL covers both. Advertised only while security.iam_api_enabled is on and a public API URL is configured. (pkg/s3compat/sosapi.go)
Presigned URLs are covered by IAM policies — they bypass the S3 auth middleware, so without this a presigned URL would have been a way around an attached policy. (pkg/s3compat/presigned.go)
S3 Select streams the object instead of holding it — it parsed the whole object into memory and then loaded that into an in-memory database, so one authenticated query could exhaust the node. The loaders now insert rows as they read — the JSON schema grows mid-stream rather than being discovered by buffering — and the query engine is backed by a temporary file. Measured on a 9 MB input: 11 MB at peak, against 217 MB before. (pkg/s3compat/select.go)
Dependencies updated — Pebble v2 2.1.7, logrus 1.10.2, testify 1.12.1, modernc sqlite 1.57.0, the AWS SDK, jsdom 30 and @testing-library/jest-dom 7. Built with Go 1.26.6, which closes six standard-library advisories govulncheck reported as reachable, including the encoding/xml recursion guard and the quadratic net/url path resolution.
A test now fails if a Server field that can be stopped is not released at shutdown, through the registry or by name. (internal/server/lifecycle_coverage_test.go)
The users table is more compact — the status column folded into a colour-coded toggle that now shows the current state rather than the action, 2FA reduced to an icon, and uniform icon-only actions carrying title and aria-label. (web/frontend/src/pages/users/index.tsx)
docs/CLUSTER.md describes what happens when a node fails — the re-election among survivors, the rejoin and sync on return, and the partition trade. Documented alongside a constraint worth knowing first: every node must use the same cluster_listen port.

[1.5.2] - 2026-07-18¶

Note: v1.5.1 was withdrawn shortly after publication and is not available. Its unclean-shutdown reconciliation could remove latest-object index entries for versioned objects after a crash (object data and per-version metadata, including Object Lock retention, were unaffected). If you installed it, upgrade and run maxiofs repair-pointers --dry-run with the server stopped.

Fixed¶

Listing pagination silently lost one object per page boundary — NextMarker was set to the first key of the NEXT page while marker consumption follows S3 semantics (resume strictly AFTER the marker), so any client paging with the returned marker skipped exactly one object per 1,000 (console listings, S3 ListObjects, SOSAPI, search). At backup-target scale this made verification/repair tools (Veeam, Duplicati) see existing files as missing — ~500 invisible objects on a 500K-file listing. Fixed in all three paginated paths (ListObjects, ListObjectsDelimited — including resuming after a common-prefix marker by skipping the whole prefix group — and searchObjectsByScan); found by an end-to-end torture test (1,090 files, 5 hard kills, worker conversions, KEK rotation, churn — zero losses after the fix), covered by pagination round-trip regression tests at page sizes 1-100. (internal/metadata/pebble_objects.go, internal/metadata/pebble_pagination_test.go)

Changed¶

Dead-code sweep across backend and frontend — removed code with zero production callers, verified independently with deadcode (whole-program reachability), staticcheck (U1000) and per-symbol git grep, plus manual confirmation of the live equivalent for every removal. Highlights:
Parallel implementations that were never wired (each has a live counterpart that stays): internal/object/lock.go + retention.go (live object lock/retention = the Manager interface methods in manager.go), internal/middleware/ratelimit.go (live = authManager.CheckRateLimit + dynamic settings), internal/cluster/rate_limiter.go (the server constructed it but never called it — including a startup log line that falsely claimed cluster APIs were rate-limited; cluster port protection is TLS + HMAC), internal/cluster/metrics.go, middleware.SignRequest (live signer = cluster/proxy.go), the CSR pair GenerateKeyAndCSR/SignCSR (live cert issuance = GenerateNodeCert), the s3compat cluster-proxy auth shim, the unregistered prometheusCollector (live /metrics = the manager's own registry), and pkg/encryption's pre-envelope EncryptionService/KeyManager.
Dead helpers and config variants: unused CORS/logging config constructors, tracing context helpers, S3AuthHelper's V2/V4 parsing chain (GetS3Action/GetResourceARN deliberately kept as IAM primitives), unused bucket validation/permission helpers, assorted single functions (getDiskUsage, queryBucketsFromNode server duplicate, calculateMultipartHash, httpHeadersFromObject, etc.) and the main.go assignment that fed trusted_proxies into the dead middleware (the live consumer getClientIP reads the config directly).
Deliberately kept: Stop() lifecycle methods on sync managers, test seams (MigrateTo, NewManagerWithS3Factory, detectMode, s3compat presigned URL generation — it round-trips the presigned validation tests), and bandwidth.Manager.Remove (documented optional housekeeping).
Frontend (knip): removed 8 unused files (PageHeader, PermissionsEditor, useLockedUsers, 3 stale test mocks, MetricBarChart, MetricPieChart) and 2 unused dependencies (i18next-browser-languagedetector, @tailwindcss/vite).

Added¶

Pebble durability on hard kill (bounded loss + self-healing boot) — hot-path metadata commits were NoSync, so a hard kill (power loss, OOM, kill -9) could lose the last seconds of Pebble writes: a just-PUT object kept its data file + sidecar but vanished from listings, and a lost delete tombstone resurrected a ghost entry. Three-layer fix, none of which meaningfully costs throughput:
Periodic WAL fsync (everysec): a background loop in PebbleStore fsyncs the WAL once per second whenever there are unsynced writes (one fsync/second flat, regardless of write volume — Pebble's group commit makes everything before it durable), bounding hard-kill metadata loss to ≤1s. WALBytesPerSync smooths background flushing so the periodic fsync stays cheap. Interval configurable via PebbleOptions.WALSyncInterval.
Sync on destructive, low-frequency operations: object/version deletes, bucket create/update/delete, multipart complete/abort and raw-KV deletes now commit with pebble.Sync — a lost delete tombstone would leave a ghost listing entry pointing at a removed file (unservable), and a completed multipart upload is the durability point the client paid for. Hot paths (object puts, parts, metrics) stay NoSync under the 1s loop.
Unclean-shutdown reconciliation: Close() writes a CLEAN_SHUTDOWN sentinel that the next open consumes; when a pre-existing store opens without it, the server reconciles the metadata store against the on-disk object tree in the background while serving traffic (reads already work via the sidecar fallback). Repairs are non-destructive: entries lost in the crash window are rebuilt from their sidecars (original timestamps preserved), but metadata entries and sidecars are never removed merely because an object path is absent. Bucket stats are recalculated after changes. Rebuilding metadata after a genuine storage incident is an explicit, operator-initiated offline operation, never an automatic one. (internal/metadata/pebble_store.go, internal/recovery/reconcile.go, internal/server/unclean_shutdown_reconcile.go)
maxiofs repair-pointers offline recovery tool — rebuilds missing latest-object index entries from the surviving per-version metadata (version: keyspace), for use when a metadata index is damaged and objects stop appearing in listings while their files remain on disk. Strictly additive: it only writes entries that are currently missing, never deletes a key, never modifies a version entry and never touches object files; Object Lock retention and legal hold are preserved because they are copied from the version metadata, and objects whose latest version is a delete marker are correctly left deleted. Supports --dry-run. Run with the server stopped. (internal/recovery/repair_pointers.go, cmd/maxiofs/repair_pointers.go)

[1.5.0] - 2026-07-10¶

Added¶

Envelope encryption with the KEK in the database (always-on encryption) — server-side encryption no longer depends on a fixed key in config.yaml and is now always active, matching AWS S3's SSE-S3-by-default model. Highlights:
KEK in the DB: new encryption_keys SQLite table (migration 16) and internal/kek package. On startup the KEK is resolved in order: existing DB value → seeded from config.yaml storage.encryption_key (existing deployments keep decrypting everything; keep the config file as a key backup) → freshly generated 32-byte key persisted in the DB (deployments started with only --data-dir self-provision, no config needed). The DB is the source of truth; the config value is only ever a seed.
Envelope (per-object DEK): every new object is encrypted with its own random Data Encryption Key (AES-256-GCM streaming); the DEK is wrapped with the current KEK and stored in the on-disk .metadata sidecar (wrapped-dek, wrapped-dek-iv, kek-version). Because the crypto material lives in the sidecar — not only in the metadata DB — objects remain recoverable from the filesystem alone given a KEK backup, and future KEK rotation only re-wraps the small DEK instead of re-encrypting object data.
Multi-format reader (full backward compatibility): reads branch per object on its stored format — plaintext objects (from key-less deployments) are served as-is, legacy directly-encrypted objects decrypt with KEK version 1 (the former config key), envelope objects unwrap their DEK. Verified by tests covering all three formats plus restart-persistence.
Writes always encrypt: the shouldEncryptObject/enable_encryption conditional and the plaintext write path are gone (single and multipart uploads); storage.enable_encryption is deprecated and ignored. Folder markers (zero-byte key/ entries) are exempt — this also fixes a goroutine leak where the encryption pipe for a folder marker was never read.
Sidecar protection: UpdateObjectMetadata previously replaced the whole .metadata sidecar with just the caller's keys, silently destroying the encryption entries of an encrypted object (making it undecryptable). It now merges into the existing sidecar and system keys (encrypted, wrapped-dek, original-size, ...) can never be overridden.
The misleading per-bucket "disable encryption" toggle path in bucket creation now consistently records AES256 on every bucket; the console encryption endpoints report encryption as always enabled. (internal/kek/, internal/db/migrations/versions.go, internal/object/manager.go, internal/server/server.go, internal/server/console_api.go)
Background encryption worker — objects stored before encryption became mandatory (deployments that ran without an encryption key) are now converted to envelope encryption automatically in the background. One shared worker walks every bucket/object and acts on each object's state: plaintext → convert (fresh DEK, wrapped with the current KEK); already encrypted → skip. This is the same component that will re-wrap DEKs on future KEK rotations. Safety first, since it rewrites production data: the plaintext is staged to a temp file, the original is only replaced by an atomic rename, the per-key lock serialises against concurrent writers, and every converted object is read back, decrypted and MD5-compared before counting as done (with automatic plaintext restore and concurrent-overwrite detection on mismatch). All stored versions of a key are converted, folder/delete markers and multipart-format ETags are handled. Scheduling is load-aware (backs off while CPU >60% or RAM >85%) with gentle pacing, and progress is checkpointed in Pebble so a restart resumes where it left off; a full pass re-runs daily to catch strays. New Settings → Security block shows live status (running/paused-by-load/done, converted/skipped/failed counters, per-bucket progress bar) with a "Run now" button; endpoints GET /api/v1/settings/encryption/worker-status and POST /api/v1/settings/encryption/worker-run (global admin only). Translated in all 9 UI languages. Verified end-to-end with the real binary: a plaintext object planted on disk was picked up by a triggered pass, converted (sidecar gained wrapped-dek/kek-version, disk bytes became ciphertext) and downloaded back byte-for-byte intact. (internal/object/encryption_migration.go, internal/server/encryption_worker.go, web/frontend/src/pages/settings/EncryptionWorkerStatus.tsx)
maxiofs recover — offline disaster recovery (metadata DB lost, files intact) — new CLI subcommand that rebuilds the Pebble metadata store from the on-disk object tree, run with the server stopped. Buckets are rediscovered from their .maxiofs-bucket markers (owning tenant included), objects and versions from each file's .metadata sidecar (original size/ETag, content-type, encryption fields; versions in bucket/.versions/key/versionID are rebuilt in order with the newest as latest, and bucket metrics/versioning status are recomputed). With --recovery-bundle the encryption keys are restored into the auth SQLite (so the recovered server's bootstrap loads them instead of generating a fresh KEK) and every envelope object's wrapped DEK is verified to unwrap before being counted as recovered. Safety: always writes into a FRESH directory laid out exactly like the live store (activate with a single mv), refuses a non-empty output, --dry-run reports without writing, and the report classifies objects (verified/unverified encrypted, legacy, plaintext) with per-file failures. Verified end-to-end with the real binary: seeded objects → deleted the entire Pebble store AND the SQLite database → recover rebuilt 1 bucket / 2 objects with 2 DEKs verified and 1 KEK restored → the server started with "KEK loaded from database" and served both objects decrypted byte-for-byte. (internal/recovery/, cmd/maxiofs/recover.go)
Encryption key (KEK) rotation — global admins can rotate the encryption key from Settings → Security (or POST /api/v1/settings/encryption/rotate-kek). Rotation creates a new key version and makes it current; object data is never re-encrypted — the background worker re-wraps each object's small wrapped-DEK to the new version (verified per object: sidecar update is atomic and the unwrapped DEK is compared byte-for-byte), and legacy directly-encrypted objects (pre-envelope) are now fully converted to a current-KEK envelope during the same pass (decrypted with their original key, staged, re-encrypted and read-back-verified, with the original ciphertext as the restore source). Old key versions are deliberately kept forever: they cost nothing, the recovery bundle includes them, and deleting one would orphan any file still referencing it. Safety tie-ins: rotating resets the recovery-bundle-downloaded flag so the console banner reappears prompting a fresh bundle download (the old bundle lacks the new key), and the worker pass starts immediately. In a cluster the new key is created cluster-shared, pushed to all healthy peers at rotation time and reconciled every 60s by the global-config sync (nodes offline during the rotation catch up automatically; raw replication degrades per-node to the legacy path until they do). UI: current version + "Rotate key" button with confirmation in Settings → Security, translated in all 9 UI languages. Verified end-to-end with the real binary: upload (v1) → download bundle → rotate → status flips to v2/not-downloaded → worker re-wrapped the sidecar to v2 with byte-identical object data → download returns the content intact. (internal/kek/store.go, internal/object/encryption_migration.go, internal/server/encryption_recovery_handlers.go, internal/cluster/global_config_sync.go, web/frontend/src/pages/settings/EncryptionRecovery.tsx)
Ciphertext HA replication (cluster-shared KEK) — HA write fanout no longer decrypts on the primary and re-encrypts on each replica. On the first cluster join, the initiating node creates a cluster-shared KEK (next free version, marked current) and ships every cluster-shared key inside the existing join package (same channel as the JWT secret); each joining node adopts them — keeping its own local v1 for its pre-join objects — and a version conflict (same version, different key material) aborts the join with a clear error. From then on every node wraps new objects with the same KEK, so the fanout sends the stored ciphertext + sidecar + Pebble metadata as-is (X-HA-Raw headers); the replica stores a byte-identical copy (with full replica bookkeeping: Pebble entry, metrics, implicit folders) and decrypts only on read. Objects not yet eligible (plaintext, legacy direct-encrypted, or wrapped with a node-local KEK) transparently use the previous decrypt/re-encrypt path and converge to raw-replicable as the rotation worker re-wraps them; a replica that cannot decrypt a KEK version declines with 412 and the primary falls back per node. New cluster_shared column on encryption_keys (migration 17). Verified end-to-end with a real 2-node cluster: join created/adopted KEK v2 on both nodes, an uploaded object landed on the replica with byte-identical ciphertext and wrapped DEK, and the replica decrypted it locally with its own key store. (internal/kek/, internal/cluster/ha_object_manager.go, internal/cluster/manager.go, internal/server/cluster_handlers.go, internal/server/cluster_object_handlers.go, internal/object/raw_replication.go)
Encryption recovery bundle (Settings → Security) — the KEK now lives in the database, so a lost database without a key backup would mean losing every encrypted object. Global admins can download a recovery bundle: an export of all KEK versions encrypted under an admin passphrase (PBKDF2-SHA256, 310k iterations + AES-256-GCM; the passphrase is never stored). The console shows an informative banner to global admins until the bundle has been downloaded (tracked in system_settings), linking to the new section in Settings → Security with passphrase + confirmation inputs. kek.DecryptBundle() provides the opening side for the future offline maxiofs recover tooling. New endpoints GET /api/v1/settings/encryption/recovery-status and POST /api/v1/settings/encryption/recovery-bundle (global admin only). Translated in all 9 UI languages. Verified end-to-end: fresh server → status not-downloaded → short passphrase rejected (400) → bundle downloads with correct headers → status flips → bundle decrypts and its KEK matches the database's byte-for-byte. (internal/kek/bundle.go, internal/server/encryption_recovery_handlers.go, web/frontend/src/pages/settings/EncryptionRecovery.tsx, web/frontend/src/components/layout/RecoveryBundleBanner.tsx)
Per-tenant bandwidth throttling — global admins can cap a tenant's aggregate transfer bandwidth (upload + download combined) so one client can't saturate the link for others. It throttles (slows), never rejects — legitimate bursts are smoothed, not failed. All of a tenant's concurrent transfers share one token-bucket budget (internal/bandwidth), enforced on S3 object up/downloads and multipart parts (pkg/s3compat). Buckets inherit their tenant's limit (global buckets are exempt); changes apply hot without restart. New tenant field max_bandwidth_bytes_per_sec (0 = unlimited) with SQLite migration, tenant API, and a "Max Bandwidth (MB/s)" field in the tenant form (translated in all 9 UI languages). Verified end-to-end: a tenant capped at 2 MB/s transferred 20 MB in ~10 s (vs ~1 s unlimited), the cap updated live, and two concurrent transfers shared the budget. In a cluster the cap is set once and synced to every node via tenant-sync, then enforced per-node. (internal/bandwidth/, internal/auth/, pkg/s3compat/, internal/server/console_api.go, internal/cluster/tenant_sync.go, tenant UI)
Per-bucket storage quota — buckets can now have their own size and object-count limits, independent of (and in addition to) tenant quotas. This closes the gap where only tenants had quotas, so a global bucket (no tenant) — e.g. a dedicated Veeam/backup target — could not be capped. Highlights:
Model: optional Quota { maxSizeBytes, maxObjectCount } on bucket metadata (0 = unlimited for that dimension), persisted in Pebble. Current usage reuses the already-cached bucket TotalSize/ObjectCount, so no new counter is introduced.
Enforcement: uploads that would exceed the cap are rejected with S3 QuotaExceeded (both PutObject and CompleteMultipartUpload, the path Veeam uses) and 403 on the console upload endpoint. Correct handling of overwrites (net delta, not double-counted), versioned buckets (each version counts), the object-count cap (only on new keys), and HA replica writes (counter updated, enforcement bypassed on replicas). For tenant buckets, both the bucket quota and the tenant quota apply — the most restrictive wins.
Veeam/SOSAPI exposure: the SOSAPI capacity.xml a client reads per bucket now advertises capacity with precedence per-bucket quota → tenant quota → physical disk, so Veeam (and any SOSAPI client) sees the real bucket limit and its live usage (Capacity/Used/Available), not just the disk or tenant figure.
API + UI: GET/PUT/DELETE /api/v1/buckets/{bucket}/quota (returns quota + live usage; routed to the bucket's owner node in a cluster). New Quota tab in bucket settings with a usage bar, translated in all 9 UI languages.
Validation: a tenant bucket's size quota may not exceed the space assigned to its tenant (maxSizeBytes ≤ tenant.MaxStorageBytes), rejected with 400.
Alerts: SSE + email notifications (bucket_quota_warning / bucket_quota_critical, with a matching bucket_quota_resolved) fire as a bucket approaches its quota, using the existing warning/critical disk thresholds and escalation-only de-duplication — mirroring the tenant quota alert system.
Cluster: quota is enforced/aggregated correctly per bucket without cross-node summing (buckets are placed on an owner node and HA replicas are full mirrors, so the owner's local total is authoritative — summing would double-count).

Changed¶

Removed the vestigial RBAC stub and its SEC-03 startup warning — the CheckPermission/CheckBucketPermission/CheckObjectPermission chain was unreachable dead code from a superseded design: its only entry point (AuthorizeRequest) had no callers, while fine-grained authorization has been enforced since v1.4.0 by the live stack (roles + tenant scoping, the 11-capability role system on both S3 and console, bucket policies with Condition evaluation, and ACLs). The startup warning misled operators into believing permissions were inactive. GetS3Action/GetResourceARN are kept as primitives for the future IAM policy engine. (internal/auth/manager.go, internal/auth/s3auth.go, SECURITY_AUDIT.md)
Removed the broken make docker-cluster targets — docker-cluster and docker-cluster-monitoring referenced a cluster compose profile that never existed, so they silently started a single node while the Makefile and docs promised a 3-node cluster at :8081/:8083/:8085. Beyond being broken, a multi-node cluster inside one Docker host provides no real HA (shared disk/power/kernel) — the targets and their documentation are gone, and CLUSTER.md now carries explicit guidance for containerized deployments (one node per physical host, local volume pinning, port 8082 between hosts). (Makefile, docker/README.md, docs/DEPLOYMENT.md, docs/CLUSTER.md)
Bucket creation no longer shows an encryption opt-out — the "enable encryption" checkbox in the create-bucket dialog was misleading: unchecking it never stored anything in plaintext (server-side encryption is always on). The Encryption tab is now informational only, and the creation request always records AES256 on the bucket. (web/frontend/src/pages/buckets/create.tsx)

Fixed¶

Console: the notification stream now reconnects automatically — the SSE connection that delivers disk/quota/bucket alerts to the console had no reconnection: a dropped stream (server restart, reverse-proxy idle timeout, network blip) silently ended alert delivery for the rest of the session until a full page reload. The hook now reconnects with exponential backoff (1s → 30s max, reset after a successful connection). Also hardened the four dangerouslySetInnerHTML translation renders to route through the shared HTML sanitizer (i18n runs with escapeValue: false, so any future user-influenced interpolation in those keys would have reached the DOM unescaped), and throttled the idle-timer's localStorage activity writes to once per second (previously a synchronous write per mousemove). (web/frontend/src/hooks/useNotifications.ts, web/frontend/src/hooks/useIdleTimer.ts, web/frontend/src/lib/modals.tsx, web/frontend/src/components/BucketIntegrityModal.tsx, web/frontend/src/components/ObjectLockConfigModal.tsx)
Windows: a transient sharing violation no longer orphans a deleted object — on Windows, an external scanner (antivirus, search indexer) briefly holding a just-written file without FILE_SHARE_DELETE made the physical delete fail with "used by another process"; since the metadata had already been removed, the file stayed behind as an orphan and, worse, HEAD/GET found it again through the sidecar fallback — the object appeared to survive its own deletion. The filesystem backend now retries the file removal briefly (5 attempts with backoff) before giving up. Reproduced consistently by the TestDeleteObjectErrorCases flake, now 20/20 green. (internal/storage/filesystem.go)
Legacy HA transfers now preserve client checksums — objects replicated through the decrypt/re-encrypt path (plaintext/legacy objects, or a replica missing the cluster KEK) lost their x-amz-checksum-* value: the replica's metadata had no ChecksumAlgorithm/ChecksumValue, so GetObjectAttributes answered differently depending on which node served it. All legacy senders (write fanout, sync worker, anti-entropy push) and the internal HA GET now forward the checksum headers; the receiver recomputes and validates them, which also catches a corrupted transfer in flight. (internal/cluster/ha_object_manager.go, internal/cluster/ha_sync_worker.go, internal/cluster/anti_entropy.go, internal/server/cluster_object_handlers.go)
Concurrent KEK rotations no longer collide — Rotate, EnsureClusterKey and AdoptClusterKeys computed the next key version without serialising against each other, so two concurrent calls (e.g. rotation racing the periodic cluster sync) could pick the same version and fail on the UNIQUE constraint with a cryptic error. Multi-step KEK mutations are now serialised by a store-level write lock, and AdoptClusterKeys applies the whole received key set in a single SQL transaction (a mid-way failure can no longer leave a partial adoption). (internal/kek/store.go)
KEK sync endpoint returns 400 for malformed payloads (409 only for real conflicts) — every adoption failure was answered with 409 Conflict, including malformed key material that is a plain bad request. A new kek.ErrKeyConflict sentinel marks the genuine same-version-different-material case; the handler maps it to 409 and everything else to 400. (internal/kek/store.go, internal/server/encryption_recovery_handlers.go)
KEK distribution after a rotation survives a dropped admin connection — the post-rotation push to cluster peers ran under the HTTP request's context, so an admin closing the browser mid-request cancelled the distribution (peers kept wrapping with the old version until the next 60s sync). The push now runs under the server context. Also GlobalConfigSyncManager.Stop() is idempotent (a second call panicked on double channel close). (internal/server/encryption_recovery_handlers.go, internal/cluster/global_config_sync.go)
A write acknowledged as successful could be invisible in listings — when the Pebble metadata save failed after the object file was already stored (disk full, I/O error), PutObject only logged a warning and returned success: the S3 client got a 200 for an object that no listing would ever show (backup software verifies uploads by listing — a silent gap). The write now fails with an error so the client retries; the stored data file is deliberately kept (on a non-versioned overwrite it is the only copy, and sidecar-only files are exactly what maxiofs recover reindexes). The raw HA replica write applies the same contract, so the primary counts the node as unconfirmed and the sync worker retries instead of leaving an invisible replica. (internal/object/manager.go, internal/object/raw_replication.go)
Encryption worker: a stale checkpoint skipped the entire pass — if the worker was interrupted mid-pass and its checkpointed bucket was deleted before the next run, the resume logic waited forever for a bucket that no longer existed and silently skipped every remaining bucket, ending the pass as "done" with nothing converted. A checkpoint pointing at a nonexistent bucket now falls back to a full scan. (internal/server/encryption_worker.go)
maxiofs recover: objects with an unverifiable DEK no longer disappear from the rebuilt index — an envelope object whose wrapped DEK failed to unwrap with the bundle keys (e.g. corrupt sidecar entry) was reported as a failure but EXCLUDED from the rebuilt Pebble store, making bytes that still exist on disk invisible in every listing — unrecoverable in practice. Such objects are now indexed (counted as encrypted-unverified) with the failure kept in the report for the operator. (internal/recovery/recovery.go)
maxiofs recover: KEK restore could leave two keys marked current — restoring a bundle into a database that already held encryption keys with its own current marker inserted the bundle's current flag alongside the existing one, leaving two is_current=1 rows and making the server's bootstrap pick one non-deterministically. Bundle keys are now always inserted non-current and the marker is applied once, transactionally: the database's existing current key wins (it is more recent than any external bundle); the bundle's current is applied only when the database had none. (internal/recovery/recovery.go)
A quota-rejected overwrite destroyed the existing object — on a non-versioned bucket, PutObject stored the new bytes over the final object path BEFORE validating the tenant/bucket quota; when the quota rejected the write, the cleanup deleted that path — which by then held the only copy, so the client got a quota error AND lost the object it was updating (its stale Pebble entry pointed at nothing). The quota check now runs before the store step (the original size is already known from the streaming stage), so a rejection leaves the existing object completely untouched. Regression tests cover both the bucket-quota and tenant-quota paths. (internal/object/manager.go)
Crash window in object overwrites could leave an undecryptable object — the filesystem backend committed an overwrite as two consecutive renames (sidecar first, then data). A crash or failure between them left the NEW sidecar (new DEK) over the OLD bytes — undecryptable under envelope encryption — and a failed data rename deleted the previous object's sidecar entirely. Writes now use a staged-sidecar two-phase commit: the new sidecar is staged at <object>.metadata-staging, the data is renamed into place (data commit), then the stage is renamed over the final sidecar (metadata commit). A crash at any point is resolved deterministically on the next access: if the stored bytes match the staged etag the metadata commit is completed (roll forward), otherwise the stage is discarded and the old pair remains fully intact (roll back). Per-path striped locks keep concurrent readers from resolving an in-flight stage; staged files are excluded from listings and from maxiofs recover; object keys ending in .metadata/.metadata-staging (which always collided with sidecar files) are now rejected. (internal/storage/filesystem.go, internal/object/manager.go, internal/recovery/recovery.go)
Replicas and recovered stores no longer falsify LastModified (perpetual anti-entropy churn on multipart objects) — the Pebble store stamped LastModified = now on every metadata write, so an HA replica recorded its receive time instead of the primary's modification time, and a maxiofs recover rebuild stamped recovery time instead of the original sidecar timestamp. Consequences: the anti-entropy scrubber's multipart comparison (size + mtime ±1s; multipart ETags can't be checksum-compared) flagged every replicated multipart object as divergent on every cycle and ping-ponged full re-transfers forever, LWW could prefer a stale copy just because it was written locally later, and lifecycle timers reset after recovery. Pebble now preserves a caller-provided LastModified (stamping only when unset), raw HA transfers already carry the primary's metadata, legacy HA transfers propagate it via a new X-HA-Last-Modified header (set by write fanout, sync worker and anti-entropy push; returned by the HA GET endpoint and honored by both pull paths), and recovery keeps the sidecar's original timestamp. (internal/metadata/pebble_objects.go, internal/object/manager.go, internal/cluster/ha_object_manager.go, internal/cluster/ha_sync_worker.go, internal/cluster/anti_entropy.go, internal/cluster/stale_reconciler.go, internal/server/cluster_object_handlers.go)
HA object replication requests were 404ing against the dedicated cluster port — every HA sender (write fanout, HA sync worker, anti-entropy repair, stale-node reconciler) targeted /api/internal/ha/..., but since the dedicated inter-node port was introduced those routes have been registered under /api/internal/cluster/ha/.... Result: synchronous write fanout, catch-up sync and anti-entropy repair silently failed with 404 on every call (with factor=2 zero replica confirmations are required, so writes still "succeeded" without ever replicating — matching previously observed ghost sync errors). All 13 sender URLs now use the registered prefix; verified with a real 2-node cluster where fanout previously 404'd and now lands replicas. (internal/cluster/ha_object_manager.go, internal/cluster/ha_sync_worker.go, internal/cluster/anti_entropy.go, internal/cluster/stale_reconciler.go)
S3 API → console redirect now honors public_console_url (surfaced via GitHub issue #6) — when a browser hits the S3 API, it is redirected to the web console; that redirect hardcoded the internal console port (8081) and only used public_console_url when X-Forwarded-* proxy headers were present. With direct or port-mapped access (Docker 39000:8081, NAT) there are no such headers, so the redirect pointed at the inaccessible internal port. The redirect now uses public_console_url whenever it is configured, independent of proxy-header detection; the request-derived fallback applies only when it is unset. (internal/api/handler.go)
Identical public_api_url and public_console_url are now rejected at startup (surfaced via GitHub issue #6) — the S3 API and the web console are distinct services; when both public URLs are set to the exact same value, generated links collide (S3 share/presigned URLs, which use public_api_url, would point at the console) and a reverse proxy cannot tell S3 traffic from console traffic. Startup validation now fails with a clear message when the two normalized URLs match, requiring a distinct hostname, port, or path prefix for each. (internal/config/config.go)
TLS enabled via config file is now honored (GitHub issue #6) — setting enable_tls: true with cert_file/key_file in the configuration file had no effect: the server still started on plain HTTP and logged "TLS disabled - servers will use HTTP". The startup code treated the --tls-cert/--tls-key CLI flags as the sole switch for TLS and, when they were absent (the common Docker/config-file setup), forced EnableTLS = false, overriding the value already loaded from the config file. TLS resolution now respects the merged configuration: viper already merges cert_file/key_file with the correct precedence (CLI flag > env > config file), the CLI flags additionally act as an explicit switch that enables TLS, and in their absence the config-file enable_tls value is preserved instead of being clobbered. Config-file-driven HTTPS now works. (cmd/maxiofs/main.go)
Object share URLs are now percent-encoded — sharing an object whose key contained spaces, #, [ ], or non-ASCII/UTF-8 characters (accents, emoji) produced a broken share URL: the key was interpolated raw, so the # truncated the key at the URL fragment and spaces/emoji made the link invalid, and the shared object failed to resolve. The share URL is now built with an AWS S3-compatible path encoder (s3EncodePath) that preserves / and RFC 3986 unreserved characters while percent-encoding every other byte (space→%20, #→%23, ❌→%E2%9D%8C, 💸→%F0%9F%92%B8, etc.), so the generated link round-trips back to the exact key and resolves the real object. Applied to all four URL constructions in handleShareObject (both the existing-share and new-share branches, with and without a configured PublicAPIURL). Verified end-to-end against the S3 API: uploading an object with such a key and opening its share URL now returns 200 with the correct content. (internal/server/console_api.go)

[1.4.2] - 2026-06-30¶

Changed¶

Dependencies updated (Go + frontend) — Go modules bumped to their current minor/patch releases: aws-sdk-go-v2 v1.41.12→v1.42.0, aws-sdk-go-v2/service/s3 v1.103.2→v1.104.1, aws-sdk-go-v2/credentials v1.19.22→v1.19.25, modernc.org/sqlite v1.52.0→v1.53.0, plus transitive indirect bumps. No major-version changes; the Pebble metadata engine (cockroachdb/pebble/v2 v2.1.6, with pebble v1.1.5 retained only for legacy v1→v2 migration) is untouched. Frontend npm dependencies were refreshed within their existing semver ranges (package-lock.json only; package.json unchanged). go build ./..., go mod verify, the Go core test packages (auth/metadata/object), the frontend production build, and the full vitest suite (106 tests) all pass on the updated set. (go.mod, go.sum, web/frontend/package-lock.json)

Security¶

Timing attack on S3 secret key comparison eliminated — ValidateCredentials compared secret keys with the native != operator, allowing character-by-character timing side-channel attacks. Replaced with crypto/subtle.ConstantTimeCompare. (internal/auth/manager.go)
Account-status enumeration via different error codes fixed — ValidateConsoleCredentials ran bcrypt before checking user.Status, leaking whether a valid disabled account existed. Active status is now verified before the password check; both paths return ErrInvalidCredentials on failure. (internal/auth/manager.go)
RBAC stub now emits a startup warning and logs all denials — CheckPermission was a silent stub that returned ErrUnauthorized without logging. NewManager now logs a prominent warning that RBAC enforcement is in stub mode; every denied call is logged with user, action, and resource fields for audit trail visibility. (internal/auth/manager.go)
S3 access secret keys are now encrypted at rest (AES-256-GCM) — access keys were stored and loaded in plaintext from SQLite. GenerateAccessKey now encrypts secrets before persisting (enc: prefix + base64url(nonce‖ciphertext)); GetAccessKey decrypts on load. Existing plaintext keys are returned as-is (backward compatibility). Encryption key derived with deriveStorageKey() using PBKDF2-SHA256 over the JWT secret, 310 000 iterations, 32-byte output. (internal/auth/manager.go)
Inter-node proxy loop header now stripped at ingress — the X-MaxIOFS-Proxied loop-detection header and all other X-MaxIOFS-* cluster-internal headers can be spoofed by an external client, allowing request smuggling into the cluster's internal routing. StripInternalClusterHeaders is called on every inbound proxied request in ProxyRead and TryProxyRead before the request is forwarded. (internal/cluster/proxy.go, internal/cluster/manager.go)
Key derivation upgraded from single-pass SHA-256 to PBKDF2-SHA256 — DeriveKey in the encryption package used sha256.Sum256(append(password, salt...)) with the hardcoded salt "maxiofs-salt". Replaced with pbkdf2.Key with a minimum of 310 000 iterations (NIST SP 800-132 recommendation for AES-256), enforced even when EncryptionConfig.KeyDerivationRounds is set lower. (pkg/encryption/encryption.go)
Legacy SHA-256 password function marked as deprecated — hashPasswordSHA256 is retained only for reading legacy records; a // SEC-07: Do NOT use for new passwords comment prevents accidental reuse. All new passwords go through bcrypt via SQLiteStore.CreateUser. (internal/auth/manager.go)
Tenant-scoped admins no longer receive cross-tenant SSE notifications — notification filtering treated any user with the admin role as global, even when TenantID was set. Global delivery now requires role == "admin" and an empty tenant; tenant admins receive only notifications matching their tenant. (internal/server/sse_notifications.go)
Console API request bodies are now capped at 1 MiB where uploads are not expected — mutating Console API requests now pass through http.MaxBytesReader by default, independent of Content-Type, while preserving large object uploads through the console. (internal/server/console_api.go)

Fixed¶

PutObject quota TOCTOU eliminated with atomic SQL CAS — two concurrent uploads could both pass CheckTenantStorageQuota before either incremented the counter, allowing tenants to exceed their limit by up to 2× the upload size in the worst case. IncrementTenantStorage now uses a single conditional UPDATE: SET current_storage_bytes = current_storage_bytes + ? WHERE … current_storage_bytes + ? <= max_storage_bytes. If 0 rows are affected the quota is exceeded and ErrStorageQuotaExceeded is returned atomically. (internal/auth/tenant.go)
GET /api/v1/users now returns the real username instead of the internal user ID — the list-users response populated username from User.ID, which broke SSO/imported accounts where IDs are generated (user-...) and usernames are emails or provider names. (internal/server/console_api.go)
Global-admin checks no longer depend on role ordering — settings and integrity endpoints now use the shared admin-role check plus TenantID == ""/TenantID != "", so global admins remain global regardless of role order while tenant admins stay tenant-scoped. (internal/server/console_api.go, internal/server/object_integrity_handler.go)
Console refresh responses and object tags are now unwrapped consistently in the frontend — the API client now accepts both plain and { success, data } responses, preventing silent token-refresh failures and empty tag editors when the backend returns wrapped Console API payloads. (web/frontend/src/lib/api.ts)
Group screens and object helper actions now unwrap Console API responses consistently — group list/detail/member endpoints plus object rename and folder-size responses now accept both plain and { success, data } payloads, preventing empty group pages/dropdowns and incorrect folder-size results when the backend returns wrapped payloads. (web/frontend/src/lib/api.ts)
Bucket permissions are now scoped by the bucket tenant — permissions now store and filter bucket_tenant_id, so buckets with the same name in different tenants no longer share, overwrite, list, or revoke each other's grants. Cluster permission sync and bucket migration payloads include the new scope. (internal/auth/permissions.go, internal/server/console_api.go, internal/db/migrations/migration14_bucket_permission_scope.go, internal/cluster/bucket_permission_sync.go, internal/cluster/migration.go)
Global admins can manage object ACL and legal hold on tenant buckets — object ACL and legal-hold handlers now honor the tenantId query parameter for global admins, matching the existing object tags/restore behavior and avoiding accidental lookups against a same-named global bucket. (internal/server/console_api.go)
Versioning metrics race eliminated with per-key shard mutex — two concurrent writers to the same object key both read the same existingObjBeforeSave snapshot and applied the same size delta to the bucket counter, permanently corrupting bucket metrics. objectManager now has a muShards [256]sync.Mutex array; lockKey(bucket, key) hashes the key to a shard and holds the lock for the entire read-existing / write-metadata / update-metrics sequence, serialising writes to the same key. (internal/object/manager.go)
deleteSpecificVersion no longer uses a stale version snapshot — the function read GetObjectVersions before the delete and then computed the "next latest" from that pre-delete snapshot. A concurrent PutObject between the snapshot and PutObjectVersion could mark an older version as latest, hiding the new write from subsequent GETs. The function now re-queries GetObjectVersions after DeleteObjectVersion to obtain a fresh post-delete list. (internal/object/manager.go)
fanoutPut now pins replication to the exact locally-written version — the HA fanout re-read the object with GetObject(ctx, bucket, key) after the local write, with no version pin. Under concurrent load a newer version could have been written by then and that newer version would be replicated instead. fanoutPut now accepts a versionID string and calls GetObject(ctx, bucket, key, versionID) when non-empty, so only the intended version is replicated. (internal/cluster/ha_object_manager.go)
Login rate limiter TOCTOU fixed with atomic CheckAndRecord — AllowLogin (read lock) and RecordFailedAttempt (write lock) were separate calls: two goroutines could both pass the check before either incremented the counter. Added CheckAndRecord(ip string) bool that atomically checks the limit and increments under a single write lock. authManager.CheckRateLimit now calls CheckAndRecord. (internal/auth/rate_limiter.go, internal/auth/manager.go)
Cleanup goroutines in rate limiters no longer leak on shutdown — LoginRateLimiter.cleanupLoop and InMemoryRateLimitStore.cleanupRoutine had no exit condition, leaking goroutines if the limiter was replaced or the server shut down. Both structs now carry a stopCh chan struct{}; the goroutines select on it alongside the ticker. A new Stop() method closes the channel. Additionally, authManager.SetSettingsManager now calls Stop() on the existing limiter before replacing it with a settings-configured one, so the limiter created in the constructor no longer leaks its cleanup goroutine on every startup. (internal/auth/rate_limiter.go, internal/auth/manager.go, internal/middleware/ratelimit.go)
Proxy no longer buffers entire request body in RAM — ProxyRequest and ProxyToNodeAPIURL called io.ReadAll(originalReq.Body), attempting to allocate the full request size (up to 10 GB for large PUTs) on the heap. Both functions now pass originalReq.Body directly as the outgoing body with defer originalReq.Body.Close(). (internal/cluster/proxy.go)
SearchObjects scan limit reduced from 100 000 to 10 000 when using delimiter — the hardcoded scanLimit = 100000 when delimiter != "" could load 100 k metadata records into RAM on large buckets. The limit is now max(maxKeys*10, 10000), giving 10× headroom over the requested page size while capping the worst-case allocation. (internal/object/manager.go)
UpdateObjectMetadata returned 404 for all objects in versioning-enabled buckets — the function always checked storage.Exists at the plain (non-versioned) path. Versioned objects have no file there. The metadata store is now queried first; when metaObj.VersionID != "" the path is resolved via getVersionedObjectPath, matching the logic in GetObjectMetadata. (internal/object/manager.go)
validateObjectName now rejects path-traversal sequences missed by string checks — strings.Contains(key, "../") can be bypassed with sequences like a/b/../../etc/passwd. Validation now splits the key on / and rejects it when any path segment is exactly .. (and rejects absolute paths beginning with /), catching traversal regardless of where it appears in the key. (internal/object/manager.go)
unsafe-eval removed from console Content-Security-Policy — ConsoleHeaders() included 'unsafe-eval' in script-src, enabling eval(), Function(), and setTimeout(string) — the primary escalation vector for XSS payloads in modern SPAs. Vite/React production builds never emit eval(), so the directive is removed with no functional impact. (internal/middleware/s3headers.go)
Strict-Transport-Security (HSTS) now set on the web console endpoint — S3Headers() already sent the HSTS header but ConsoleHeaders() did not, leaving the web UI vulnerable to HTTPS-stripping on a user's first unprotected request. max-age=31536000; includeSubDomains is now included in both middleware functions. (internal/middleware/s3headers.go)
generateID() in share manager now propagates crypto/rand errors — the function silently discarded the error from rand.Read, so on entropy exhaustion it would return the all-zero hex string "000…000", causing silent ID collisions or UNIQUE-constraint failures in the shares table. The function now returns (string, error) and CreateShare propagates the failure to the caller. (internal/share/manager.go)
Object storage directories created with restrictive permissions (0750) — os.MkdirAll used mode 0755 (world-readable/executable) for the storage root and all object subdirectories. On a shared host any local OS user could read object files directly from disk, bypassing S3 access controls, encryption, and audit logging. Changed to 0750 so only the process owner and its group can access the data. (internal/storage/filesystem.go)
ValidateRefreshToken now rejects inactive/suspended accounts — the access-token validation path (ValidateJWT) was already patched to block deactivated users, but the refresh-token path fetched the user record without checking user.Status. A deactivated account could therefore exchange a still-valid refresh token for a fresh access+refresh pair, bypassing the account-deactivation control. The same user.Status != UserStatusActive guard is now applied in ValidateRefreshToken. (internal/auth/manager.go)
Bucket inventory could not be turned off from the web console once enabled — the Inventory tab nested the Save button (and the entire configuration body) inside the {inventoryForm.enabled && (…)} conditional, so un-checking "enabled" hid the only control that could persist the change and left inventory stuck on. The tab now keeps a single, always-visible Save button: un-checking "enabled" and saving turns inventory off (the backend no longer requires a destination bucket when enabled=false), and the destination is still required only while enabling. Save is enabled only when the form actually differs from the saved configuration, so clicking it with no pending changes can no longer trigger a no-op save error. The Delete button was removed (disabling and saving is how inventory is turned off). Additionally, getBucketInventory now returns null when the backend reports no configuration — the no-config response is { "success": true } with no data field, and the previous response.data?.data ?? response.data (and a naive unwrap) fell back to that truthy wrapper, making the UI behave as if a configuration existed. (web/frontend/src/pages/buckets/[bucket]/settings.tsx, web/frontend/src/lib/api.ts, internal/server/inventory_handlers.go)
Collapsed sidebar hid every item under the Users submenu — when the sidebar was collapsed, a parent navigation item with children rendered only the parent icon (linking to /users), so the submenu entries (Groups, Access Keys, Tenants, Identity Providers, Role Capabilities) became unreachable. Collapsed mode now flattens a parent's children into individual top-level icons at the same level as every other item, each keeping its own tooltip, route, and exact active-state highlight, and still honoring the existing permission filtering. (web/frontend/src/components/layout/SidebarNav.tsx)

[1.4.1] - 2026-05-18¶

Added¶

Event-driven cluster config sync — all mutating handlers now fan out immediately — previously, all sync managers (tenant, user, access key, bucket permission, IDP provider, group mapping) propagated changes to peer nodes only on a periodic timer (default 30 s), so a user created on node A could be rejected by node B for up to 30 seconds. Every sync manager now exposes a TriggerSync(ctx) method that launches an immediate full sync to all healthy nodes in a background goroutine without blocking the HTTP response. All mutating handlers (create/update/delete) for tenants, bucket permissions, IDP providers, and group mappings call TriggerSync on success; user, access key, group, and capability handlers already had this wired. The periodic polling loop remains as a reconciliation safety net for nodes that were temporarily unreachable. (internal/cluster/tenant_sync.go, internal/cluster/bucket_permission_sync.go, internal/cluster/idp_provider_sync.go, internal/cluster/group_mapping_sync.go, internal/server/console_api.go, internal/server/console_idp.go)
Multilingual UI — 9 languages now supported — the web console previously supported only English and Spanish. All 19 i18n namespaces (~2,200 string keys per locale) are now translated into French (fr), German (de), Italian (it), Brazilian Portuguese (pt), Simplified Chinese (zh), Japanese (ja), and Russian (ru). English is bundled in the main JS chunk as the guaranteed fallback; all other languages are loaded on demand as a single Vite chunk per locale (lang-es.js, lang-fr.js, … lang-ru.js) via import.meta.glob + manualChunks, so switching language triggers exactly one extra network request and the main bundle is unaffected. The Language type union, getSavedLanguage() guard, LanguageContext, TopBar, UserPreferences, AppLayout, and useAuth are all extended to accept all nine locale codes. The TopBar language selector uses a nameKey field on each entry so the language name is resolved through the active locale's translations rather than a hardcoded fallback chain (the previous chain silently showed "Portuguese" for any unrecognised code). (web/frontend/src/locales/{es,fr,de,it,pt,zh,ja,ru}/, web/frontend/src/i18n.ts, web/frontend/src/contexts/LanguageContext.tsx, web/frontend/src/components/layout/TopBar.tsx, web/frontend/src/components/preferences/UserPreferences.tsx, web/frontend/src/components/layout/AppLayout.tsx, web/frontend/src/hooks/useAuth.ts, web/frontend/src/locales/{en,es,fr,de,it,pt,zh,ja,ru}/layout.json, web/frontend/src/locales/{en,es,fr,de,it,pt,zh,ja,ru}/preferences.json)

Removed¶

BadgerDB removed from the codebase — Pebble is now the sole metadata engine — github.com/dgraph-io/badger/v4 has been dropped from go.mod and go.sum along with all BadgerDB implementation files (badger.go, badger_objects.go, badger_rawkv.go, badger_multipart.go, badger_test.go, badger_comprehensive_test.go) and the Badger→Pebble migration logic (migration.go, migration_recovery_test.go). All key-builder functions (bucketKey, objectKey, multipartUploadKey, etc.), ErrNotFound, extractObjectKeyFromKey, matchesFilter, matchesTags, and hasPrefix that were previously defined in the Badger files and reused by the Pebble implementation have been moved verbatim into a new shared file internal/metadata/keys.go so all existing Pebble behaviour is unchanged. server.go no longer calls MigrateFromBadgerIfNeeded on startup. The consistency test (store_consistency_test.go) now runs only against PebbleStore. internal/metrics/badger_history.go is deliberately preserved — despite the name it does not import BadgerDB and operates via the metadata.RawKVStore interface backed by Pebble. Any installation that was still running on BadgerDB metadata (pre-v1.0.0-beta) would need to have migrated first using a v1.x release; upgrading directly from such an installation to this version will start with an empty metadata store. (internal/metadata/keys.go, internal/metadata/pebble_store.go, internal/metadata/store_consistency_test.go, internal/metadata/pebble_test.go, internal/server/server.go, go.mod, go.sum)

Fixed¶

EntityIsNewerThanTombstone LWW check and tombstone insert are now atomic — applyRemoteTombstones performed the LWW freshness check (EntityIsNewerThanTombstone) and the tombstone insert (RecordDeletion) as two separate database operations with no locking in between. A concurrent goroutine deleting the same entity locally could slip in between: the check would see the entity as newer (correct), but by the time RecordDeletion ran, the entity was already gone — leaving the cluster with a missing tombstone and no live entity. Fixed by introducing a sqlQuerier interface (satisfied by both *sql.DB and *sql.Tx) and changing both functions to accept it. Each tombstone in applyRemoteTombstones is now processed inside a single LevelSerializable transaction, making the check and the insert one indivisible operation. (internal/cluster/deletion_log.go, internal/cluster/stale_reconciler.go)
User profile page leaked all access keys to global admins viewing their own profile — the user detail page (/users/{id}) called GET /api/v1/access-keys (the global admin list endpoint, which returns every key from every user) instead of GET /api/v1/users/{id}/access-keys when the viewer was looking at their own profile (isEditingSelf = true). A global admin opening their own profile would see their own keys mixed with keys belonging to every other user in the system. The fix always uses the per-user endpoint regardless of whether the viewer is editing themselves or another user — the userId parameter is always available and unambiguous. (web/frontend/src/pages/users/[user]/index.tsx)
TestClusterAuth_TimestampSkew updated to match the ±30 s HMAC replay window — the test cases "4 minutes in past / future (should pass)" expected the old ±5-minute window and failed after H14 tightened maxSkew to 30 seconds. The cases now use ±20 seconds, which is within the window and still meaningfully distinct from the "6 minutes (should fail)" boundary cases. (internal/middleware/cluster_auth_test.go)
FilesystemBackend.Put now cleans up orphaned metadata when the data-file rename fails — after the A5 change (metadata renamed before data), a failed data rename left the metadata file at its final path with no corresponding data file. Get would return ErrObjectNotFound correctly (it os.Stats the data file first), but the orphaned .metadata file accumulated on disk indefinitely. The failure path now explicitly removes the metadata file before returning the error. (internal/storage/filesystem.go, internal/storage/filesystem_test.go)
Replication credential minimum-length validation now documented in tests — decryptAndValidateCredential rejects any encrypted credential that decrypts to fewer than 8 characters, but the test used "secret" (6 chars) as the destination_secret_key. The test now uses a realistic-length key, confirming that correctly-sized credentials round-trip through AES-256-GCM encryption and the length check without error. (internal/server/server_test.go)
Stale node now retries reconciliation with exponential backoff when no peers are available — Reconcile was called once at startup in a fire-and-forget goroutine; if no healthy peers existed at that moment it returned nil and exited, leaving is_stale = true permanently until the server restarted. The function now returns a typed sentinel ErrNoPeers when peers are absent (preserving is_stale so writes are still blocked), and the server launch loop retries with exponential backoff starting at 5 s, doubling up to a 5-minute cap, stopping when reconciliation succeeds or the server shuts down. (internal/cluster/stale_reconciler.go, internal/server/server.go)
Stale-node reconciliation mode detection is now race-free — detectMode previously read last_local_write_at in a standalone query while HAObjectManager could concurrently set it, creating a TOCTOU window: a node that made writes during a network partition could be classified as offline, causing pushNewerEntities to be skipped and those writes to be permanently lost. The detection now uses a new snapshotAndClearWriteFlag helper that reads is_stale and last_local_write_at together inside a single serializable transaction and immediately clears last_local_write_at. Any write arriving after the transaction commits sets the flag anew, cleanly starting the next reconciliation window rather than leaking into the current one. (internal/cluster/stale_reconciler.go)
HA quorum formula for factor=2 is now clearly documented — the formula neededReplicas = ceil(factor/2) - 1 produces 0 for factor=2, meaning the write is acknowledged once it succeeds locally without waiting for the peer. This was previously undocumented, causing it to appear as a bug. A detailed comment and quorum table have been added to replicaTargets explaining the intentional design: requiring 1 peer confirmation with only 2 cluster nodes would block all writes whenever the single peer is unreachable, which is a worse trade-off than best-effort replication caught up by the stale-object reconciler. (internal/cluster/ha_object_manager.go)
Dead-node last-survivor check is now atomic — markDeadIfSafe previously read the non-dead node count in one query and then performed the status UPDATE in a separate statement, leaving a window where two concurrent callers (e.g., the periodic reconciler and a manual drain) could both observe a safe count and both mark their respective nodes dead, silently dropping the cluster below the configured replication factor. The count query and the UPDATE are now executed inside a single BEGIN IMMEDIATE / SERIALIZABLE SQLite transaction, ensuring no concurrent writer can change the non-dead count between the check and the write. (internal/cluster/dead_node_reconciler.go)
Cluster inter-node HMAC now covers the request body — the HMAC payload for cluster-internal API calls (/api/internal/...) previously signed only method, path, timestamp, and nonce, leaving the body unprotected. An attacker with access to the intra-cluster network could intercept a signed PUT request, substitute an arbitrary body, and forward it to the target node — which would accept the modified payload as authentic. The payload is now METHOD\nPATH\nTIMESTAMP\nNONCE\nSHA256(body) on both the signing side (cluster/proxy.go) and the verification side (middleware/cluster_auth.go). Empty-body requests (GET, DELETE, HEAD) use the fixed SHA-256 of the empty string. (internal/cluster/proxy.go, internal/middleware/cluster_auth.go)
FilesystemBackend.Put now writes metadata before data to prevent silent corruption — the rename order was data-first, metadata-second. If the metadata rename failed (disk full, permission error), the data file was already at its final path and subsequent reads would open it successfully then fail on the metadata read — appearing as a corrupt object. The order is now metadata-first: if the metadata rename fails the data is still in the temp file (no visible change); if the data rename then fails the metadata is in place but the data file is absent, so Get returns a clean ErrObjectNotFound via os.Stat. (internal/storage/filesystem.go)
Bucket size not decremented prematurely when physical versioned file delete fails — deleteSpecificVersion swallowed storage.Delete errors with a Warn log and still decremented the bucket size counter, making the bucket's reported size smaller than actual disk usage. The function now tracks a physicalDeleteOK flag and passes freedBytes = 0 to DecrementObjectCount when the file could not be removed, preserving an accurate bucket size until the integrity scrubber recovers the orphaned file. (internal/object/manager.go)
Orphaned combined file on multipart upload metadata failure — if writing final object metadata (PutObjectVersion / PutObject) failed after CompleteMultipartUpload had already physically combined all parts into the destination file, the combined file was left on disk indefinitely with no metadata record, accumulating unrecoverable storage. The failure path now deletes the orphaned file via storage.Delete before returning the error. (internal/object/manager.go)
Lifecycle delete-marker cleanup now has tests that guard against infinite-loop regression — processExpiredDeleteMarkers deletes expired delete markers by calling DeleteObject with the marker's specific versionID. Without a versionID, calling DeleteObject on a versioned bucket creates a new delete marker instead of removing the existing one, which would cause the lifecycle worker to loop indefinitely. Two tests were added: one that asserts the exact versionID is forwarded to DeleteObject (preventing the unversioned-delete regression), and one that verifies the worker handles a concurrent deletion (the marker is already gone when the worker processes it) gracefully without panicking. (internal/lifecycle/worker_test.go)
HA replica writes no longer enforce tenant quota limits — the primary validates and increments the tenant quota before fanning a write out to replicas. Replicas called the full PutObject path including CheckTenantStorageQuota, which could reject a replicated write if the replica's local counter was already near the limit (e.g., because the primary incremented it in a prior cycle). Replicas must accept the write unconditionally (the primary already validated it) and only update their own local usage counter so they have an accurate view if they later become primary. A new WithBypassQuotaEnforcement context function in the object package is called by cluster.WithHAReplicaContext; PutObject checks this flag and skips CheckTenantStorageQuota while still calling IncrementTenantStorage. (internal/object/manager.go, internal/cluster/ha_object_manager.go)
computeMultipartETag now rejects part ETags that are not valid 32-character MD5 hex strings — previously, when hex.DecodeString failed on a part ETag it silently fell back to md5.Sum([]byte(etag)), producing a different ETag than the S3 specification requires and masking data corruption (e.g., a truncated or misformatted ETag stored during an earlier bug). The fallback is removed; the function now returns an explicit error if any part ETag is not exactly 32 hex characters or cannot be decoded, preserving the integrity of the final object's ETag and making corruption detectable at upload-completion time rather than silently. (internal/object/manager.go)
Bucket metrics after versioned PUT no longer race against concurrent writes — updateBucketMetricsAfterPut contained a fallback path for the versioned-bucket branch: when existingObjBeforeSave was nil it re-queried GetObjectVersions after the save to decide whether to increment the object count or only adjust the size. Because the query ran after the save, a concurrent PutObject on the same key could insert another version between the save and the query, making the total version count appear ≥ 2 even for a first-time upload — the size would be adjusted instead of the object count being incremented, silently undercounting the bucket. The fallback is removed; all callers are expected to supply existingObjBeforeSave captured before the save (the primary production call site at PutObject already did this correctly at line 617). The coverage test for additional versions was updated to fetch the pre-save state from the metadata store before the second PutObject call. (internal/object/manager.go, internal/object/manager_final_coverage_test.go)
Replication credential decryption failures now surface as an error with an explicit log — getRule and GetRule called decryptCredential and returned its result without additional logging; if the key had rotated or the ciphertext was corrupt, the caller received a generic wrapped error with no indication of the root cause. A new decryptAndValidateCredential wrapper logs at Error level before returning a typed ErrDecryptionFailed sentinel, and additionally validates that an encrypted (enc1:-prefixed) credential decrypts to at least 8 characters — a partial plaintext write during a previous save could otherwise produce a valid but useless short string. Legacy plaintext credentials (no enc1: prefix) are returned as-is to preserve backwards compatibility. (internal/replication/credentials.go, internal/replication/manager.go, internal/replication/worker.go)
Lifecycle worker now paginates object expiration so objects beyond the first page are cleaned up — processObjectExpiration fetched at most 10,000 objects in a single call and stopped, leaving older objects in buckets with more than 10,000 entries permanently unexpired. The worker now loops with a pagination marker until IsTruncated is false, processing 1,000 objects per page. A context-cancellation check at the top of each loop iteration ensures the worker stops promptly on server shutdown. (internal/lifecycle/worker.go)
DeleteBucket empty-check and metadata delete are now protected against late object writes — the previous implementation called isBucketEmpty and DeleteBucket as two separate store calls with no lock in between, so a concurrent PutObject could create object metadata after the empty-check passed but before the bucket key was removed. A first fix moved the check into DeleteBucketIfEmpty, but the active Pebble v2 store still performed a prefix scan and delete as separate Pebble operations, leaving a smaller in-process race. Pebble now serializes bucket deletion against object metadata writes (PutObject, PutObjectVersion, and completed multipart metadata commits) with a per-bucket mutation lock and rejects late writes to buckets deleted during the process lifetime. Added a regression test that verifies a write after DeleteBucketIfEmpty fails with ErrBucketNotFound and does not leave orphan object metadata. BadgerDB is no longer part of the codebase, so this protection is implemented only in the Pebble-backed metadata store. (internal/metadata/store.go, internal/metadata/pebble_store.go, internal/metadata/pebble_objects.go, internal/metadata/pebble_multipart.go, internal/metadata/pebble_test.go, internal/bucket/manager_impl.go)
Replication queue race — same object could be replicated twice — when the in-memory worker channel was full, loadPendingItems released the claim back to pending immediately. The next loader tick would re-claim and re-enqueue the same item, so a worker could process it a second time before the first worker finished. Fixed by keeping the claim in retrying state instead of releasing it; a new resetStaleClaimedItems sweep runs at the start of each loader tick and automatically recovers any item that has been in retrying for more than 60 seconds without being picked up by a worker (e.g., because the channel drained faster than the process could proceed). claimQueueItem now also stamps processed_at = CURRENT_TIMESTAMP so the staleness threshold is always well-defined. (internal/replication/manager.go)
CreateBucket rollback now also reverts the ACL when storage marker creation fails — when the storage .maxiofs-bucket marker Put failed, the rollback deleted the bucket metadata entry but left the ACL key (acl:bucket:<tenantID>:<name>) in the store, creating an orphaned ACL record that would be silently applied to any future bucket created with the same name. A new DeleteBucketACL method on acl.Manager deletes the ACL key (ignoring ErrNotFound to remain idempotent), and the rollback path now calls it immediately after the metadata rollback so the store is fully clean on failure. (internal/acl/interface.go, internal/acl/manager.go, internal/bucket/manager_impl.go)
Lifecycle prefix filter now applied consistently to all action paths — processAbortIncompleteMultipartUploads listed all in-progress multipart uploads for a bucket and aborted any that exceeded DaysAfterInitiation, regardless of the rule's prefix filter. This meant lifecycle rules scoped to a prefix (e.g., logs/) would abort uploads for any key in the bucket. The function now skips uploads whose key does not start with the configured prefix, consistent with how processObjectExpiration, processNoncurrentVersionExpiration, and processExpiredDeleteMarkers already apply the filter. (internal/lifecycle/worker.go)
CompleteMultipartUpload now defaults Content-Type to application/octet-stream when not set — if the initiating CreateMultipartUpload request did not include a Content-Type header, the assembled object was stored and returned with an empty ContentType field, which caused subsequent GetObject responses to omit the Content-Type header. The returned object, the encrypted storage metadata, and the unencrypted storage metadata all now fall back to application/octet-stream when the multipart metadata does not include a content-type, matching the behavior of regular PutObject. (internal/object/manager.go)
Nil S3 client from replication factory no longer causes a panic — replicateObject and replicateDelete called w.s3ClientFactory(...) and immediately used the returned S3Client interface value without checking for nil. A factory that returns a nil interface (possible in test overrides or future custom factories) would cause a nil-pointer panic on the first method call. Both functions now check the result and return a descriptive error instead of panicking. (internal/replication/worker.go)
Audit log failures are now surfaced as Warn logs instead of being silently discarded — several audit call sites still discarded auditManager.LogEvent errors with _ = ..., so a failed audit write (DB full, I/O error, SQLite contention) could be invisible in compliance-sensitive deployments. The bucket, auth, replication-rule, disk-alert, cluster-node-alert, and integrity audit paths now route through non-blocking helpers that log failures at Warn level instead of suppressing them. A follow-up scan confirms there are no remaining discarded LogEvent calls in internal/auth, internal/server, or internal/bucket. (internal/bucket/manager_impl.go, internal/auth/audit_helpers.go, internal/server/console_api_replication.go, internal/server/cluster_node_alerts.go, internal/server/disk_alerts.go, internal/server/object_integrity_handler.go, internal/server/integrity_scrubber.go)
HA metadata fanout now retries with per-attempt timeout and logs divergence on permanent failure — fanoutMetadata dispatched a single goroutine per replica with no timeout or retry. If a transient network error hit during the fire-and-forget, the metadata update was silently lost. Each replica now gets up to 3 attempts with a 5-second per-attempt timeout (using a background context so the replica goroutine outlives the original request) and 200 ms / 400 ms inter-attempt delay. If all retries fail, the failure is logged at Error level identifying the replica and op so operators can detect metadata divergence from logs. (internal/cluster/ha_object_manager.go)
Last-survivor protection now counts only healthy nodes — markDeadIfSafe checked health_status != dead (including UNAVAILABLE and DEGRADED nodes) when deciding whether marking a node dead would drop the cluster below the replication factor. A cluster with many unavailable nodes could have its count inflated above factor, allowing the last truly healthy node to be marked dead. The query now counts health_status = healthy AND id != candidate so only write-capable nodes are counted. (internal/cluster/dead_node_reconciler.go)
Anti-entropy pull now logs when an object is absent on the peer instead of silently returning — pullObjectFromPeer returned nil on HTTP 404 with a code comment ("race; skip") but no log output. Operators had no way to see that a divergence had resolved itself remotely. A short Info log is now emitted identifying the peer, bucket, and key so the event appears in the scrubber trace. (internal/cluster/anti_entropy.go)
Anti-entropy ETag tie now resolved deterministically instead of being left unresolved indefinitely — lwwClassify returned actNone when two replicas had the same LastModified timestamp but different ETags, leaving the divergence in the "found but not fixed" state forever (visible in scrub metrics as a perpetual unfixed divergence count). The function now uses lexicographic ETag order as a stable tie-breaker: the node with the smaller ETag defers to its peer (pulls), the node with the larger ETag wins (pushes). Both sides reach the same conclusion independently, so the cluster converges in one scrub cycle. (internal/cluster/anti_entropy.go, internal/cluster/anti_entropy_test.go)
Duplicate dead-node SSE events suppressed when node was already dead — markDeadIfSafe emitted a node_dead SSE event unconditionally after the UPDATE, even when the AND health_status != dead guard in the SQL meant 0 rows were affected (node was already dead from a concurrent call). The function now checks RowsAffected() and returns early without emitting the event if the UPDATE was a no-op. (internal/cluster/dead_node_reconciler.go)
Storage-pressure SSE callback now runs in a goroutine with panic recovery — the storagePressureFn callback was called directly inside the health-check goroutine. A blocking callback (e.g., a slow SSE write) would stall the health ticker for all nodes in the cluster; a panicking callback would crash the goroutine entirely. The callback is now dispatched in a dedicated goroutine with a defer recover() that logs the panic at Error level without propagating it. (internal/cluster/health.go)
HASyncWorker now holds the sync-running lock through the DB existence check — runTrigger released mu before querying the DB for an existing sync job, creating a window where two concurrent Trigger calls could both see no running job and start duplicate syncs for the same node. The lock is now held through the DB query; if no job is found, the slot is reserved in w.running before the lock is released. If the subsequent DB insert fails, the slot is released. (internal/cluster/ha_sync_worker.go)
Inter-node HMAC replay window reduced from ±5 minutes to ±30 seconds — the cluster proxy and authentication middleware accepted signed requests up to 5 minutes old, giving an attacker on the intra-cluster network a large capture-and-replay window. Inter-node clocks are NTP-synchronised; the window is tightened to ±30 seconds (matching common practice for signed API tokens on private networks). (internal/cluster/proxy.go, internal/middleware/cluster_auth.go)
Tenant quota not enforced on versioned-bucket uploads — PutObject skipped CheckTenantStorageQuota entirely when versioning was enabled (condition included !versioningEnabled), so tenants could upload unlimited data to versioned buckets. For non-versioned buckets the delta between old and new object sizes was correctly checked; for versioned buckets the new version size must be added in full (all versions are preserved). The !versioningEnabled guard is removed; the function now branches on versioning to compute the correct size increment before the quota check. (internal/object/manager.go)
PutObjectTags concurrent update race — last write silently discarded — PutObjectTags read the object record, merged the new tags, and wrote the result back without holding the per-bucket mutation lock. Two concurrent tag updates on the same key could both read the pre-merge state and one set of tags would be lost. All other write paths (PutObject, PutObjectVersion, multipart completion) already hold getBucketMutationMutex for their read-modify-write. The mutex is now acquired at the top of PutObjectTags as well. (internal/metadata/pebble_objects.go)
OAuth tokens exposed in redirect URL — replaced with server-side one-time code exchange — handleOAuthCallback placed the access token and refresh token directly in the redirect URL query string (?token=…&refresh_token=…). Tokens appeared in server access logs, browser history, proxy logs, and any Referer header on the next navigation, violating RFC 6819 §4.2.2. The callback now generates a cryptographically random 32-byte hex one-time code (TTL 60 s, stored in a sync.Map on the server) and redirects with only ?code=<code>. A new public endpoint GET /api/v1/auth/oauth/exchange-code atomically consumes the code (LoadAndDelete) and returns the token pair in the JSON response body, where it is not logged or cached by intermediaries. The frontend oauth-complete.tsx is updated to call the exchange endpoint instead of reading tokens from the URL. (internal/server/console_idp.go, internal/server/server.go, internal/server/console_api.go, web/frontend/src/pages/auth/oauth-complete.tsx)
Deactivated users bypass JWT validation — ValidateJWT looked up the user by username and returned it without checking user.Status, so an admin deactivating a user had no immediate effect: the user's access tokens (up to 15-min TTL) and refresh tokens (up to 24-h TTL) remained valid. ValidateCredentials and ValidateConsoleCredentials already checked user.Status != UserStatusActive. The same check is now applied in ValidateJWT, returning ErrUserInactive immediately and preventing any further use of tokens issued before deactivation. (internal/auth/manager.go)
Data race in replication rule scheduler — lastSync map written from multiple goroutines — the main ruleScheduler goroutine read and deleted entries from lastSync while spawned per-rule goroutines wrote to it with no synchronization, constituting a classic Go map race that would trigger a runtime panic under concurrent replication activity. Fixed by recording lastSync[ruleID] = now in the main goroutine before spawning the rule goroutine; the goroutine no longer writes to the map, eliminating the shared mutable state entirely without adding a mutex. (internal/replication/manager.go)
Replication queue cleanup silently discarded database errors — cleanupOldQueueItems discarded the error from db.ExecContext with _. If the DELETE failed (disk full, locked table), the replication queue would grow without bound with no operator-visible indication. The error is now checked and logged at Warn level. (internal/replication/manager.go)
JSON marshal errors silently ignored in HA metadata fanout — six json.Marshal calls in HAObjectManager (for object metadata, tags, ACL, retention, legal-hold, and restore-status operations) discarded their error with _. A marshal failure would pass a nil byte slice to fanoutMetadata, which would send empty or malformed data to replica nodes, causing silent metadata divergence that would only surface during a subsequent anti-entropy scrub. Each call now checks the error; on failure it logs at Warn level with bucket and key fields and skips the replica fanout rather than sending bad data. (internal/cluster/ha_object_manager.go)
Stop() panics on double-call in lifecycle worker and deletion-log sync manager — both Worker.Stop() and DeletionLogSyncManager.Stop() closed the stopChan unconditionally, so a second call (possible if the HTTP server and the context cancel race during shutdown) would panic with "close of closed channel". Added stopOnce sync.Once to both structs; both Stop() methods now wrap the close() inside stopOnce.Do(...). (internal/lifecycle/worker.go, internal/cluster/deletion_log.go)

[1.4.0] - 2026-05-03¶

Added¶

Role capabilities system — service-level permission matrix stored entirely in the database — previously all users shared the same implicit permission set: any authenticated user with the user role could create buckets, delete buckets, upload and delete objects, and manage policies. There was no mechanism to restrict what a specific user or role could do at the service level, making it impossible to create read-only or limited-access users without relying exclusively on per-bucket ACLs. The new capabilities system introduces a two-layer model on top of the existing bucket_permissions: role defaults (what a role allows by default) and per-user overrides (explicit allow/deny set by an admin on top of the role default). Eleven capabilities are defined: bucket:create, bucket:delete, bucket:configure, bucket:manage_policy, object:upload, object:download, object:delete, object:manage_tags, object:manage_versions, console:access, keys:manage_own. Resolution order for every capability check: (1) explicit user override deny → false; (2) explicit user override grant → true; (3) role admin safety net → true; (4) role default in role_capabilities table → true/false; (5) default → false. The admin role always has all capabilities regardless of the table, so global admins can never accidentally lock themselves out. (internal/auth/capabilities.go, internal/db/migrations/migration13_capabilities.go)
role_capabilities and user_capability_overrides database tables — migration 13 creates two new SQLite tables. role_capabilities (role TEXT, capability TEXT, PRIMARY KEY(role, capability)) stores the per-role defaults and is seeded at migration time with opinionated defaults: admin gets all capabilities; user gets all except bucket:manage_policy; read gets object:download, console:access, keys:manage_own; readonly gets object:download, console:access; guest gets object:download only. user_capability_overrides (id TEXT, user_id TEXT, capability TEXT, granted INTEGER, granted_by TEXT, created_at INTEGER, UNIQUE(user_id, capability)) stores per-user overrides created by admins. All defaults are database-stored and editable by a global admin at runtime — no hardcoded values in Go code. (internal/db/migrations/migration13_capabilities.go, internal/db/migrations/versions.go)
Five new capability management API endpoints — GET /api/v1/users/{id}/capabilities returns the full effective capability matrix for a user with per-capability source annotation (role or override); accessible by any admin scoped to the user's tenant. PUT /api/v1/users/{id}/capabilities/{capability} with body {"granted": true|false} sets an explicit per-user override; DELETE /api/v1/users/{id}/capabilities/{capability} removes the override and reverts to the role default. Both write endpoints trigger a cluster user-sync so overrides propagate to all nodes immediately. GET /api/v1/roles/capabilities returns the full role→capabilities map alongside all_capabilities for UI enumeration; global admin only. PUT /api/v1/roles/{role}/capabilities with body {"capabilities": [...]} atomically replaces a role's capability set; global admin only. All endpoints validate capability names against the AllCapabilities constant list and enforce tenant-scoping (tenant admins can only manage users in their own tenant). (internal/server/capability_handlers.go, internal/server/console_api.go)
Capability enforcement in S3 and console API handlers — HasCapabilityInContext checks are wired into all relevant handlers so capability violations return the correct HTTP error rather than silently allowing the operation. Console API: handleCreateBucket checks bucket:create; handleCreateAccessKey checks keys:manage_own when a non-admin user creates a key for themselves; handleLogin checks console:access after credentials are validated and before 2FA, returning 403 Forbidden for users who are not permitted to use the console. S3 API: CreateBucket checks bucket:create; DeleteBucket checks bucket:delete; PutObject checks object:upload; DeleteObject checks object:delete; GetObject checks object:download (presigned URL access and anonymous public-bucket requests are exempt to preserve existing share workflows); PutBucketVersioning checks bucket:configure; PutBucketPolicy / DeleteBucketPolicy check bucket:manage_policy; PutBucketLifecycle, PutBucketCORS, PutBucketTagging, PutBucketEncryption check bucket:configure; PutObjectTagging / DeleteObjectTagging check object:manage_tags. Capability checks are bypassed when authManager is nil (S3 unit tests without auth initialisation). (internal/server/console_api.go, internal/server/capability_handlers.go, pkg/s3compat/handler.go, pkg/s3compat/bucket_ops.go, pkg/s3compat/object_ops.go)
Capability overrides travel with user sync in cluster — UserData in the cluster user-sync payload was extended with a CapabilityOverrides []CapabilityOverrideData field carrying each override's Capability, Granted, GrantedBy, and CreatedAt. listLocalUsers loads overrides from the DB per user before building the payload; computeUserChecksum includes the override signature so a change in overrides triggers a full re-sync. The receiving node's upsertUser performs a full replace of overrides for the incoming user (DELETE all existing + INSERT from payload) — Last-Writer-Wins semantics consistent with the rest of the user sync. (internal/cluster/user_sync.go, internal/server/cluster_tenant_handlers.go)
"Manage Permissions" button and modal in the user detail page — admins see a new "Manage Permissions" button in the user profile header (alongside Edit and Change Password). Clicking it opens a modal showing the user's full capability matrix as a grid of toggle switches: green = granted, gray = denied. When a capability has an explicit override a labelled badge and a reset button (↺) appear alongside the toggle, allowing the admin to revert to the role default in one click. The modal fetches capabilities on demand (no extra request when simply viewing a profile). Mutations call the PUT/DELETE capability endpoints immediately and refetch on success. (web/frontend/src/pages/users/[user]/index.tsx, web/frontend/src/lib/api.ts, web/frontend/src/types/index.ts, web/frontend/src/locales/en/users.json, web/frontend/src/locales/es/users.json)
Role Capabilities management page for global admins — new page at /roles/capabilities (accessible via the sidebar under Users → Role Capabilities, visible to global admins only) renders an interactive matrix table with roles as columns and capabilities as rows. Each cell is a toggle; the admin column is read-only (always all-granted). Unsaved changes highlight the column header and reveal a per-role Save button, allowing the admin to review all pending changes for a role before committing. Changes are saved atomically via PUT /api/v1/roles/{role}/capabilities and take effect immediately for all users of that role. (web/frontend/src/pages/roles/capabilities.tsx, web/frontend/src/components/layout/SidebarNav.tsx, web/frontend/src/components/layout/AppLayout.tsx, web/frontend/src/App.tsx, web/frontend/src/locales/en/users.json, web/frontend/src/locales/es/users.json, web/frontend/src/locales/en/navigation.json, web/frontend/src/locales/es/navigation.json)

Fixed¶

Release and package builds now strip debug symbols consistently — build, build-server, build-all, platform-specific builds, Docker images, Debian source builds, .deb, and RPM package paths now use stripped Go linker flags (-s -w), -trimpath, and CGO_ENABLED=0 for cleaner, smaller binaries without embedded local source paths or accidental native-library dependencies. The .deb targets also no longer reference the undefined FLAGS_RELEASE variable, which previously caused Debian packages to be built without release linker flags. (Makefile, Dockerfile, debian/rules)
Multipart uploads stored all HTTP request headers as object user metadata — CreateMultipartUpload iterated over every header in the incoming request and stored them all in the upload's metadata map without filtering. When CompleteMultipartUpload committed the final object, that unfiltered map was used as both the storage metadata and the user metadata (x-amz-meta-*) of the resulting object. Any client using the AWS SDK or CLI would therefore permanently embed Authorization, User-Agent, Amz-Sdk-Invocation-Id, X-Amz-Date, X-Amz-Content-Sha256, and other internal request headers into the object record, exposing sensitive values (including the full SigV4 authorization token) via HeadObject to any caller with read access. Fixed by replacing the raw header loop in CreateMultipartUpload with extractMetadataFromHeaders — the same helper used by PutObject — which whitelists only Content-Type, Content-Disposition, Content-Encoding, Cache-Control, Content-Language, and x-amz-meta-* headers. x-amz-acl is additionally preserved as an internal-only key in the upload metadata (consumed by CompleteMultipartUpload to apply the canned ACL after the object is written) and excluded from user metadata by filterStorageMetadataKeys so it does not appear in HeadObject responses. The bug was present since the multipart upload feature was introduced and is not specific to any single release. (internal/object/manager.go)
Frontend inactivity timeout no longer treats background token refresh as user activity — token refreshes updated last_activity and dispatched a keep-alive event consumed by the idle timer, allowing an inactive browser tab to stay authenticated indefinitely as long as refresh succeeded. The idle timer now advances only on real browser interaction events, while active users still keep their session alive through normal UI activity. (web/frontend/src/lib/api.ts, web/frontend/src/hooks/useIdleTimer.ts, web/frontend/src/__tests__/useIdleTimer.test.ts)
Tenant-scoped Object Lock and legal-hold console actions now preserve tenant context — several frontend API helpers dropped tenantId for object legal hold, object ACL updates, and Object Lock configuration updates, so global admins managing tenant buckets could hit the wrong bucket scope or receive false failures. Those calls now propagate tenantId, and the Object Lock modal invalidates the tenant-specific bucket query after saving. (web/frontend/src/lib/api.ts, web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/pages/buckets/[bucket]/settings.tsx, web/frontend/src/components/ObjectLockConfigModal.tsx)
Bucket replication settings now resolve the correct source bucket context without constraining destinations — the console replication endpoints and frontend helpers did not carry tenantId when a global admin opened a tenant-scoped bucket, so listing or managing rules could target the global bucket namespace instead of the tenant bucket. The tenant context now applies only to the MaxIOFS source bucket lookup; destination endpoint, bucket, region, and credentials remain fully S3-compatible and can point to MaxIOFS, AWS S3, MinIO, or any object server. The frontend replication helpers also unwrap the console API's standard {success,data} envelope correctly, and the replication rule form now preserves enabled: false instead of forcing saved rules back to enabled. (internal/server/console_api_replication.go, internal/server/server_test.go, web/frontend/src/lib/api.ts, web/frontend/src/pages/buckets/[bucket]/settings.tsx)
Frontend object operations now encode reserved characters in console API routes — several object helpers interpolated object keys or tenant IDs directly into URLs, so keys containing reserved URL characters such as ?, #, %, or spaces could be truncated or routed incorrectly from the web console. Console object reads, deletes, renames, tags, shares, version operations, and presigned URL generation now encode path/query components consistently. (web/frontend/src/lib/api.ts)
OAuth completion cookies now work on HTTP console deployments — the OAuth callback always wrote auth_token and refresh_token cookies with the Secure attribute, unlike the normal login flow. In HTTP lab or local deployments those cookies were ignored by the browser, leaving OAuth sessions partially initialized. The callback now adds Secure only when the current page is served over HTTPS. (web/frontend/src/pages/auth/oauth-complete.tsx)
Cluster bucket-permission sync test no longer leaks a background sync loop — TestBucketPermissionSyncManager_Start/disabled_auto_sync assumed auto-sync was disabled by default, but the replication schema seeds auto_bucket_permission_sync_enabled=true. The test now disables the setting explicitly before calling Start, preventing a background goroutine from continuing after the test database is closed and making go test ./internal/cluster reliable again. (internal/cluster/bucket_permission_sync_test.go)
Folder delete confirmation in the web console now matches recursive delete behavior — the bucket browser warned that folder deletion would fail when the folder contained objects, but the frontend already performs recursive deletion before removing the folder marker. The warning now states that all objects inside the folder will be deleted, and the delete modal text is localized through the bucket locale files. (web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/locales/en/buckets.json, web/frontend/src/locales/es/buckets.json)
JSON-typed dynamic settings now reject malformed JSON — settings declared with type json were accepted without validation in both single-setting updates and bulk updates, allowing invalid JSON to be stored in SQLite and fail later when consumers parsed it. JSON settings now use encoding/json validation before commit, and bulk updates still roll back atomically if any JSON value is invalid. (internal/settings/manager.go, internal/settings/manager_test.go)
Readiness checks now reflect real object/auth dependencies — ObjectManager.IsReady() and AuthManager.IsReady() always returned true, so /ready could report the node as ready even after the metadata store or SQLite auth database was closed/unavailable. Object readiness now verifies storage, metadata store readiness, filesystem root availability, and encryption service presence when encryption is enabled; auth readiness now pings SQLite. The API /ready endpoint also includes auth readiness instead of checking only bucket/object managers. (internal/object/manager.go, internal/auth/manager.go, internal/api/handler.go)
Metrics collector no longer reports fabricated network and file-descriptor values — the system collector returned hardcoded open-file-descriptor and network byte counters, and runtime collection events encoded numeric fields with string(rune(...)), producing unreadable control characters instead of decimal values. The collector now reads process FDs and network counters from gopsutil, serializes runtime event fields as decimal strings, and can be stopped and started again without reusing a closed stop channel. (internal/metrics/collector.go, internal/metrics/collector_test.go)
HA object replication now preserves version IDs for versioned objects and delete markers — HA fanout replays used PutObject/DeleteObject on replica nodes without carrying the primary node's versionId, so each replica generated different version IDs for the same object version or delete marker. That could break version-specific reads, deletes, Object Lock operations, and repair workflows when a request hit a replica. HA PUT now carries the source version ID through an internal header and trusted context, HA DELETE carries either the specific version ID or the delete-marker version ID, and stale/anti-entropy pulls preserve the peer's current version ID as well. (internal/cluster/ha_object_manager.go, internal/server/cluster_object_handlers.go, internal/object/manager.go, internal/cluster/anti_entropy.go, internal/cluster/stale_reconciler.go)
S3 clients can now delete real filesystem folder markers even when they omit the trailing slash — clients such as S3 Browser may issue DeleteObject/DeleteObjects for folder while MaxIOFS stores the real directory marker as folder/. The object manager now resolves that alias only when no exact object exists, so folder metadata is removed consistently, and the filesystem backend detects directories by stat instead of relying only on a trailing / in the key. (internal/object/manager.go, internal/storage/filesystem.go, internal/object/manager_cleanup_test.go, internal/storage/filesystem_test.go, pkg/s3compat/s3_test.go)
Frontend sessions now keep refreshing past the 15-minute access-token window — the web client scheduled proactive refresh by decoding the JWT payload with plain atob, which is not safe for base64url-encoded JWT segments and could fail silently. When that happened, active users could still be logged out as soon as the 15-minute access token expired. JWT payload decoding now handles base64url padding/characters correctly, restoring proactive refresh for active console sessions. (web/frontend/src/lib/api.ts, web/frontend/src/__tests__/api-token.test.ts)
Bucket deletion now treats metadata-only objects, versions, and delete markers as real bucket contents — DeleteBucket previously checked only physical storage, so a bucket with object metadata but missing files, or a bucket containing only version/delete-marker metadata, could be deleted as if empty. isBucketEmpty now checks the metadata version index, and ForceDeleteBucket removes nested object/version metadata before physical cleanup. Failed physical bucket creation also rolls back the metadata bucket record. (internal/bucket/manager_impl.go, internal/bucket/delete_bucket_test.go)
S3 object operations now resolve the bucket's actual tenant instead of assuming the request user's tenant — authenticated tenant users accessing global buckets were routed to tenantID/bucket even though global buckets intentionally have no tenant prefix. GetObject, PutObject, DeleteObject, HeadObject, and GetObjectAttributes now resolve the bucket owner from metadata first, preserving bare global bucket paths and avoiding tenant quota checks for writes into non-tenant buckets. (pkg/s3compat/handler.go, pkg/s3compat/object_ops.go, pkg/s3compat/s3_test.go)
GetObjectAttributes now honors versionId — the handler read the versionId query parameter but always returned latest metadata, so clients asking for attributes of an older object version received the current version's ETag, size, and version header. Versioned requests now read the specific version through GetObject. (pkg/s3compat/object_ops.go, pkg/s3compat/s3_test.go)
S3 Object Lock retention and legal hold operations now honor versionId — GetObjectRetention and GetObjectLegalHold always read the latest version, and PutObjectLegalHold ignored the requested version entirely. Clients targeting an older object version could read the wrong lock state or accidentally apply legal hold to the latest version. The Object Manager now resolves lock metadata for a specific version when requested, and the S3 handlers propagate versionId consistently for retention/legal-hold reads and writes. (internal/object/manager.go, pkg/s3compat/object_ops.go, pkg/s3compat/s3_test.go)
S3 object tagging operations now honor versionId — GetObjectTagging, PutObjectTagging, and DeleteObjectTagging always operated on the latest version even when clients targeted a specific object version. Versioned tagging now reads and mutates the requested version metadata, and HA metadata fanout carries the version ID so replicas apply the same per-version tag update. (internal/object/manager.go, internal/cluster/ha_object_manager.go, internal/server/cluster_object_handlers.go, pkg/s3compat/object_ops.go, pkg/s3compat/s3_test.go)
S3 RestoreObject now honors versionId — restore requests for a specific object version verified and updated latest metadata instead, so a restore marker could be written to the wrong version. Restore status now resolves the requested version and HA metadata fanout carries the version ID for replica replay. (internal/object/manager.go, internal/cluster/ha_object_manager.go, internal/server/cluster_object_handlers.go, pkg/s3compat/handler.go, pkg/s3compat/s3_test.go)
S3 object ACL operations now honor versionId — object ACLs were stored only in the legacy object-ACL namespace keyed by bucket/key, so GetObjectACL and PutObjectACL could not distinguish versions and always affected the effective latest object ACL. Object ACLs now persist into object metadata, versioned ACL requests target the requested version, unversioned requests keep updating the legacy ACL store for compatibility, and HA metadata fanout carries the version ID for replica replay. (internal/object/manager.go, internal/cluster/ha_object_manager.go, internal/server/cluster_object_handlers.go, pkg/s3compat/object_ops.go, pkg/s3compat/s3_test.go)
Multipart upload validation now rejects missing uploads, wrong ETags, and invalid part order before mutating state — UploadPart wrote the physical part file before verifying that the upload ID existed, leaving orphaned data when metadata rejected the part. ListParts could return an empty list for a nonexistent upload, and AbortMultipartUpload could report success for an invalid upload ID. CompleteMultipartUpload also accepted duplicate/out-of-order parts and ignored client-supplied ETags, allowing completion with the wrong part list. The object manager now verifies upload existence before writing parts, maps missing uploads consistently to NoSuchUpload, validates strict ascending part order and ETag matches, and preserves the upload for retry after failed completion validation. (internal/object/manager.go, pkg/s3compat/multipart.go, internal/object/multipart_validation_test.go)
Console API mutating routes now enforce service-level capabilities consistently — after adding the capability system, many console routes still required only an authenticated user: bucket deletion, object upload/delete, lifecycle, bucket tags, CORS, ACLs, policies, versioning, Object Lock, legal hold, notifications, encryption, public access block, inventory, website hosting, object tag edits, object rename, and version restore. Those routes now use the same capability guard as S3: bucket configuration changes require bucket:configure, bucket policy changes require bucket:manage_policy, bucket deletion requires bucket:delete, object upload/delete require their matching object capabilities, object tag edits require object:manage_tags, and version/legal-hold restore operations require object:manage_versions. (internal/server/capability_guard.go, internal/server/console_api.go, internal/server/object_extra_handlers.go, internal/server/inventory_handlers.go, internal/server/website_handlers.go)
HA object replication now escapes object keys in internal URLs — HA fanout, initial sync, stale-node pull, and anti-entropy repair built /api/internal/ha/objects/{key} URLs with raw object keys. Keys containing query or fragment control characters such as ?, #, spaces, or literal % could be truncated or routed incorrectly during replication. Internal HA callers now escape each path segment while preserving slash-separated prefixes. (internal/cluster/ha_url.go, internal/cluster/ha_object_manager.go, internal/cluster/ha_sync_worker.go, internal/cluster/stale_reconciler.go, internal/cluster/anti_entropy.go)
S3 versioned object responses now match AWS semantics for delete markers and HEAD — GET/HEAD over a current delete marker returned a plain NoSuchKey without x-amz-delete-marker or x-amz-version-id, explicit versionId requests could collapse missing versions into NoSuchKey or idempotent delete success, and HEAD Object omitted x-amz-version-id for real versions. S3 handlers now return delete-marker headers for current markers, MethodNotAllowed for explicitly requested delete-marker versions, NoSuchVersion for missing explicit versions, and version headers on HEAD responses. (pkg/s3compat/handler.go, pkg/s3compat/s3_test.go)
S3 ListObjectsV2 KeyCount now includes common prefixes — when clients used delimiter, KeyCount counted only <Contents> entries and ignored <CommonPrefixes>, which made paginated SDK listings see fewer returned results than the response actually contained. KeyCount now follows S3 semantics by counting both object entries and common-prefix entries. (pkg/s3compat/handler.go, pkg/s3compat/s3_test.go)
S3 x-amz-tagging and CopyObject tag directives are now honored — PutObject ignored the x-amz-tagging header, and CopyObject created the destination without copying source tags even though S3 defaults x-amz-tagging-directive to COPY. PUT now persists URL-encoded tag headers, CopyObject copies source tags by default, and x-amz-tagging-directive=REPLACE applies replacement tags from the request. (pkg/s3compat/handler.go, pkg/s3compat/object_ops.go, pkg/s3compat/s3_test.go)
S3 CopyObject now goes through source read and destination write authorization — copy requests were dispatched before the normal PutObject permission path, so x-amz-copy-source operations could bypass service-level object capabilities and bucket/object ACL checks. CopyObject now requires source read permission plus destination write permission, including object:download and object:upload capabilities for authenticated S3 requests. (pkg/s3compat/object_ops.go, pkg/s3compat/s3_test.go)
S3 multipart copy now validates upload state, source conditionals, and capabilities consistently — UploadPartCopy fetched the source object before verifying that the multipart upload existed, did not apply x-amz-copy-source-if-* preconditions, and did not enforce the same source download/destination upload capabilities as CopyObject. Multipart copy now returns NoSuchUpload before source reads, honors copy-source preconditions, maps missing explicit source versions to NoSuchVersion, and checks source read plus destination upload permissions. (pkg/s3compat/multipart.go, pkg/s3compat/object_ops.go, pkg/s3compat/s3_test.go)
S3 presigned V4 URLs now enforce signed headers and expiration limits — V4 validation trusted X-Amz-SignedHeaders but only canonicalized host, so URLs that claimed to sign additional headers did not require those headers and could ignore tampering. Presigned generation also ignored PresignedURLConfig.Headers, and validation accepted X-Amz-Expires values beyond the seven-day SigV4 limit. Signed headers are now canonicalized from the request, missing/tampered signed headers fail signature validation, generated URLs include configured signed headers, and invalid expiration windows are rejected. (pkg/s3compat/presigned.go, pkg/s3compat/presigned_test.go)
S3 error mapping and multipart completion error XML are now stricter — MalformedPOSTRequest and InvalidPolicyDocument were emitted by browser POST presigned upload validation but were not mapped in the shared S3 error writer, causing malformed policy/form requests to fall through as 500 responses. Those codes now return 400 Bad Request, NoSuchVersion is covered in the error mapper tests, and CompleteMultipartUpload embedded error bodies are XML-encoded instead of string-formatted so error messages cannot produce malformed XML after the initial 200 response. (pkg/s3compat/handler.go, pkg/s3compat/handler_coverage_test.go, pkg/s3compat/multipart.go)
Expired Object Lock retentions can now be cleaned up — CleanupExpiredRetentions attempted to clear expired retention by setting it to nil, but SetObjectRetention treated any nil update as shortening retention even when the retention date was already in the past. Expired retentions are now removable, while active COMPLIANCE/GOVERNANCE retentions still reject shortening or deletion. (internal/object/manager.go, internal/object/manager_low_coverage_test.go)
Filesystem storage path validation now rejects Windows separator collisions without rejecting valid dotted object names — the backend previously rejected any key containing .., blocking valid S3 object names such as file..txt, while still accepting backslashes on Windows. Backslash keys can collide with slash-separated keys on disk (a\b vs a/b) and corrupt stored objects. Validation now rejects native Windows separators and drive-qualified paths, but only treats an exact .. path segment as traversal. (internal/storage/filesystem.go, internal/storage/filesystem_test.go)
Console access revocation now takes effect for existing access and refresh tokens — console:access was checked during login only, so a token issued before an override deny remained usable until expiry and refresh tokens could mint new sessions. The console JWT middleware and refresh endpoint now re-check console:access on every authenticated request/refresh. (internal/server/console_api.go, internal/server/console_access_test.go)
Lifecycle cleanup now sees hidden versioned keys whose latest state is a delete marker — noncurrent-version expiration and expired-delete-marker cleanup used ListObjects, which lists visible objects and can miss keys hidden by a latest delete marker. Those lifecycle paths now scan ListAllObjectVersions from metadata and group by key, so noncurrent versions and delete-marker-only keys are eligible for cleanup. (internal/lifecycle/worker.go, internal/lifecycle/worker_test.go)
Metadata tag searches no longer return stale results after PutObject overwrites an object's tags — Pebble created new tag_idx entries when storing an object but did not remove the previous tag indices for the same bucket/key. Replacing an object with a different tag set could therefore make ListObjectsByTags and tag-filtered SearchObjects return objects that no longer matched the requested tags. Fixed by deleting existing tag indices atomically before storing the overwritten object and creating the replacement indices. BadgerDB references from the original fix were removed because Pebble is now the only metadata engine. (internal/metadata/pebble_objects.go, internal/metadata/store_consistency_test.go)
Version metadata lookups no longer bleed between keys where one key is a colon-prefixed extension of another — version entries are stored under version:{bucket}:{key}:{versionID}. A key such as a:b shares the prefix version:bucket:a: used to list versions for key a, so GetObjectVersions(bucket, "a") could include versions belonging to a:b, and adding a new latest version for a could mark versions of a:b as non-latest. Fixed by filtering version-prefix scans against the stored object key in Pebble. BadgerDB references from the original fix were removed because Pebble is now the only metadata engine. (internal/metadata/pebble_objects.go, internal/metadata/store_consistency_test.go)
Versioned bucket visibility and storage metrics now stay consistent across delete markers, PUT, and multipart completion — creating a delete marker always decremented the bucket object count, even when the latest version was already a delete marker, so repeated DELETE requests could drive visible object counts down incorrectly. A later PUT over a latest delete marker added bytes as an extra hidden version but did not restore the visible object count. CompleteMultipartUpload also ignored bucket versioning entirely, writing the completed object to the unversioned path with no VersionID and calculating quota/metrics as an overwrite instead of a new stored version. Fixed by detecting delete markers explicitly, changing metrics only on real visibility transitions, generating versioned paths/version IDs for multipart completion in versioned buckets, and charging versioned multipart uploads as additional stored versions. (internal/object/manager.go, internal/object/versioning_multipart_metrics_test.go)
CopyObject and UploadPartCopy now parse x-amz-copy-source without corrupting keys that contain encoded query characters — the copy-source parser decoded the object key before separating the optional ?versionId=... query. A valid source key such as reports/a%3FversionId=v1.txt was decoded to reports/a?versionId=v1.txt and then incorrectly split as if versionId were a source version query, causing the wrong key to be copied. Fixed by parsing the raw query portion before URL-decoding the key and sharing that parser between CopyObject and UploadPartCopy. (pkg/s3compat/object_ops.go, pkg/s3compat/multipart.go, pkg/s3compat/copy_source_test.go)
S3 Select JSON Lines input no longer silently drops malformed records — loadJSONLines stopped reading on the first JSON decode error and returned the rows parsed so far, which could produce successful but incomplete query results for corrupt objects. Fixed by decoding until io.EOF and returning an InvalidRequest path on malformed JSON. The handler also now rejects unsupported JSON input types such as DOCUMENT instead of treating them as JSON Lines. (pkg/s3compat/select.go, pkg/s3compat/select_test.go)
Multipart listing pagination now reports truncation and markers correctly — ListMultipartUploads marked IsTruncated=true as soon as it returned max-uploads entries, even when there were no more uploads, and treated key-marker as inclusive when upload-id-marker was absent, allowing duplicate uploads across pages. ListParts calculated truncation with len(parts) > len(filteredParts)+partNumberMarker, incorrectly treating the numeric part marker as an array count; sparse part numbers such as 100, 200, 300 could return IsTruncated=false while more parts were still available. Fixed both paths by sorting deterministically, applying exclusive marker semantics, and detecting truncation with a real lookahead. (pkg/s3compat/multipart.go, pkg/s3compat/multipart_pagination_test.go)
Inventory XML rendering no longer panics on empty or malformed frequency values — internalToXML formatted cfg.Frequency with freqLower[:1], which crashes if a stored configuration ever reaches the handler with an empty frequency. That turns a bad record into a handler panic instead of a recoverable response. Fixed by treating an empty frequency as an empty XML field instead of slicing the string blindly. (pkg/s3compat/inventory.go, pkg/s3compat/inventory_test.go)
Presigned URL generation and validation now handle object keys with spaces and reserved characters correctly — GeneratePresignedURLV2 and GeneratePresignedURLV4 were emitting raw bucket/objectKey paths, and ValidatePresignedURLV2/V4 were signing against r.URL.Path directly. That breaks presigned access for keys containing spaces, +, or other reserved bytes because the generated URL is not RFC 3986-safe and the reconstructed canonical path no longer matches what the client sends. Fixed by applying the same S3-style path encoding on both generation and validation, preserving / as a path separator while encoding each unsafe byte. Added regression tests for keys like folder name/a+b c.txt. (pkg/s3compat/presigned.go, pkg/s3compat/presigned_test.go)
Inventory report generation now preserves the current bucket namespace model instead of collapsing tenant-scoped and global buckets together — the inventory generator previously listed source objects using only config.BucketName and uploaded the generated report using destinationBucket/reportPath, ignoring config.TenantID even when the bucket was tenant-scoped. That made inventory generation read from or write to the wrong logical bucket namespace in multi-tenant deployments. Fixed by deriving the internal bucket path as tenantID/bucket only when tenantID is non-empty, while leaving global buckets unscoped as plain bucket. Added regression tests covering both tenant-scoped and global buckets so the existing global-bucket contract stays intact. (internal/inventory/generator.go, internal/inventory/generator_test.go, internal/inventory/worker_test.go)
Inventory report uploads no longer leave partial state when storage and metadata commits diverge — uploadReport persisted report metadata before uploading the actual report object. If the storage write failed, the metadata store could still expose an inventory report whose payload did not exist. The write order is now reversed: the report is uploaded to storage first, then metadata is committed. If metadata persistence fails after the object upload, the uploaded object is deleted as compensation so no orphaned report object remains behind. Tests were updated to assert the new commit order and cleanup path. (internal/inventory/generator.go, internal/inventory/generator_test.go)
Replication queue items are now claimed before enqueueing, preventing duplicate remote replication of the same object — the queue loader periodically selected rows with status='pending' and pushed them into the in-memory worker channel without first marking them as claimed in SQLite. If the loader ticked again before a worker changed the row to in_progress, the same item could be enqueued multiple times, resulting in duplicate PUT/DELETE operations against the destination. Fixed by atomically claiming eligible rows with status='retrying' before enqueueing, releasing the claim if the in-memory queue is full, and resetting stale retrying/in_progress rows back to pending on manager start. (internal/replication/manager.go, internal/replication/manager_test.go)
Notification webhook delivery now closes response bodies per attempt and rejects empty DNS resolutions cleanly — the webhook sender deferred resp.Body.Close() inside the retry loop, so repeated failed attempts held bodies open until the function returned. The SSRF-protected dialer also assumed LookupHost always returned at least one address, which could panic or mis-handle an edge case where DNS returned an empty set without an error. Fixed by closing each response body immediately after the attempt and treating an empty resolution result as an explicit validation error. (internal/notifications/manager.go)
All user capabilities appeared as unchecked in the Manage Permissions modal regardless of role — EffectiveCapability in internal/auth/capabilities.go declared the granted field as Allowed bool with JSON tag "allowed", but the frontend TypeScript interface expected the field name "granted". Every capability in the API response arrived with the wrong key, so cap.granted was always undefined (falsy) and every toggle rendered in the off/denied state. Fixed by renaming the struct field to Granted with JSON tag "granted". This also affected users who had been granted capabilities via role defaults: after upgrade all their capabilities appeared disabled until the fix was applied. (internal/auth/capabilities.go, web/frontend/src/types/index.ts)
Legacy cluster object migration lost tenant context, breaking tenant-scoped bucket replication and deletion — the legacy inter-node object endpoints (PUT/DELETE /api/internal/cluster/objects/{tenantID}/{bucket}/{key}) extracted tenantID from the route but then called objectManager.PutObject and DeleteObject with only the bare bucket name. objectManager.parseBucketPath() interprets a bare bucket as a global bucket, so tenant-scoped objects copied through the legacy migration path were written, verified, or deleted against the wrong logical bucket namespace on the destination node. Fixed by reconstructing the full bucket path as tenantID + "/" + bucket before calling the object manager. The HA fanout path already used the correct full bucket path and was not affected. (internal/server/cluster_object_handlers.go)
Legacy cluster migration could not handle object keys containing / — the dedicated cluster router registered the legacy object endpoints as /objects/{tenantID}/{bucket}/{key}, which only captures a single path segment for {key}. Bucket migration and verification build URLs using the full S3 object key, so any object such as dir/a.txt or nested/path/file.bin produced a route mismatch on the destination node. Fixed by changing the route pattern to {key:.*} for PUT, DELETE, and the new verification HEAD endpoint, preserving the full S3 key including embedded slashes. (internal/server/server.go)
verifyData=true bucket migration failed because the internal object endpoint did not support HEAD — migration verification sampled objects and issued HEAD /api/internal/cluster/objects/{tenantID}/{bucket}/{key} expecting a 200 with X-Object-ETag, but the cluster router only exposed PUT and DELETE on that path. As a result, migrations with verification enabled received 405 Method Not Allowed during the verification phase and were marked failed even when the object had actually been copied correctly. Fixed by adding handleHeadReplicatedObject, which returns object existence plus ETag, Content-Type, Content-Length, and Last-Modified for the replicated object. (internal/server/cluster_object_handlers.go, internal/server/server.go, internal/cluster/migration.go)
Custom object metadata was corrupted during legacy bucket migration — the migration sender serialized object metadata into the X-Object-Metadata header as JSON, but the receiver stored that JSON blob under a synthetic X-Amz-Meta-Original header. objectManager.extractMetadataFromHeaders() only persists real x-amz-meta-* headers, so the destination object ended up with a single incorrect metadata key (original) containing the raw JSON string instead of the original user metadata map. Fixed by parsing the JSON payload on the receiving node and re-emitting each key/value pair as a real x-amz-meta-{key} header before calling PutObject. (internal/server/cluster_object_handlers.go, internal/cluster/migration.go, internal/object/manager.go)
Legacy migration URLs did not escape tenant or bucket path segments — the sender built internal migration URLs using fmt.Sprintf(.../%s/%s/%s, tenantID, bucket, key) without url.PathEscape on tenantID and bucket. While bucket names are normally DNS-safe, the tenant segment is an internal identifier and the route contract should still be encoded consistently; leaving it raw also made the sender/receiver contract brittle if any future internal naming scheme introduced reserved URL characters. Fixed by path-escaping the tenant and bucket segments when constructing both copy and verification URLs. (internal/cluster/migration.go)
GET /api/v1/security/status returned synthetic security data instead of the real server state — the endpoint hardcoded activeSessions: 0, failedLogins24h: 0, audit.enabled: false, and a fake lastRotation timestamp 30 days in the past, even on installations with audit logging enabled and recent failed logins recorded in the user table. This made the security dashboard actively misleading for operators. Fixed by deriving mfaEnabled and failedLogins24h from the actual user set, reading audit enablement from config.Audit plus auditManager, querying real audit event totals (totalEvents and eventsToday), and marking values that are still not tracked (activeSessions, userPolicies, key rotation history) as untracked instead of fabricating numbers. (internal/server/console_api.go)
Filesystem object writes published metadata before the object file was atomically committed — FilesystemBackend.Put() wrote {object}.metadata first and only then renamed the temporary data file into place. If the process crashed or the final rename failed, the system could be left with visible metadata for an object whose payload file did not exist, creating a false-positive object record and confusing integrity checks or manual debugging. Fixed by preparing the metadata in a temporary file, performing the object rename first, and only then atomically renaming the metadata temp file into its final location. (internal/storage/filesystem.go)
Filesystem metadata writes were non-atomic and could leave partial or orphaned .metadata files — saveMetadata() used os.WriteFile() directly on the final metadata path. A crash or interrupted write could truncate the metadata file in place, and a failed Put() path could leave partially prepared metadata visible to subsequent reads. Fixed by rewriting metadata persistence to use CreateTemp + write + close + os.Rename, so both standalone metadata updates and write-path metadata commits become atomic at the filesystem level. Added a regression test that forces a final object rename failure and verifies no stray .metadata file remains behind. (internal/storage/filesystem.go, internal/storage/filesystem_test.go)

[1.3.0] - 2026-04-21¶

Added¶

Dedicated cluster inter-node port 8082 — all cluster coordination routes (token validation, CSR signing, node registration, entity sync, JWT secret exchange) now run on a dedicated HTTP server bound to :8082 (configurable via cluster_listen). Previously these routes were multiplexed with the S3 API on port 8080, which caused routing conflicts, prevented independent firewall rules, and meant that any node that had not yet initialized a cluster (no IsClusterEnabled() at startup) could not participate in a join flow because the routes did not exist yet. Separating the concerns gives operators a clear boundary: port 8080 is S3 client traffic, port 8081 is the web console, and port 8082 is internal cluster coordination — never exposed to end users. (internal/server/server.go, internal/config/config.go)
New config option cluster_listen — controls the bind address for the cluster inter-node server. Default: :8082. Set to 127.0.0.1:8082 to restrict inter-node communication to the loopback interface (single-machine test clusters). (internal/config/config.go, docs/CONFIGURATION.md)
Replication allow_internal_endpoints config option — new replication.allow_internal_endpoints YAML setting disables SSRF protection for replication destination endpoints, allowing connections to private/internal IP ranges (10.x, 172.16.x, 192.168.x, 127.x). Required when replicating between MaxIOFS instances on the same Kubernetes cluster, Docker network, or LAN where endpoints resolve to private IPs. When false (default), the existing SSRF-blocking dialer rejects connections to RFC 1918 addresses and the cloud metadata endpoint (169.254.169.254). (internal/config/config.go, internal/replication/types.go, internal/replication/s3client.go, internal/replication/manager.go, internal/server/server.go, config.example.yaml)
Realtime replication mode now functional — mode=realtime replication rules are now triggered automatically on every S3 PutObject and DeleteObject operation. Previously the realtime mode was accepted in the UI and stored in the database, but no hook existed in the S3 handlers to enqueue objects — replication only worked via manual sync or the scheduled mode timer. Added QueueRealtimeObject() to the replication manager (filters matching rules by mode=realtime only) and async hooks in the S3 PutObject and DeleteObject handlers that call it after the response is sent. The scheduled and batch modes are unaffected — the existing ruleScheduler continues to process those independently. (internal/replication/manager.go, pkg/s3compat/handler.go, internal/api/handler.go, internal/server/server.go)
HA write quorum — writes now block until a quorum of replicas acknowledge — previously fanoutPut and fanoutDelete in HAObjectManager fired replica requests in background goroutines and returned immediately, meaning the client received HTTP 200 after only the local write completed. If the local node died before the background fanout finished, data was silently lost with no indication to the client. Rewritten as a synchronous quorum protocol: the quorum threshold is ceil(factor/2) confirmations (the local write counts as 1, so ceil(factor/2)-1 replica acknowledgements are required). New Manager.ClusterCanAcceptWrites(ctx) pre-checks that enough healthy non-local nodes are available before the local write begins, saving a write-then-rollback cycle when the cluster is already degraded. If quorum is not reached after all replica goroutines complete, collectAndCheckQuorum returns the new cluster.ErrClusterDegraded error: the S3 handler maps it to HTTP 503 with Retry-After: 30 on PutObject, DeleteObject, and CompleteMultipartUpload. On a PUT or CompleteMultipart quorum failure the local write is immediately rolled back via Manager.DeleteObject under a WithHARollbackContext(ctx) so the rollback itself is not fanned out to replicas. DELETE quorum failures are not rolled back — a delete is a tombstone and the anti-entropy scrubber (see below) will reconcile the missing tombstone on lagging replicas; the client receives 503 and is expected to retry. factor=2 special-case: needed replica confirmations = 0, preserving best-effort second-copy semantics. Strict two-copy guarantees require factor=3. (internal/cluster/ha_object_manager.go, internal/cluster/manager.go, pkg/s3compat/handler.go, pkg/s3compat/multipart.go; tests in internal/cluster/ha_quorum_test.go)
HA read fallback with ordered retry across replicas — GetObject previously selected a single read node via SelectReadNode and surfaced the error directly to the client if that node was unavailable or returned 404. In an HA cluster this caused spurious 404s and connection errors even when the requested object existed on other replicas. Replaced with Manager.SelectReadNodes, which returns an ordered list of all ready replicas sorted by latency_ms ascending → priority ascending → name, then rotated by an atomic readCounter % N to preserve round-robin load distribution while still providing a deterministic per-request retry path. New Manager.TryProxyRead(ctx, w, r, node) peeks the replica's HTTP status before committing anything to the response writer: 2xx/3xx/non-404 client errors (401/403/412/416) are streamed to the client and short-circuit the retry loop; 404 closes the response and tries the next candidate (object not yet synced on this replica); 5xx or transport failure closes the response, tries the next candidate, and flips the failing node to Unavailable via markNodeUnavailable. The GetObject S3 handler iterates the candidate list and falls through to a local read on a full miss. Mid-stream failures after a 200 response has been started surface as truncated bodies — those are unavoidable once bytes are committed. The old SelectReadNode is kept as a deprecated thin wrapper for other callers. (internal/cluster/manager.go, pkg/s3compat/handler.go; tests in internal/cluster/ha_read_test.go)
Anti-entropy scrubber — continuous background reconciliation across all HA replicas — added AntiEntropyScrubber in internal/cluster/anti_entropy.go. One goroutine per node runs a scheduler with a jittered 5–60 minute first-run delay then a configurable cycle interval (ha.scrub_interval_hours, default 24 h). Each cycle iterates all buckets in randomized order (so the full keyspace is covered within one interval rather than alphabetically-biased detection) and processes objects in batches of ha.scrub_batch_size (default 500). Per batch, the scrubber calls POST /api/internal/ha/checksum-batch on every healthy peer and reconciles divergences using Last-Writer-Wins (LWW): a peer missing the object or with an older ETag receives a push via the existing PUT /api/internal/ha/objects/{key} endpoint under WithHAReplicaContext; a peer with a newer ETag triggers a pull (GET from peer, local PUT under replica context). Multipart objects (ETag contains <md5>-N) skip ETag comparison and rely on existence + size + 1 s mtime tolerance instead of expensive whole-file recompute. Identical timestamps with different ETags are logged for manual triage and not auto-fixed. The scrubber is crash-safe: after every batch a JSON checkpoint (bucket_idx, last_key) is written to Pebble at ha:scrub:checkpoint so a restart resumes the same cycle from where it left off. Each run is recorded in the new ha_scrub_runs SQLite table (last 30 runs, pruned on completion). Throttled to ha.scrub_rate_limit object comparisons per second (default 50) via time.Sleep between objects. Four new live-reloadable global config keys: ha.scrub_enabled, ha.scrub_interval_hours, ha.scrub_rate_limit, ha.scrub_batch_size. Status exposed via GET /cluster/ha/scrub-status (global admin only): last 10 runs plus in-progress checkpoint snapshot. (internal/cluster/anti_entropy.go, internal/cluster/sync_schema.go, internal/server/cluster_object_handlers.go, internal/server/cluster_ha_handlers.go, internal/server/server.go, internal/server/console_api.go; tests in internal/cluster/anti_entropy_test.go)
Dead-node redistribution — automatic removal of permanently unavailable nodes from write paths — introduced HealthStatusDead as a new terminal health state alongside unknown / healthy / unavailable. A new unavailable_since TIMESTAMP column (applied via idempotent applyDeadNodeMigration) records the start of a continuous outage; markNodeUnavailable and CheckNodeHealth use COALESCE(unavailable_since, ?) so the timestamp survives repeated failed probes and is cleared on the first healthy transition. Once the outage exceeds ha.dead_node_threshold_hours (default 24 h, live-reloadable), the new DeadNodeReconciler in internal/cluster/dead_node_reconciler.go flips the node to dead and calls HASyncWorker.Trigger() to start a catch-up sync on remaining nodes — because every healthy node already holds every bucket in N-way replication, no per-bucket replica reassignment is needed. The reconciler loop runs every ha.redistribution_check_interval_minutes (default 5 min) with a 30 s jittered first pass and reloads the interval on config change; kill switch via ha.redistribution_enabled. Last-survivor protection: if marking a node dead would drop the non-dead node count below the replication factor, the reconciler refuses and writes a human-readable reason into ha.cluster_degraded_reason, which is exposed via GET /cluster/ha/degraded-state and broadcast over SSE as cluster_degraded / cluster_degraded_resolved events. Admin drain shortcut: POST /cluster/nodes/{id}/drain accepts an optional {"reason"} body and immediately flips a node to dead, bypassing the threshold timer. The drain endpoint rejects requests targeting the local node (the responding server cannot safely flip itself to dead mid-call) and nodes already in the dead state. The SSE bridge lives in internal/server/dead_node_events.go, which decouples the cluster package from the server via an EventEmitter callback registered at startup. (internal/cluster/types.go, internal/cluster/schema.go, internal/cluster/sync_schema.go, internal/cluster/manager.go, internal/cluster/health.go, internal/cluster/dead_node_reconciler.go, internal/server/dead_node_events.go, internal/server/cluster_ha_handlers.go, internal/server/server.go, internal/server/console_api.go; tests in internal/cluster/dead_node_reconciler_test.go)
Storage-pressure health state — graceful degradation under high disk usage — added HealthStatusStoragePressure as a node-level health state between healthy and degraded. The existing health checker already reads capacity_total and capacity_used from disk on each probe; it now computes usage% = used/total*100 and transitions a node to storage_pressure when usage exceeds ha.storage_pressure_threshold_percent (default 90 %). The node returns to healthy only when usage drops below ha.storage_pressure_release_percent (default 85 %), providing hysteresis that prevents rapid oscillation. A misconfiguration where release ≥ threshold is auto-clamped to threshold - 5 so the hysteresis band is never zero. Nodes in storage_pressure are excluded from write target selection (GetHealthyNodes uses a strict = healthy filter) but remain eligible for reads (GetReadyReplicaNodes accepts IN (healthy, storage_pressure)) because they still hold valid data. HealthStatusStoragePressure never overrides dead, unavailable, or degraded — the transition only fires for reachable low-latency nodes. Two new live-reloadable global config keys: ha.storage_pressure_threshold_percent and ha.storage_pressure_release_percent. SSE events node_storage_pressure (carries usage_percent + threshold_percent) and node_storage_pressure_resolved are emitted as one-shots on state transitions via cluster.StoragePressureEmitter callback and the new SSE bridge in internal/server/storage_pressure_events.go. (internal/cluster/types.go, internal/cluster/sync_schema.go, internal/cluster/manager.go, internal/cluster/health.go, internal/server/storage_pressure_events.go, internal/server/server.go; tests in internal/cluster/storage_pressure_test.go)
Stale-node reconciler — safe catch-up when a partitioned or offline node rejoins — added StaleReconciler in internal/cluster/stale_reconciler.go. Runs once at node startup when is_stale = true is detected in cluster_nodes. Detects one of two modes based on last_local_write_at: ModeOffline (node was unreachable but did not accept writes — only tombstones need immediate propagation) or ModePartition (node was isolated but continued serving writes — a full LWW push of locally-newer entities is performed before the stale flag is cleared). In ModePartition the reconciler builds a local StateSnapshot (entity IDs + updated_at timestamps for tenants, users, access keys, bucket permissions, IDP providers, group mappings) and fetches the corresponding snapshot from each peer via GET /api/internal/cluster/state-snapshot, then pushes any entity where the local timestamp is strictly greater. Both modes perform bidirectional tombstone sync: local tombstones absent or newer on the peer are pushed via the existing deletion-log-sync endpoint; remote tombstones absent or newer locally are applied with LWW (EntityIsNewerThanTombstone prevents overwriting a local entity that was re-created after the deletion). After all entity reconciliation completes, the stale flag is cleared. When the node was previously a ready HA replica (a ha_sync_jobs record with status=done exists), a further HA object delta sync runs: the reconciler contacts any healthy peer via GET /api/internal/ha/objects/changed-since (paginated by bucket), then pulls each changed key with GET /api/internal/ha/objects/{key} and writes it locally under WithHAReplicaContext, creating a tracked ha_sync_jobs entry for progress visibility. A NewDynamicProxyClient is used so TLS config changes after construction (e.g. a cluster join after the reconciler was created) are picked up automatically on each request. (internal/cluster/stale_reconciler.go, internal/cluster/proxy.go, internal/cluster/manager.go, internal/server/server.go)
Inter-node S3 proxy with HMAC authentication — added a dedicated S3 proxy layer in internal/cluster/proxy.go that handles forwarding S3 client requests to the node that owns the requested bucket or object. ProxyClient was extended with NewDynamicProxyClient(getTLS func() *tls.Config): unlike the original NewProxyClient (which captured TLS config once at construction), the dynamic variant re-evaluates the getter on each request and transparently rebuilds the underlying http.Client when the TLS config pointer changes — critical for sync managers and the stale reconciler that outlive cluster join. Two new functions handle inter-node S3 forwarding: ProxyToNodeAPIURL derives the target URL from node.Endpoint (always an internal IP, avoiding public-cert issues) via buildInternalS3URL, copies headers from the original request, and replaces the client's Authorization header with cluster HMAC auth; AddClusterProxyHeaders computes HMAC-SHA256(clusterToken, "proxy:{nodeID}:{timestamp}") and adds X-MaxIOFS-Node-Hmac + forwarded user context (X-MaxIOFS-Forwarded-User, X-MaxIOFS-Forwarded-Tenant, X-MaxIOFS-Forwarded-Roles). The receiving node validates the proxy auth via ValidateClusterProxyAuth, which checks the X-MaxIOFS-Proxied: true header, recomputes the HMAC, and enforces a ±5-minute timestamp window to prevent replay attacks. Loop prevention is enforced at ProxyClient.ProxyRequest: if the incoming request already carries X-MaxIOFS-Proxied: true, the proxy immediately returns an error instead of forwarding, preventing infinite A→B→A proxy chains. buildInternalS3URL extracts the IP from node.Endpoint (cluster port) and derives the S3 port from node.APIURL's explicit port if set (skipping defaults 80/443), falling back to 8080 — ensuring inter-node S3 traffic always travels over internal IPs regardless of the configured public API URL. (internal/cluster/proxy.go, internal/cluster/stale_reconciler.go, internal/cluster/router.go, all six *_sync.go managers, internal/server/console_api.go; tests in internal/cluster/proxy_test.go)

Changed¶

Frontend dependency toolchain upgraded to current releases — all frontend runtime, test, lint, and build packages were brought up to date, including the major compiler/tooling jumps to Vite 8, @vitejs/plugin-react 6, ESLint 10, and @eslint/js 10. The Vite production build now uses Oxc minification to match the new Vite 8 toolchain and avoid relying on the old esbuild minifier path. (web/frontend/package.json, web/frontend/package-lock.json, web/frontend/vite.config.ts, web/frontend/eslint.config.js)
Add Node and Join Cluster forms accept plain IP addresses — previously both cluster setup forms required a full URL (e.g. https://node2.example.com:8081). Both now accept a plain IP address (192.168.1.10) or IP:port pair (192.168.1.10:9000). The backend resolves the input to a full scheme://host:port URL automatically, applying default ports: 8081 for Add Node (console contact to reach the remote node's join API) and 8082 for Join Cluster (cluster port of the node being joined). A user-provided port always overrides the default. (internal/server/cluster_handlers.go, web/frontend/src/pages/cluster/Nodes.tsx, web/frontend/src/pages/cluster/Overview.tsx)

Fixed¶

Cluster sync tables never created — all inter-node synchronization permanently disabled — server.New() called cluster.InitSchema(db) to create the cluster membership tables but never called cluster.InitReplicationSchema(db), which is responsible for creating the sync-tracking tables (cluster_entity_sync, cluster_sync_state, etc.) and the cluster_global_config table with default auto-sync flags (auto_users_sync_enabled=true, auto_access_keys_sync_enabled=true, auto_tenants_sync_enabled=true). Every sync manager's main loop queries cluster_global_config at startup to determine whether synchronization is enabled; because the table did not exist, the query returned a SQL error that was silently treated as "disabled" and the manager returned immediately on every tick. As a result, access keys, users, tenants, bucket permissions, IDP providers, and group mappings were never propagated between cluster nodes in any production deployment. (internal/server/server.go, internal/cluster/sync_schema.go)
Remote nodes always showed 0 GB disk capacity in the cluster node list — handleAPIHealth on the cluster inter-node port wrapped its response in the APIResponse envelope ({"success":true,"data":{"status":"healthy"}}). The health checker that parses remote node responses reads capacity_total and capacity_used as top-level flat JSON fields; because both fields were nested inside the data object they were never resolved and always defaulted to 0, making every remote node appear to have no available disk space in the console. Fixed by rewriting handleAPIHealth to emit a flat JSON object containing status, capacity_total, and capacity_used directly from s.systemMetrics.GetDiskUsage(). (internal/server/console_api.go)
Inter-node sync requests rejected with TLS certificate errors — each sync manager (AccessKeySync, UserSync, TenantSync, BucketPermissionSync, IDPProviderSync, GroupMappingSync) created its proxyClient (an http.Client with the cluster TLS configuration) inside server.New(), before cluster initialization had run at runtime. At that point the cluster CA certificate had not yet been loaded, so the already-built clients had no trusted CA and all subsequent HTTPS inter-node sync requests were rejected with TLS verification errors. Fixed by refreshing m.proxyClient = NewProxyClient(m.clusterManager.GetTLSConfig()) at the start of each manager's Start() method, after cluster TLS has been fully initialized. (internal/cluster/access_key_sync.go, internal/cluster/user_sync.go, internal/cluster/tenant_sync.go, internal/cluster/bucket_permission_sync.go, internal/cluster/idp_provider_sync.go, internal/cluster/group_mapping_sync.go)
Access keys, users, and tenants unavailable on newly-joined nodes for up to 30 seconds — after handleAddClusterNode registered a new node in the database its health_status was unknown. GetHealthyNodes(), called by all sync manager loops to obtain the list of peers, filters exclusively for health_status='healthy' rows. The new node was therefore excluded from every sync iteration until the health-check goroutine ran (30-second interval), validated the node, and updated its status. During this window any S3 client that connected to the new node received InvalidAccessKeyId for every request. Fixed by adding kickstartNewNodeSync(): immediately after the join sequence completes a goroutine calls CheckNodeHealth to mark the node healthy, then calls tenantSyncMgr.SyncToNode, userSyncMgr.SyncToNode, and accessKeySyncMgr.SyncToNode in order to push all existing data to the new node without waiting for the periodic loops. (internal/server/cluster_handlers.go, internal/cluster/user_sync.go, internal/cluster/access_key_sync.go, internal/cluster/tenant_sync.go)
Cluster join permanently broken: primary node sent its console URL instead of S3 API URL to joining nodes — handleAddClusterNode built the node_endpoint payload from s.config.PublicConsoleURL (port 8081) instead of s.config.PublicAPIURL (port 8080). The joining node stored this as the primary's endpoint and used it for all subsequent inter-node calls (/api/internal/cluster/validate-token, /api/internal/cluster/register-node, /api/internal/cluster/sign-csr, /api/internal/cluster/nodes, /api/internal/cluster/jwt-secret). All of these routes are registered exclusively on the S3 API port; sending them to the console port returned 404 on every call, making cluster join always fail in production. Only worked in tests because both ports were localhost and tests happened to hit the right server. Fixed by moving all cluster routes to the dedicated cluster port 8082. (internal/server/cluster_handlers.go, internal/server/server.go)
Inter-node TLS verification always failed in real multi-node deployments — GenerateNodeCert and GenerateKeyAndCSR hardcoded only localhost and 127.0.0.1 in the certificate Subject Alternative Names. When nodes communicated via real IP addresses or hostnames (e.g. 192.168.1.10, node2.internal), TLS verification rejected the connection because the address did not match any SAN in the peer certificate. This silently broke all HTTPS inter-node traffic outside of single-machine test setups. Fixed by adding an endpointURL parameter to both functions; the hostname or IP is extracted from the URL and appended to the SANs alongside the existing localhost entries. All call sites in manager.go now pass m.publicAPIURL. (internal/cluster/tls.go, internal/cluster/manager.go)
Cluster inter-node communication and direct console access broken after configuring public_console_url — setupConsoleRoutes extracted the path component from public_console_url and mounted all console routes (API and SPA) under that subpath. Because cluster nodes communicate with each other directly by IP and port (not through the public URL), any deployment that set public_console_url to a domain name or load-balancer address caused every inter-node API call to receive 404, effectively breaking cluster formation, node join, and all intra-cluster coordination. Direct operator access to individual nodes via IP:port was equally broken for the same reason. spaHandler.ServeHTTP compounded the problem by also returning 404 for requests whose path did not start with the configured base path. public_console_url and public_api_url define the public-facing URLs used for link generation, OAuth callbacks, and SPA <base> tag injection — they must not control which paths the server listens on; that is the sole responsibility of listen and console_listen. Fixed by always mounting all console routes at / regardless of the configured public URL. The spaHandler now strips the base-path prefix only when present (for reverse-proxy deployments that do not strip the prefix) and falls back to serving from root otherwise. (internal/server/server.go, internal/server/embed.go)
Session always expired after exactly 15 minutes regardless of user activity — the refresh token issued at login was never stored in the browser because APIClient.login read response.data.refreshToken (camelCase) while the backend sends refresh_token (snake_case). As a result the in-memory refreshToken field was always undefined, which TokenManager.setTokens silently ignored. Every subsequent mechanism that depends on the refresh token — the proactive timer that fires 2 minutes before access-token expiry, the sliding-window background refresh that fires on each API response, and the 401 retry interceptor — found no token and did nothing. The access token (default 15 min) expired, the 401 interceptor had no refresh token to exchange, and the user was logged out. Fixed by reading response.data.refresh_token ?? response.data.refreshToken (snake_case first, camelCase as fallback). (web/frontend/src/lib/api.ts)
Inactivity timeout did not account for tab close / page reload — the idle timer lived entirely in memory (useRef), so closing the browser tab or reloading the page reset the inactivity counter to zero. A user who left the tab open for 10 minutes, closed it, and reopened it an hour later would not be logged out because no timer had been running during the closure. Fixed by persisting last_activity to localStorage on every user-interaction event (mousedown, keydown, scroll, click, etc.). On mount, useAuthProvider.initializeAuth reads this timestamp and, if it is older than the 15-minute inactivity window, immediately clears auth and redirects to the login page before attempting any API call. The timestamp is cleared on explicit logout. (web/frontend/src/hooks/useIdleTimer.ts, web/frontend/src/hooks/useAuth.ts)
Inactivity timeout was set to 30 minutes instead of the documented 15 minutes — useAuthProvider passed timeout: 30 * 60 * 1000 to useIdleTimer. Changed to 15 * 60 * 1000. The constant is now extracted as IDLE_TIMEOUT_MS and shared between the idle timer and the tab-close check so both use the same value. (web/frontend/src/hooks/useAuth.ts)
Duplicate folders in S3 clients when listing large buckets — S3 clients (S3 Browser, AWS CLI, SDKs) that paginate a bucket with delimiter=/ received the same common prefix (folder name) on multiple pages. Root cause had two parts: (1) ListObjects in manager.go set nextMarker to the common prefix string (e.g. "folder_1000/") when truncating a page that ended on a folder boundary. On the next page the Pebble iterator seeked to that key and, if it existed as a folder marker or if objects inside the folder were encountered, extracted the same common prefix again. (2) When the store's own pagination nextMarker fell inside a returned common prefix (large folder with >100 000 objects across multiple 100 k-scan chunks), the same folder was emitted again on the next chunk's page. Fixed by introducing advancePastPrefix(prefix, delimiter) which computes the lexicographically-smallest key that is greater than every key starting with prefix (e.g. advancePastPrefix("folder_1000/", "/") → "folder_1001"). The function is applied in two places: when truncating by prefix count, and when the store's nextMarker falls inside any returned common prefix. (internal/object/manager.go)
Pebble ListObjects / searchObjectsByScan skipped all results when marker key was not in store — the skip-to-marker loop used objKeyStr == marker to detect when to start collecting. SeekGE positions the iterator at the first key ≥ marker; if the marker is a synthesised sentinel (e.g. "folder_1001" produced by advancePastPrefix) that does not exist as a real object key, the first key from SeekGE is already past the marker, but started never became true and every subsequent key was discarded. Fixed by promoting started = true immediately when objKeyStr > marker (SeekGE guarantees the key is ≥ marker, so > means the marker was not found and the current key is the correct start), falling through to process the key instead of continuing. The == marker branch still skips the marker itself for exclusive-lower-bound semantics. Same fix applied to searchObjectsByScan. (internal/metadata/pebble_objects.go)
handleCreateBucket panicked with nil pointer dereference in server tests — the handler called s.clusterManager.GetLocalNodeID(r.Context()) unconditionally to assign the HA primary node. In the test environment s.clusterManager is nil (cluster is not initialised for unit tests), causing a panic. Every other call site in console_api.go guards with if s.clusterManager != nil. Added the same guard around the primary-node assignment. (internal/server/console_api.go)
Bucket migration available when HA replication factor > 1 — the POST /cluster/buckets/{bucket}/migrate handler did not check the current replication factor. In HA mode every node already holds a complete copy of all data, so migrating a bucket to a specific node is meaningless and, if delete_source was set, would destroy data that is still serving live read traffic. Added a guard that reads GetReplicationFactor and returns HTTP 400 with a clear message when factor > 1. (internal/server/console_api_migration.go)
Bucket migration delete_source option was a no-op — executeMigration in migration.go contained a // TODO: Implement source deletion comment and no code. Setting delete_source: true silently succeeded without removing anything from the source node. Implemented by adding bucketManagerForMigration — a minimal interface exposing ForceDeleteBucket — to cluster.Manager alongside a SetBucketManager setter. executeMigration now calls ForceDeleteBucket(ctx, tenantID, job.BucketName) after all copy, permission, ACL, configuration, and verification steps complete. SetBucketManager is called from server.go during startup alongside the existing SetStorage call. If bucketManager is nil when delete_source: true is requested, an explicit error is returned instead of silently skipping the deletion. (internal/cluster/migration.go, internal/cluster/manager.go, internal/server/server.go)

Changed¶

Object Lock / WORM bucket creation UI simplified and Veeam guidance added — the warning banner was reduced from four bullet points to two (the mode-specific bullets were redundant with the mode descriptions below them). A new "No default retention (app-managed)" option is now the first choice in the Retention Mode selector, selected by default, with an explicit note: Use this for Veeam Backup & Replication — Veeam sets its own mode and retention directly on each object. The retention-period inputs (days / years) are hidden when this option is selected since they do not apply to app-managed retention. The COMPLIANCE and GOVERNANCE descriptions were trimmed to one sentence each. Both en and es locales updated. (web/frontend/src/pages/buckets/create.tsx, web/frontend/src/locales/en/createBucket.json, web/frontend/src/locales/es/createBucket.json)

[1.2.0] - 2026-04-02¶

Fixed¶

Background token refresh triggered immediate logout on transient failure — the handleResponse activity-based background refresh (fired after every successful API call, debounced to once per minute) called doLogout() directly when the refresh endpoint returned 401 or 403. This meant that if the refresh token expired between two user actions, the user was silently logged out the next time any API call completed successfully — even though their current access token was still valid for up to another 14 minutes. The race was particularly common when the security.session_timeout was set shorter than intended: both the access token and refresh token would share the same short TTL, expire together, and the next background refresh would immediately terminate the session. Fixed by removing doLogout() from the background refresh catch path; the 401 error interceptor (which fires only when an actual API request is rejected) remains the sole authoritative logout trigger. Additionally, setTokens no longer clears the in-memory refresh token to null when called without a second argument (preventing a silent accidental clear if a refresh response ever omits the refresh_token field), and getRefreshToken() now falls back to localStorage if the in-memory copy is missing. (web/frontend/src/lib/api.ts)
COMPLIANCE Object Lock retention did not prevent bucket deletion by global admins — the handleDeleteBucket console API handler blocked non-admin users from deleting Object Lock–enabled buckets but allowed global administrators to delete them regardless — including via the ?force=true path that calls ForceDeleteBucket, which bypasses the empty-bucket check entirely. Any user with global admin credentials could destroy WORM-protected data, defeating the guarantee offered to Veeam B&R users and similar backup workloads that rely on COMPLIANCE retention. Fixed by adding a pre-deletion scan (HasActiveComplianceRetention) that iterates all object metadata and versioned object entries for the bucket and returns true if any record has Retention.Mode == "COMPLIANCE" with a future RetainUntilDate, or has LegalHold == true. If active immutable data is found, the request is rejected with HTTP 409 Conflict for all callers — including global admins and force-delete. HasActiveComplianceRetention was added to the metadata.Store interface, its Pebble implementation, the object.Manager interface, and objectManager. The bucket list UI now disables the delete button for all users on Object Lock buckets (previously only blocked non-global-admins) with an updated tooltip explaining why. The onError handler in the bucket list no longer offers the force-delete dialog when the server responds with a COMPLIANCE error. (internal/metadata/store.go, internal/metadata/pebble_objects.go, internal/object/manager.go, internal/server/console_api.go, web/frontend/src/pages/buckets/index.tsx)
GOVERNANCE Object Lock bypass ignored when deleting a specific version — DeleteObject reads the x-amz-bypass-governance-retention: true header and passes bypassGovernance=true to objectManager.DeleteObject, but when a versionId is present the call is dispatched to deleteSpecificVersion, which did not accept the flag. The GOVERNANCE retention check inside deleteSpecificVersion always blocked the deletion regardless of the header, so --bypass-governance-retention had no effect on versioned deletes. Fixed by threading bypassGovernance bool through deleteSpecificVersion and applying the same bypass logic that deletePermanently already had. (internal/object/manager.go)
GET /{bucket}?uploads (ListMultipartUploads) was captured by ListObjects — ListMultipartUploads was registered on the bucket router after ListObjects (which has no query-parameter constraint). gorilla/mux matches routes in registration order, so every GET /{bucket}?uploads request was handled by ListObjects, which returned a ListBucketResult instead of ListMultipartUploadsResult. The AWS CLI parsed the wrong XML and showed an empty response with no Uploads array even for in-progress uploads. Fixed by moving the ListMultipartUploads route registration inside the per-path loop, immediately before ListObjects, so the more-specific query-constrained route is evaluated first. (internal/api/handler.go)
Panic pebble: closed on graceful shutdown — BucketAccessLogger.Stop() closed the done channel to signal the background goroutine to stop and flush, but returned immediately without waiting for the goroutine to finish. The shutdown sequence then closed the Pebble metadata store while the goroutine was still in flushEntries → PutObject → pebbleGet, causing a panic: pebble: closed. Fixed by adding a sync.WaitGroup (wg.Add(1) before starting the goroutine, defer wg.Done() at the top of run). Stop() now calls wg.Wait() after closing the channel, blocking until the goroutine has completed its final flush before Pebble is closed. (internal/server/bucket_logging.go)
Inventory report ETag was the content length encoded as hex, not an MD5 checksum — when the inventory generator uploaded a report object it set ETag: fmt.Sprintf("%x", len(content)), which produces the byte count in hexadecimal (e.g. "f5a" for a 3 930-byte CSV). The integrity scanner compares stored ETags against the actual MD5 of the file content, so every inventory report was flagged as corrupt on the next scrub cycle. Fixed by computing md5.Sum(content) and storing it as a plain hex string via fmt.Sprintf("%x", md5.Sum(content)). ETags are stored without surrounding double-quote characters in the metadata store (the HTTP " quoting is an on-wire convention only); a prior attempt that used fmt.Sprintf("\"%x\"", ...) stored literal " characters in Pebble, causing the integrity check to always report the object as corrupted even after the MD5 fix. (internal/inventory/generator.go)
Object deletion order left metadata intact after physical file was removed — deletePermanently deleted the physical storage file in step 1 and the metadata in step 2. If step 2 failed (or if the process crashed between the two steps), the object remained visible in metadata but its data was gone; the integrity scanner would then report it as corrupt on the next scan. Swapped the order: metadata is now deleted first, then the physical file. If the physical deletion fails the file becomes an orphan (harmless, cleaned up by the next bucket directory removal or orphan scrub), which is the safe failure direction. (internal/object/manager.go)
isBucketEmpty silently deleted orphaned physical files as a side effect — when DeleteBucket called isBucketEmpty to verify the bucket was safe to remove, the helper walked all physical files and deleted any that had no corresponding metadata entry. This caused unexpected "Found orphaned physical file without metadata - deleting" warnings in the logs before the bucket-deleted audit event, making it appear the storage layer was making unilateral deletions outside of any explicit user action. Removed the destructive side effect: isBucketEmpty now simply skips files with no metadata (treating them as absent for the empty-check). The DeleteBucket RemoveDirectory call at the end of the deletion sequence already removes the entire directory including any residual orphaned files. (internal/bucket/manager_impl.go)
Console UI allowed deleting Object Lock–enabled buckets — any authenticated user (including non-admins) could delete a bucket that had Object Lock enabled through the console API, bypassing WORM protection. Added a server-side guard in handleDeleteBucket: if the bucket has ObjectLockEnabled: true and the caller is not a global administrator the request is rejected with 403 Forbidden. The delete button in the bucket list is also disabled in the UI for Object Lock buckets for non-global-admins, with a tooltip explaining why. (internal/server/console_api.go, web/frontend/src/pages/buckets/index.tsx)
Non-admin users could not see buckets they created — when a global user (no tenant) created a bucket through the console API, handleCreateBucket left OwnerID and OwnerType empty on the new bucket. FilterBucketsByPermissions grants visibility when bucket.OwnerID == user.ID && bucket.OwnerType == "user", so the bucket was invisible to the creator immediately after creation. Fixed by assigning OwnerID = user.ID and OwnerType = "user" for global non-admin users on bucket creation. (internal/server/console_api.go)
2FA login failed with HTTP 500 after code was accepted — GetUserWith2FA scanned the tenant_id column directly into a string variable. tenant_id is NULL in SQLite for users without a tenant assignment; database/sql does not allow scanning NULL into a plain string and returns an error, which the handler mapped to a 500 response. Fixed by scanning into sql.NullString and only assigning user.TenantID when Valid is true. (internal/auth/sqlite.go)
Vite compiled frontend with absolute /assets/… import paths — vite.config.ts had base: '/ui/' hardcoded, so dynamic import() chunks were compiled with /ui/assets/chunk-xxx.js absolute paths. In a deployment where the console is served under a different prefix, the browser fetched those paths from the S3 API port, which returned a MIME-type error and caused a blank page. Changed to base: './' so all asset paths are relative to the HTML document. embed.go injects a <base href="…"> tag that resolves relative imports to the correct runtime prefix regardless of the configured console path. (web/frontend/vite.config.ts)
RestoreAlreadyInProgress returned HTTP 500 instead of 409 — the error code was missing from the writeError status-code table, so the server logged it as an internal error and the AWS CLI retried the request 4 times before giving up. Added RestoreAlreadyInProgress to the 409 Conflict case. (pkg/s3compat/handler.go)
NoSuchConfiguration (inventory) returned HTTP 500 instead of 404 — same root cause: the error code was not mapped in writeError. Added NoSuchConfiguration to the 404 Not Found case so GetBucketInventoryConfiguration on a missing id returns a clean 404 instead of an internal error. (pkg/s3compat/handler.go)
RestoreObject allowed re-restoring an already-restored object — the in-progress check only tested RestoreStatus == "ongoing" but the handler sets status to "restored" (not "ongoing") on success. A second restore call within the expiry window silently succeeded with 200 instead of returning 409. Fixed by also checking RestoreStatus == "restored" && RestoreExpiresAt.After(now). (pkg/s3compat/handler.go)
Metadata store not closed on shutdown — all recent writes silently lost on restart — s.metadataStore.Close() was missing from the server Shutdown() sequence. Pebble uses pebble.NoSync for all writes (intentional for performance), meaning writes go to the WAL in memory and are only flushed to disk when db.Close() is called. Without this call, every metadata change since the last background compaction flush — object deletes, version promotions, renames, tag updates, bucket config changes — was silently discarded on process exit. On the next startup Pebble replayed only what had been physically written to disk, reverting the database to an earlier state. This caused symptoms like: deleting a file and seeing it reappear after restart, restoring a version (removing a delete marker) only for the marker to come back after restart, and renamed objects reverting to their original key. Fixed by adding s.metadataStore.Close() at the end of Shutdown(), after all workers have stopped (workers may still write metadata during their own shutdown, so order matters). (internal/server/server.go)
Session expiry while actively working in the UI — the access token used a fixed 15-minute window from the moment it was issued, with no sliding behaviour. A user who worked continuously for 15 minutes was silently logged out regardless of their activity, because the token timer was never reset by API calls. Implemented a true sliding-window session: after every successful API response a background token refresh is triggered (debounced to at most once per minute via lastActivityRefreshAt). Any action that produces an API call — navigation, bucket operations, uploads, settings changes — now resets the 15-minute window from that moment. The existing proactive refresh timer (fires 2 minutes before expiry) is kept as a safety net for idle tabs where no API calls are in flight. (web/frontend/src/lib/api.ts)
Proactive refresh silently abandoned on transient failure — if the proactive background refresh failed due to a network glitch or server hiccup, the catch block discarded the error and no retry was scheduled. The token then expired with no further recovery attempt before the 401 interceptor. The proactive refresh now retries every 30 seconds on transient errors (non-401/403 responses) for as long as the current token is still alive. Auth rejections (401/403) are not retried — those mean the refresh token itself is invalid and the next API call's 401 interceptor handles the final logout. (web/frontend/src/lib/api.ts)
Idle timer logged out user during active file upload — the idle timer tracks UI events (mouse, keyboard) to detect inactivity. During a large file upload the user is not interacting with the UI, so after 30 minutes of upload progress the idle timer fired handleIdle and redirected to login, aborting the transfer. The idle timer now checks an activeUploadCount counter before firing: if any upload is in progress the logout is postponed by a full 30-minute interval instead of executing. The counter is incremented when uploadObject starts and decremented in finally so it is always correct even on error. (web/frontend/src/lib/api.ts, web/frontend/src/hooks/useIdleTimer.ts, web/frontend/src/hooks/useAuth.ts)
Console UI allowed deleting Object Lock–protected objects — the bucket browser did not check an object's retention or Legal Hold status before calling the delete API. Objects marked as immutable by VEEAM B&R appeared locked in the UI (lock icon + retention countdown) but the delete button remained enabled. On versioned buckets the console API created a delete marker, making the object disappear from the default view while the retained version remained — giving a false impression of successful deletion. Fixed by adding an isObjectLocked helper (checks active non-expired retention and legalHold.status === 'ON'), disabling the Actions dropdown delete button when any selected item is locked, blocking handleDeleteObject before confirmation with a specific error toast, and filtering locked items out of bulk delete with a skip count shown to the user. (web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/locales/en/buckets.json, web/frontend/src/locales/es/buckets.json)
Stale "account locked" notification shown on fresh installations — notifications were persisted in browser localStorage without any session binding. Opening a new MaxIOFS deployment in a browser that had previously used another instance could show an old user_locked notification (e.g. "admin account locked 7 days ago") even though the new server had no lock state at all. Fixed by storing the current auth token alongside the notifications in localStorage. On mount, if the token does not match the one stored with the notifications, the stale notifications are discarded and replaced with an empty list for the new session. (web/frontend/src/hooks/useNotifications.ts)
cleanupEmptyDirectories never cleaned up directories — the function compared a relative object path (bucket/key) against the absolute rootPath from FilesystemBackend.GetRootPath(). strings.HasPrefix always returned false, so the cleanup loop broke on the first iteration and no empty directory was ever removed after object deletion. Fixed by constructing the full path with filepath.Join(rootPath, getObjectPath(bucket, key)) before entering the loop. (internal/object/manager.go)
Actions dropdown opened downward even when near the bottom of the viewport — the dropdown always rendered below the button, causing it to overflow off-screen when the table was scrolled near the bottom. Fixed by measuring the remaining viewport space on click and flipping to bottom-full positioning when less than 520px remain below the button. (web/frontend/src/pages/buckets/[bucket]/index.tsx)
Actions button enabled with no objects selected — the Actions toolbar button could be clicked with an empty selection, opening a dropdown with no applicable targets. Fixed by adding selectedObjects.size === 0 to the button's disabled condition. (web/frontend/src/pages/buckets/[bucket]/index.tsx)
Inventory settings always showed as disabled — the console API wraps inventory configuration responses in { "success": true, "data": { … } } but getBucketInventory in the frontend returned response.data (the outer envelope) instead of response.data.data (the actual config). As a result inventoryConfig.enabled was always undefined, which the form treated as false regardless of the stored value. Fixed by unwrapping the inner data field: return response.data?.data ?? response.data. (web/frontend/src/lib/api.ts)
MockMetadataStore, MockObjectManager, and mockManager missing HasActiveComplianceRetention — adding HasActiveComplianceRetention to the metadata.Store and object.Manager interfaces caused compilation failures in three test packages (internal/inventory, internal/api, internal/object) because their mock structs did not implement the new method. Added stub implementations returning false, nil to all three. (internal/inventory/generator_test.go, internal/api/handler_security_test.go, internal/object/lock_test.go)
Bucket encryption not reflected in dashboard — BucketWithLocation in the cluster aggregator did not have an Encryption field, so the encryption config was dropped on every list request and the dashboard always showed 0 encrypted buckets. Fixed by adding Encryption *bucket.EncryptionConfig to BucketWithLocation, populating it when converting local buckets, and forwarding it in bucketWithLocationToBucket. (internal/cluster/bucket_aggregator.go, internal/server/console_api.go)
Bucket encryption metadata not persisted on creation — when req.Encryption was nil but global encryption was enabled, the bucket was saved without encryption metadata. Fixed by falling back to {type: "AES256"} when s.config.Storage.EnableEncryption is true. (internal/server/console_api.go)
Non-admin users could not delete their own buckets — the isOwner check used camelCase fields (bucket.ownerType, bucket.ownerId) but the API returns snake_case (owner_type, owner_id), so isOwner was always false. Fixed by reading both variants with fallback. (web/frontend/src/pages/buckets/index.tsx)
Regular tenant users had buckets assigned to the tenant instead of themselves — handleCreateBucket used ownerType = "tenant" for all tenant users including non-admins, preventing them from ever satisfying the isOwner condition. Non-admin tenant users now get ownerType = "user" and ownerID = user.ID; tenant admins retain tenant ownership. (internal/server/console_api.go)

Changed¶

Pebble metadata engine upgraded from v1.1.5 to v2.1.4 — the embedded LSM-tree engine that stores all object and bucket metadata has been updated to the latest stable release. Pebble v1 and v2 use incompatible on-disk formats, so the server performs an automatic one-time migration on first start with this version:
The existing Pebble v1 database (metadata/) is opened read-only.
All key-value pairs are streamed into a temporary v2 directory (metadata_pebblev2/).
The directories are swapped atomically: metadata/ is renamed to metadata_pebblev1_backup_{timestamp}/ (preserved as a safety backup), and metadata_pebblev2/ is promoted to metadata/.
A sentinel file metadata/PEBBLE_FORMAT_V2 is written. All subsequent restarts detect it and skip the migration check immediately.

The migration is fully transparent — no manual steps, no downtime window, no data loss. The backup directory can be deleted after verifying the server operates normally. If the server is killed mid-migration, the next start automatically detects the incomplete state and either retries or completes the rename. Upgrading from BadgerDB (pre-v1.0.0-beta) is also covered: the BadgerDB migration path now writes to Pebble v2 directly and stamps the same sentinel, so the v1→v2 check is skipped for those deployments too.

Upgrade path summary: - Pre-v1.0.0-beta (BadgerDB) → first upgrade to v1.0.0-beta, which migrates to Pebble v1, then upgrade to this version, which migrates to Pebble v2. Alternatively, upgrading directly to this version also works if the BadgerDB→Pebble v2 direct path runs first. - v1.0.0-beta (Pebble v1) → upgrade directly to this version; automatic migration runs on first start. - Fresh install → Pebble v2 is created directly, no migration needed.

(internal/metadata/pebble_v1_migration.go, internal/metadata/pebble_store.go, internal/metadata/migration.go, internal/server/server.go, go.mod)

New config option: storage.metadata_cache_size_mb — controls the size of the Pebble block cache, which is the primary lever for metadata read performance. When MaxIOFS lists a folder or fetches object info, Pebble first checks this in-RAM cache. A cache hit is essentially instant (~0.1 ms); a miss requires a disk read. The cache is populated on first access and persists for the lifetime of the process.

Default is 256 MB. Recommended values by workload:

Deployment	Objects / bucket	Recommended
Small / development	< 5 000	256 MB (default)
Medium	5 000–20 000	512 MB
Veeam B&R / large buckets	20 000–100 000	1 024 MB
Very large deployments	100 000+	2 048 MB+

Configure in config.yaml:

storage:
  metadata_cache_size_mb: 1024

Or via environment variable (no restart of config file needed):

MAXIOFS_STORAGE_METADATA_CACHE_SIZE_MB=1024

Note: this is a startup-time setting. It cannot be changed at runtime via the console or API — the cache is allocated once when the database opens. Restart is required after changing it.

(internal/config/config.go, internal/metadata/pebble_store.go, internal/server/server.go, config.example.yaml, docs/CONFIGURATION.md)

Pebble pre-tuned for write-heavy workloads — several internal engine parameters were tuned to improve performance under the write-burst patterns produced by tools like Veeam Backup & Replication (tens of thousands of objects written in rapid succession):
MemTable size: 64 MB (up from the default 4 MB) — writes are buffered in RAM before being flushed to L0 SSTables. A larger MemTable absorbs longer bursts without triggering a flush, reducing write amplification and keeping the LSM tree cleaner.
MemTableStopWritesThreshold: 12 — the number of unflushed MemTables before incoming writes are stalled. Raising this gives compaction more time to catch up during sustained write loads without blocking the client.
L0StopWritesThreshold: 12 — similarly, writes are only stalled when 12 L0 files are pending compaction (default is 4). This prevents false write stalls during backup job peaks.
Compaction concurrency: 2–4 goroutines — Pebble now runs 2 to 4 background compaction goroutines simultaneously. This keeps the LSM tree compact between backup sessions, directly improving list and point-lookup latency.
Bloom filters on L1–L6 (10 bits/key) — a probabilistic structure that avoids unnecessary disk reads for point lookups where the key is absent. For example, a Veeam existence check (HEAD /{bucket}/{object}) on a key that doesn't exist would previously require scanning each LSM level on disk; with bloom filters the answer is ~99% certain in RAM. L0 deliberately has no bloom filter because range scans dominate there.
Block size: 32 KB — each SSTable block read from disk contains 32 KB of contiguous key-value data. Larger blocks are more efficient for sequential range scans (folder listings, prefix iteration) at the cost of slightly more memory per cached block.

These values are fixed in the binary. The only user-tunable parameter is the cache size described above. (internal/metadata/pebble_store.go) - Actions dropdown in bucket browser is now context-aware — previously the dropdown mixed bucket-level info (S3 URI, ARN, Endpoint URL, Website URL) with object actions in a single list regardless of selection state. Redesigned so that with no objects selected the dropdown shows only bucket information (S3 URI, ARN, Endpoint URL, Website URL) under a "Bucket Info" header, and with one or more objects selected it shows only object actions under an "Object Actions" header. All new labels internationalised in both en and es locales. (web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/locales/en/buckets.json, web/frontend/src/locales/es/buckets.json) - Bucket list search moved inline into table header — the standalone search card above the buckets table was removed. The search input is now embedded directly in the table header row alongside the bucket count, matching the layout used inside the bucket browser. Pagination row height reduced (py-4 → py-2) and page-navigation buttons made more compact. (web/frontend/src/pages/buckets/index.tsx) - CI/CD GitHub Actions updated to Node.js 24 runtime — all actions in both workflow files (main.yml, docker-publish.yml) updated to versions that declare runs.using: node24: actions/checkout v4→v6, actions/setup-go v5→v6, actions/setup-node v4→v6, actions/setup-python v5→v6, aws-actions/configure-aws-credentials v4→v6, docker/setup-qemu-action v3→v4, docker/setup-buildx-action v3→v4, docker/login-action v3→v4, docker/build-push-action v6→v7, peter-evans/dockerhub-description v4→v5. Eliminates the "Node.js 20 actions are deprecated" warning ahead of GitHub's June 2, 2026 enforcement deadline. (.github/workflows/main.yml, .github/workflows/docker-publish.yml)

Added¶

Docker multi-arch images published to DockerHub — added Dockerfile with multi-stage build (frontend → Go binary → runtime), docker-entrypoint.sh for privilege drop via runuser and first-run config.yaml generation, and two CI workflows: docker-publish.yml (triggered on version tags, builds linux/amd64 + linux/arm64, pushes latest + versioned tag, updates DockerHub description) and nightly Docker build in main.yml. Runtime image uses debian:trixie-slim with apt-get upgrade on every build to minimise CVE exposure. (Dockerfile, docker-entrypoint.sh, .github/workflows/docker-publish.yml, .github/workflows/main.yml, DOCKERHUB.md)
docker-compose.yaml uses bind mount by default — data directory mounted as ./maxiofs-data:/data so config.yaml is directly accessible on the host for editing without needing a helper container. (docker-compose.yaml)
Replication worker unit tests — worker_test.go covers processItem (PUT/COPY success, source not found, S3 error, DELETE with replicate-deletes on/off, unknown action, rule not found, rule disabled), handleError (retry path and max-retries → StatusFailed), getRule (found and not found), completeItem (status=completed, bytes_replicated set), and updateReplicationStatus (insert new record and update existing record). (internal/replication/worker_test.go)
Replication manager sync unit tests — manager_sync_test.go covers SyncBucket (success, rule not found, rule disabled, no objects, list error, prefix filter), SyncRule (success, lock contention), processScheduledRules (no rules, real-time mode skipped, overdue scheduled rule triggered, recently-synced rule skipped, stale lastSync entries cleaned up), and cleanup (expired items removed, recent completed items kept, pending items never deleted). (internal/replication/manager_sync_test.go)
Replication credential unit tests — credentials_test.go covers encryptCredential/decryptCredential round-trip, multiple value types, random nonce, empty key passthrough, empty plaintext/stored, legacy plaintext, wrong key error, corrupt base64 error, too-short data error, and deriveCredentialKey (32-byte length, determinism, different inputs → different keys). (internal/replication/credentials_test.go)
SSRF validation unit tests — s3client_test.go covers all blocked IP ranges in isBlockedIP (loopback, RFC-1918 class A/B/C, link-local/metadata 169.254.x, IPv6 ULA/link-local, unspecified) and validateReplicationEndpoint (empty, valid HTTP/HTTPS, invalid scheme, literal private IPs). (internal/replication/s3client_test.go)
Object adapter unit tests — adapter_test.go covers RealObjectAdapter.CopyObject (success and object-not-found), DeleteObject (no-op), and GetObjectMetadata (success and error). (internal/replication/adapter_test.go)
cleanupEmptyDirectories unit tests — manager_cleanup_test.go covers single empty directory removal, nested empty directory chain, stopping at a non-empty parent, .maxiofs-folder and .metadata files treated as system files, root path never removed, and non-filesystem backend returning early. (internal/object/manager_cleanup_test.go)
Object integrity unit tests — manager_integrity_test.go covers VerifyObjectIntegrity (folder marker skipped, metadata not found, delete marker skipped, multipart ETag skipped, OK match, corrupted ETag mismatch) and VerifyBucketIntegrity (empty bucket, all OK, mixed results with counts, default maxKeys, listing error, duration set, multipart objects skipped). (internal/object/manager_integrity_test.go)
RestoreObject — POST /{bucket}/{object}?restore — implements the S3 RestoreObject API used by Glacier-aware tools (Veeam, Commvault, NetBackup, AWS CLI restore-object). Parses <RestoreRequest> XML (Days, GlacierJobParameters/Tier). Since MaxIOFS has no cold storage tier, the restore completes immediately: object metadata is updated with RestoreStatus="restored" and RestoreExpiresAt = now + Days. Returns HTTP 409 RestoreAlreadyInProgress if a restore is already active. HeadObject and GetObject responses now include the x-amz-restore header (ongoing-request="false", expiry-date="...") when restore metadata is present. New metadata fields RestoreStatus and RestoreExpiresAt added to ObjectMetadata. New SetRestoreStatus() method added to object.Manager. (internal/api/handler.go, pkg/s3compat/handler.go, internal/object/manager.go, internal/object/adapter.go, internal/metadata/types.go)
OwnershipControls — GET/PUT/DELETE /{bucket}?ownershipControls — implements the S3 Bucket Ownership Controls API. AWS SDK v2 (Go, Python, Java, JS) calls GetBucketOwnershipControls automatically on bucket creation and ACL operations; without this endpoint the SDK throws OwnershipControlsNotFoundError. GET returns BucketOwnerEnforced as the default when no explicit configuration is set (matches AWS new-bucket behaviour). PUT accepts and validates Rule.ObjectOwnership values (BucketOwnerEnforced, BucketOwnerPreferred, ObjectWriter); rejects unknown values with InvalidArgument. DELETE clears the stored config so a subsequent GET returns 404 OwnershipControlsNotFoundError. Configuration persisted in bucket metadata. (internal/api/handler.go, pkg/s3compat/bucket_ops.go, internal/bucket/manager_impl.go, internal/bucket/interface.go, internal/bucket/types.go, internal/metadata/types.go)
SelectObjectContent — POST /{bucket}/{object}?select — implements the S3 Select API. Loads the object into an in-memory SQLite database and executes the SQL expression, giving full SQL support (SELECT, WHERE, GROUP BY, ORDER BY, aggregate functions) without a custom parser. Streams results back using the Amazon Event Stream binary protocol (CRC32-framed Records → Stats → End events, 1 000-row batches with http.Flusher for low latency). Supported input formats: CSV (with FileHeaderInfo USE/NONE/IGNORE and configurable FieldDelimiter) and JSON Lines. Supported output formats: CSV and JSON (per-record JSON objects preserving SELECT column order). Compression is not supported (GZIP/BZIP2 inputs return InvalidRequest). (pkg/s3compat/select.go, internal/api/handler.go)
GetObjectTorrent — GET /{bucket}/{object}?torrent — returns 501 NotImplemented. BitTorrent manifests have no practical use in self-hosted object storage; the route is now registered so clients receive a proper error instead of a 404 or silent failure. (pkg/s3compat/handler.go, internal/api/handler.go)
Transfer Acceleration (?accelerate) — documented as a no-op. GET /?accelerate returns an empty <AccelerateConfiguration/> and PUT /?accelerate accepts and discards any config. Clients that configure acceleration see no difference in behavior, which is correct for self-hosted deployments without CDN edge nodes. No code changes needed — already behaved this way.
BucketNotifications — GET/PUT/DELETE /{bucket}?notification — notification configuration was already stored but never delivered. Wired real async webhook delivery: after every PutObject, DeleteObject, CopyObject, and CompleteMultipartUpload, matching notification targets receive an HTTP POST with the S3 event payload (event name, bucket, key, ETag, size, timestamp). Event filtering by prefix/suffix and event type (wildcards supported, e.g. s3:ObjectCreated:*) is applied before dispatch. SSRF protection blocks delivery to loopback, RFC-1918, and link-local addresses. Failures are logged as warnings without retries to avoid blocking the request path. Added DELETE /{bucket}?notification route for compatibility with Terraform and SDKs (clears all configurations; AWS S3 itself uses empty-body PUT for this). (internal/server/bucket_logging.go, internal/bucket/interface.go, internal/bucket/manager_impl.go, pkg/s3compat/bucket_ops.go, internal/api/handler.go)
BucketLogging — GET/PUT /{bucket}?logging — access log configuration was already stored but the BucketAccessLogger worker was never started or connected to the request path. Wired a response-capturing middleware (s3AccessLoggingMiddleware) into the S3 router after the auth middleware so every S3 request is captured with: bucket, object key, operation name (e.g. REST.GET.OBJECT, REST.PUT.PART), remote IP, authenticated user, request ID, HTTP status, and bytes sent. The worker groups entries by source bucket, looks up the logging target, formats lines in AWS S3 access log format, and writes a single object per bucket per flush. Flush triggers: 100 entries buffered or 5-minute ticker. Buffer is drained on server shutdown. (internal/server/bucket_logging.go, internal/server/server.go)
BucketInventory — GET/PUT/DELETE /{bucket}?inventory&id=, GET /{bucket}?inventory — implements the full S3 Bucket Inventory Configuration API. Supports multiple named configurations per bucket (identified by id). PUT creates or updates a configuration: validates destination bucket, schedule frequency (Daily/Weekly), format (CSV/JSON), and optional fields; maps S3 OptionalFields names (Size, LastModifiedDate, ETag, StorageClass, IsMultipartUploaded, EncryptionStatus, ReplicationStatus, ObjectACL) to internal field constants; maps IncludedObjectVersions: All to version_id/is_latest fields. GET by id returns the configuration in S3 XML wire format. DELETE by id removes the configuration. GET without id lists all configurations for the bucket as ListInventoryConfigurationsResult. The pre-existing inventory execution engine (internal/inventory/) runs the scheduled reports; these new handlers expose its configuration through the standard S3 API so AWS CLI, Terraform, and SDK-based pipelines work without changes. (pkg/s3compat/inventory.go, internal/inventory/manager.go, internal/api/handler.go, pkg/s3compat/handler.go)
"Show versions" view in bucket browser — added a Show Versions toggle button in the bucket browser table header (visible only when versioning is enabled). When active, the regular object listing is replaced by a flat versions table showing every object version and delete marker across the bucket (or current folder prefix). Each row shows: file name, type (Version / Delete Marker), last modified, size, and a truncated Version ID. Actions per row: restore old versions with a single click (creates a new PUT making the version current), remove delete markers to recover incorrectly deleted files (delete marker is permanently removed, revealing the previous version), and delete specific old versions permanently. New backend endpoints: GET /buckets/{bucket}/versions?prefix=&maxKeys= (uses metadataStore.ListAllObjectVersions) and POST /buckets/{bucket}/objects/{key}/restore (copy-to-same-key pattern for content versions, delete-marker removal for delete markers). (internal/server/object_extra_handlers.go, internal/server/console_api.go, web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/lib/api.ts, web/frontend/src/types/index.ts)

Changed¶

Bucket creation: encryption defaults to enabled when server encryption is active — the encryption toggle in the Create Bucket form always initialised to false regardless of the server's global encryption setting. If the server had enable_encryption: true, the user had to manually enable the toggle or the bucket appeared unencrypted in the UI even though the server applied encryption anyway. The toggle now initialises to true via useEffect when serverConfig.storage.enableEncryption is true, and an informational green banner explains that server-wide AES-256-GCM encryption applies to all objects regardless of this setting. (web/frontend/src/pages/buckets/create.tsx, web/frontend/src/locales/en/createBucket.json, web/frontend/src/locales/es/createBucket.json)

[1.1.0] - 2026-03-25¶

Added¶

Folder download as ZIP — new endpoint GET /buckets/{bucket}/download-zip?prefix=folder/ streams all objects under a prefix as a ZIP archive directly to the client without buffering on the server. Objects are stored without compression (zip.Store) to minimise CPU usage. Limits enforced before streaming begins: maximum 10 000 objects and 10 GB total size; requests exceeding either limit receive a 400 Bad Request JSON error. Client disconnections are detected via request context so the stream is aborted early. The download is recorded in the audit log with object count, total size, and ZIP filename. (internal/server/download_zip_handler.go, internal/server/console_api.go)
Object rename — new endpoint POST /buckets/{bucket}/objects/{key}/rename with body { "newKey": "..." } renames an object by copying data + metadata + tags to the new key then deleting the original. Blocked for objects under active COMPLIANCE retention or Legal Hold. (internal/server/object_extra_handlers.go, internal/server/console_api.go)
Object tags get/set — new endpoints GET /buckets/{bucket}/objects/{key}/tags and PUT /buckets/{bucket}/objects/{key}/tags expose object tagging through the console API. (internal/server/object_extra_handlers.go)
Folder size endpoint — new endpoint GET /buckets/{bucket}/folder-size?prefix=folder/ returns the total size (bytes) and object count for all objects under a given prefix, paging through the listing as needed. (internal/server/object_extra_handlers.go)
Actions toolbar in bucket browser — replaced per-row action buttons with a single Actions dropdown in the table toolbar, matching AWS S3 UI conventions. The dropdown appears when one or more items are checked and offers: Copy S3 URI, Copy Object URL, Download, Download as ZIP (folders), Calculate Folder Size, Share, Presigned URL, View Versions, Legal Hold toggle, Rename, Edit Tags, and Delete. Actions that require a single selection or apply only to files/folders are disabled automatically when the selection does not match. (web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/lib/api.ts)
Object detail view — clicking a file name in the bucket browser now opens a full AWS S3-style detail page that replaces the table, showing the object's S3 URI, ARN, Object URL, Key, size, last modified, ETag, content type, storage class, region, and custom metadata. The page has three tabs: Properties (all metadata fields with copy-to-clipboard buttons), Permissions (ACL owner and grants, loaded lazily), and Versions (version history with delete markers, loaded lazily). The breadcrumb tracks the full path (Buckets > bucket > folder > file) and each segment is a clickable navigation link. The header includes a Download button and an Actions dropdown with the same operations as the table toolbar. (web/frontend/src/components/ObjectDetailsView.tsx, web/frontend/src/pages/buckets/[bucket]/index.tsx)
Bucket browser breadcrumb — replaced the "Back to Buckets" button with an inline breadcrumb (Buckets > bucket-name > folder > …) above the page title, matching AWS S3's navigation style. The page title dynamically shows the current folder name rather than always showing the bucket name. (web/frontend/src/pages/buckets/[bucket]/index.tsx)
tenantId removed from bucket browser URL — the bucket detail route no longer embeds tenantId as a URL segment; tenant context is now carried in router location.state to keep URLs clean and shareable.

Fixed¶

Inconsistent table row height in bucket browser — folder rows were rendered taller than file rows because the folder name used a native <button> element whose UA stylesheet defaults add implicit height even with p-0. Replaced with a <span role="button"> which has no UA height, matching file rows exactly. (web/frontend/src/pages/buckets/[bucket]/index.tsx)
.. (parent directory) row shorter than other rows — the parent directory row had no action buttons so its height was determined by text alone, making it shorter than item rows whose height is set by the h-9 action buttons. Added h-9 to the content wrapper div to match item row height. (web/frontend/src/pages/buckets/[bucket]/index.tsx)
Breadcrumb navigation and parent directory row in bucket browser — replaced the "Up to Parent Folder" button with an always-visible breadcrumb path below the bucket header, and added a .. row as the first table entry when inside a subfolder. Both the breadcrumb segments and the .. label link back to the correct parent prefix. (web/frontend/src/pages/buckets/[bucket]/index.tsx)
Table header background indistinguishable from row hover — TableHeader used bg-secondary and TableRow used hover:bg-secondary, which map to the same CSS variable value in both light and dark modes. Changed header to explicit bg-slate-300 dark:bg-slate-700 and row hover to hover:bg-slate-100 dark:hover:bg-slate-700/40. (web/frontend/src/components/ui/Table.tsx)
Action buttons lacked hover feedback — ghost icon buttons in the object table had no visible hover state. Each action button now has a colour-coded hover: download=blue, share=green, versions=blue, presigned URL=purple, legal hold=yellow, delete=red. (web/frontend/src/pages/buckets/[bucket]/index.tsx)
Delete from object detail view triggered back-navigation before confirmation — the "Delete object" action in ObjectDetailsView called onBack() immediately after onDelete(), returning the user to the bucket list while the SweetAlert confirmation modal was still open. If the user cancelled the deletion they would be left on the list without the object having been deleted — with no indication of what happened. Removed the premature onBack() call; the existing deleteObjectMutation.onSuccess handler already closes the detail view via detailsObjectKeyRef once the deletion is confirmed. (web/frontend/src/components/ObjectDetailsView.tsx)
tenantId not forwarded to ACL, versions, versions modal, and presigned URL modal — ObjectDetailsView made API calls to getObjectACL and listObjectVersions without tenantId, causing both requests to resolve against the global bucket namespace and return 404 for any tenant bucket. Similarly, ObjectVersionsModal and PresignedURLModal were rendered in the bucket browser without the tenantId prop. All four call sites now receive tenantId. APIClient.getObjectACL signature updated to accept an optional tenantId parameter. (web/frontend/src/components/ObjectDetailsView.tsx, web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/lib/api.ts)
Breadcrumb folder segments in object detail view all navigated to the bucket root — every intermediate folder in the breadcrumb called onBack() which closed the detail view at whatever currentPrefix was, instead of navigating to the specific prefix for that segment. For paths deeper than one level (e.g. a/b/c/file.txt) clicking a or b would take the user to a/b/c/ instead of a/ or a/b/. Added onNavigateToPrefix callback to ObjectViewCallbacks; each breadcrumb segment now computes and sets the correct prefix before calling onBack(). (web/frontend/src/components/ObjectDetailsView.tsx, web/frontend/src/pages/buckets/[bucket]/index.tsx)
Object detail view tab labels and field labels hard-coded in Spanish — all UI strings in ObjectDetailsView (tab names, field labels, error messages, tooltips) were written as Spanish string literals rather than using useTranslation. All strings replaced with t() calls; corresponding keys added to both en/buckets.json and es/buckets.json. (web/frontend/src/components/ObjectDetailsView.tsx, web/frontend/src/locales/en/buckets.json, web/frontend/src/locales/es/buckets.json)
Data race in metrics.collector.running — StartBackgroundCollection and StopBackgroundCollection wrote the running bool field concurrently without synchronization; detected with -race. Replaced bool with sync/atomic.Bool; StartBackgroundCollection uses CompareAndSwap to guarantee only one goroutine runs at a time, and defer Store(false) as the single write point on exit. (internal/metrics/collector.go)
Data race in handleTrace — the handler launched pprof.Trace(w, r) in a background goroutine and returned when the timeout fired, leaving the goroutine still writing to the ResponseRecorder while the caller was already reading it. Removed the goroutine+select pattern; pprof.Trace now receives a context.WithTimeout and blocks until completion, ensuring the handler does not return while w is still being written. (internal/server/profiling.go)
Data race in TestHandleNotificationStreamExtraCases — the test read rr.Code after a time.Sleep while the SSE handler was still writing to the same ResponseRecorder. Replaced the sleep with context.WithTimeout and a done channel: the test waits for the handler to finish before reading the result. (internal/server/server_test.go)

Changed¶

Go dependencies updated to latest — all direct and indirect Go dependencies updated to their latest available versions. Notable updates: google.golang.org/grpc v1.56.3 → v1.79.3, go.opentelemetry.io/otel/sdk v1.37.0 → v1.42.0, modernc.org/sqlite v1.40.0 → v1.47.0, golang.org/x/crypto, golang.org/x/net, golang.org/x/sys, golang.org/x/text updated across 6+ minor versions each. Two previously retracted packages resolved: github.com/klauspost/compress v1.18.1 → v1.18.5 and github.com/microcosm-cc/bluemonday v1.0.23 → v1.0.27. No vulnerabilities detected via govulncheck post-update. (go.mod, go.sum)

Security¶

Bucket policy Condition blocks now evaluated — statementMatches previously ignored the Condition field of every policy statement; conditions were silently skipped, meaning any policy that relied on aws:SourceIp, aws:SecureTransport, s3:prefix, or other condition keys had those restrictions completely unenforced. Implemented a full condition evaluator (conditionMatches) supporting the following operators: StringEquals, StringNotEquals, StringEqualsIgnoreCase, StringNotEqualsIgnoreCase, StringLike/StringNotLike (with AWS */? wildcards), Bool, IpAddress/NotIpAddress (CIDR ranges via net.ParseCIDR), ArnLike/ArnEquals/ArnNotEquals/ArnNotLike, and NumericEquals/NumericNotEquals. The condition context is populated from the real HTTP request: aws:SourceIp maps to the client's IP (net.SplitHostPort of r.RemoteAddr), aws:SecureTransport maps to whether r.TLS != nil, and additional keys can be passed via a RequestContext map. All condition operators within a statement are ANDed together; within each operator all keys are also ANDed. Unknown operators fail closed (deny). PolicyEvaluationRequest gained three new fields: SourceIP string, SecureTransport bool, RequestContext map[string]string. (internal/bucket/policy_evaluation.go, pkg/s3compat/handler.go)
Bucket policy Principal: null granted access to everyone — principalMatches returned true when principal == nil, treating a malformed or omitted Principal as a wildcard match. In AWS S3, Principal is a required field in bucket policy statements; a statement with a null/missing Principal is malformed and must not match any request. Any policy loaded from storage with a missing Principal field (e.g. a hand-edited or corrupted policy document) would silently allow any authenticated user through. Fixed by returning false for nil principal. (internal/bucket/policy_evaluation.go)
GET /{bucket}/{key}?versions removed — non-standard route returning 501 — this route was registered in internal/api/handler.go and mapped to a stub handler that unconditionally returned HTTP 501. The endpoint does not exist in the AWS S3 API; per-key version listing is accomplished via the standard ListObjectVersions (GET /{bucket}?versions&prefix={key}) which is already implemented. The dead route and its stub were removed. (internal/api/handler.go, pkg/s3compat/handler.go)
InternalError responses leaked internal filesystem paths — every error path that called h.writeError(w, "InternalError", err.Error(), ...) forwarded the raw Go error string to the XML response body. On Windows, OS errors from os.WriteFile include the full absolute path (e.g. C:\Users\aricardo\Projects\MaxIOFS\data\objects\...). Similarly, storage errors in StorageError.Error() concatenate the OS cause. Fixed by sanitizing InternalError messages inside writeError: the real error is now logged via logrus and the client always receives the generic AWS-compatible message "We encountered an internal error. Please try again." This is consistent with real AWS S3 behavior. (pkg/s3compat/handler.go)
Presigned V2 URL and header-based V2 signatures always rejected — Both the presigned URL validator (calculateSignatureV2 in presigned.go) and the Authorization-header-based V2 verifier (verifyS3SignatureV2 in manager.go) used HMAC-SHA256 with hex encoding, but AWS S3 Signature Version 2 specifies HMAC-SHA1 with standard base64 encoding (RFC 4648). Any presigned URL generated by the AWS CLI with --signature-version s3 or any V2-capable SDK was rejected with "signature does not match". Fixed in both code paths by switching to hmac.New(sha1.New, ...) and base64.StdEncoding.EncodeToString. (pkg/s3compat/presigned.go, internal/auth/manager.go)
NoSuchWebsiteConfiguration returned HTTP 500 instead of 404 — the error code was not listed in writeError's HTTP status switch, falling through to the default 500. AWS S3 returns 404 for this error. Added to both the 404 status case and the BucketName resource field mapping. (pkg/s3compat/handler.go)
Auth error XML responses missing AWS context fields — InvalidAccessKeyId, SignatureDoesNotMatch, and RequestExpired returned only <Resource> (the URL path) in the XML body. AWS S3 includes <AWSAccessKeyId> for credential/signature errors so clients can correlate the failed key, and <ExpiresDate>/<ServerTime> for RequestExpired to help diagnose clock-skew. Added these fields to the Error struct and populated them in writeError by extracting the access key from AWSAccessKeyId query param (V2), X-Amz-Credential query param (V4), or Authorization: AWS key:sig header, and computing expiration times from Expires (V2) or X-Amz-Date+X-Amz-Expires (V4). (pkg/s3compat/handler.go)
Presigned URL validation returned wrong HTTP status codes — HandlePresignedRequest returned InvalidRequest (400) for all validation failures. AWS S3 returns SignatureDoesNotMatch (403), RequestExpired (403), and InvalidAccessKeyId (403) for the respective conditions. Added a presignedValidationError type carrying the AWS error code; validators now return structured errors and HandlePresignedRequest maps them to the correct codes. InvalidAccessKeyId and SignatureDoesNotMatch were also incorrectly mapped to HTTP 401 in writeError; corrected to 403, and RequestExpired added as 403. (pkg/s3compat/presigned.go, pkg/s3compat/handler.go)
SigV2 CanonicalizedResource missing sub-resources — The StringToSign for AWS Signature Version 2 (both presigned URL validation and Authorization-header validation) used only the raw URL path as CanonicalizedResource. The AWS SigV2 spec requires appending recognized sub-resource query parameters (e.g. ?versionId=, ?acl, ?delete, ?uploadId=, etc.) to the resource path, sorted lexicographically. Any SigV2 request that included one of these parameters would fail signature verification. Implemented canonicalizedResourceV2(path, query) in both presigned.go and manager.go with the full list of 21 recognized sub-resources from the AWS documentation. (pkg/s3compat/presigned.go, internal/auth/manager.go)
DeleteBucket bypassed Object Lock on versioned buckets — isBucketEmpty walked the physical storage to determine if a bucket could be deleted. Versioned objects are stored at bucket/.versions/key/versionID; these paths do not match the plain metadata key format (obj:{bucket}:{key}), so every versioned physical file was classified as an "orphaned file" and deleted. After the cleanup the current metadata entry was a delete marker (ETag="" Size=0), so the bucket appeared empty and was deleted — silently bypassing COMPLIANCE-mode Object Lock retention on all non-current versions. Fixed by detecting /.versions/ paths in the walk, extracting the real key + versionID, and looking them up via GetObject(ctx, bucket, key, versionID). Files with active content (non-delete-marker) now correctly report the bucket as not empty. (internal/bucket/manager_impl.go)
Console port (8081) missing all security headers — the S3 port (8080) already applied Strict-Transport-Security, X-Content-Type-Options, and X-XSS-Protection via S3Headers() middleware, but the console API server had no security headers at all. Added ConsoleHeaders() middleware applied to the console router: X-Frame-Options: SAMEORIGIN (clickjacking protection), Content-Security-Policy (XSS mitigation), X-Content-Type-Options: nosniff, X-XSS-Protection: 1; mode=block, and Referrer-Policy: strict-origin-when-cross-origin. (internal/middleware/s3headers.go, internal/server/server.go)
Object Lock COMPLIANCE shortening returned InternalError instead of AccessDenied — when a client attempted to shorten a COMPLIANCE-mode retention date, the server returned <Code>InternalError</Code> with the error text as the message. AWS S3 returns AccessDenied for this operation. Fixed by adding an explicit check for ErrCannotShortenCompliance before the generic error fallback in PutObjectRetention. (pkg/s3compat/object_ops.go)

Added¶

GetObjectAttributes S3 API (GET /{bucket}/{key}?attributes) — returns object metadata (ETag, StorageClass, ObjectSize, Checksum, ObjectParts/TotalPartsCount) without downloading the object body. The caller specifies which attributes are needed via the x-amz-object-attributes header. Used by AWS CLI v2, Mountpoint S3, and SDK v3 as a lighter alternative to HeadObject. Multipart part count is inferred from the <md5>-N ETag suffix. (pkg/s3compat/object_ops.go, internal/api/handler.go)
PublicAccessBlock S3 API (GET/PUT/DELETE /{bucket}?publicAccessBlock) — stores and retrieves the four AWS public-access-block flags (BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy, RestrictPublicBuckets) in bucket metadata via the standard AWS XML format. Configuration was previously settable only through the MaxIOFS console API; it is now also reachable from any S3-compatible client. (internal/bucket/interface.go, internal/bucket/manager_impl.go, pkg/s3compat/bucket_ops.go, internal/api/handler.go)
PublicAccessBlock enforcement — checkPublicBucketAccess now reads the bucket's PublicAccessBlock configuration before evaluating ACL grants. If IgnorePublicAcls or RestrictPublicBuckets is set, all public access is denied regardless of the bucket or object ACL. Previously the flags were stored but never consulted. (pkg/s3compat/handler.go)
Real GetBucketLogging / PutBucketLogging — bucket server access logging configuration is now persisted and returned. PutBucketLogging with a <LoggingEnabled> element stores the target bucket and prefix; an empty <BucketLoggingStatus/> disables logging. A BucketAccessLogger background goroutine buffers access log entries and delivers them asynchronously to the target bucket in AWS S3 access log format (5-minute flush interval, 100-entry batch trigger). (internal/bucket/interface.go, internal/bucket/manager_impl.go, internal/metadata/types.go, pkg/s3compat/bucket_ops.go, internal/server/bucket_logging.go)
PutObject conditional write (If-None-Match: *) — AWS added this in August 2024; clients like Terraform state backends and distributed lock managers rely on it to guarantee atomic "write only if not exists" semantics. When the If-None-Match: * header is present and the object already exists, MaxIOFS now returns 412 PreconditionFailed instead of silently overwriting. (pkg/s3compat/handler.go)

Fixed¶

Global encryption not applied to new objects when no bucket-level rule was set — shouldEncryptObject returned false if the bucket had no explicit SSE rule configured, even when the server-wide encryption key was present and EnableEncryption=true. Only buckets that had previously received a PutBucketEncryption call were actually encrypting objects; all other buckets silently stored data unencrypted. Fixed by returning true as the default when the encryption service is initialized and no bucket override says otherwise. (internal/object/manager.go)
GetBucketEncryption / PutBucketEncryption / DeleteBucketEncryption were no-ops — all three handlers ignored the request body and never persisted any configuration. GetBucketEncryption always returned ServerSideEncryptionConfigurationNotFoundError. Implemented real persistence following the website-configuration pattern: GetEncryption/SetEncryption/DeleteEncryption methods added to bucket.Manager; handlers now parse the standard AWS ServerSideEncryptionConfiguration XML on PUT and return it on GET. (internal/bucket/interface.go, internal/bucket/manager_impl.go, pkg/s3compat/bucket_ops.go)
SSE response headers missing on GetObject, HeadObject, and PutObject — AWS S3 returns x-amz-server-side-encryption: AES256 on all three operations when the object is server-side encrypted. MaxIOFS never set this header. Added SSEAlgorithm string field to the Object struct; it is populated from persisted ObjectMetadata.SSEAlgorithm on read (with a storage-metadata fallback for objects created before this change), and set to "AES256" on write when encryption is active. All three response-header helpers now emit the header when non-empty. (internal/object/manager.go, internal/object/adapter.go, pkg/s3compat/handler.go)

Fixed¶

Version update checker couldn't distinguish stable from pre-release — isNewerVersion in AppLayout.tsx stripped all pre-release suffixes before comparing, so v1.0.0-beta, v1.0.0-rc1, and v1.0.0 were all treated as equal and no update notification was shown. Replaced the string-strip approach with a ranked comparison: stable > rc > beta > alpha; within the same rank, the numeric suffix is compared. This correctly identifies v1.0.0 as newer than v1.0.0-rc1, which is newer than v1.0.0-beta. (web/frontend/src/components/layout/AppLayout.tsx)
Debian package: 1.0.0-rc1 detected as newer than 1.0.0 by dpkg — dpkg treats - as the separator between upstream and Debian revision, so rc1 (letters) sorted after the empty string, making 1.0.0-rc1 > 1.0.0 from dpkg's perspective. Attempting to upgrade from rc1 to stable produced "downgrade" warnings. Fixed by adding epoch 1: to the package version (1:1.0.0), which is always sorted before the epoch field. Future pre-releases will use tilde notation (1.0.1~beta1) which sorts before any release. (debian/changelog, debian/control)
RPM packages: all pre-release and stable builds produced the same NVR — VERSION_CLEAN in the Makefile stripped all suffixes including -beta, -rc1, etc., so a build from v1.0.0-rc1 produced the same maxiofs-1.0.0-1 NVR as v1.0.0. RPM refused to upgrade because the NVR was identical. Fixed by adding a computed RPM_RELEASE variable: pre-release builds produce Release: 0.beta1 / Release: 0.rc1; stable produces Release: 1. The spec uses %{!?release: %define release 1} so the Makefile can override via --define "release $(RPM_RELEASE)" at build time. (Makefile, rpm/maxiofs.spec)
CI/CD: npm ci failing after frontend dependency updates — package-lock.json was out of sync with package.json after Vite/plugin updates were made directly to package.json. npm ci requires an exact lockfile match and aborted with EUSAGE. Regenerated package-lock.json with npm install; this added 8 missing transitive packages (@testing-library/dom, @types/aria-query, dom-accessibility-api, lz-string, pretty-format, ansi-regex, ansi-styles, react-is). (web/frontend/package-lock.json)
CORS preflight OPTIONS returning 405 on bucket and object routes — gorilla/mux returns HTTP 405 Method Not Allowed before running subrouter middleware when a route matches by path but not by method. Since no OPTIONS method was registered for /{bucket}, /{bucket}/, or /{bucket}/{object} routes, CORS preflight requests received 405 instead of being intercepted by bucketCORSMiddleware. Fixed by registering explicit OPTIONS no-op handlers for all three path patterns; bucketCORSMiddleware intercepts them and returns 200 with the correct CORS headers. (internal/api/handler.go)
WORM badge not shown in cluster mode — in cluster mode handleListBuckets uses BucketAggregator.ListAllBuckets() which converts local bucket.Bucket structs to BucketWithLocation. The BucketWithLocation struct lacked an ObjectLock field, so the Object Lock configuration was silently dropped during conversion. The console API's bucketWithLocationToBucket function propagated the missing field to the response, resulting in no objectLock field in the bucket list JSON and the WORM badge never rendering. Fixed by adding ObjectLock *bucket.ObjectLockConfig to BucketWithLocation, mapping it from the source bucket in ListAllBuckets, and including it in the bucketWithLocationToBucket conversion. (internal/cluster/bucket_aggregator.go, internal/server/console_api.go)
Object Lock retention modal fully in English regardless of UI language — ObjectLockConfigModal had no useTranslation hook; all strings (warning banner, labels, button text, error messages) were hardcoded in English. Applied full i18n via useTranslation('bucketSettings'); all strings now use t('objectLock.configModal.*') keys. (web/frontend/src/components/ObjectLockConfigModal.tsx)

Added¶

i18n: objectLock.configModal keys added to English and Spanish locales — added objectLock.configModal.* translation keys for the Object Lock configuration modal (title, warning messages, current/new retention labels, unit selectors, action buttons, success/error messages) to both en/bucketSettings.json and es/bucketSettings.json. (web/frontend/src/locales/en/bucketSettings.json, web/frontend/src/locales/es/bucketSettings.json)
i18n: stable status key added to About page locales — the About page "Status" badge referenced t('stable') which was missing from both locale files, falling back to the key name. Added "stable": "Stable" (English) and "stable": "Estable" (Spanish). (web/frontend/src/locales/en/about.json, web/frontend/src/locales/es/about.json)

Changed¶

README.md — complete rewrite with MinIO comparison and modern structure — the previous README was a flat markdown file. Rewrote with: centered header with CI/version/license/Go/S3/Security badges; tagline; honest feature-by-feature comparison table with MinIO (marking missing features clearly: no erasure coding, no S3 Select, no cloud tiering); collapsible <details> sections for S3 API, Multi-tenancy, Identity & Access, Security, Cluster & HA, and Operations features; single-command Docker quick start; docker-compose and binary install examples; performance table; documentation index; contributing and security sections. (README.md)
About page "New In v1.0.0" section expanded — the section previously listed only 4 UI changes from the last development session. Expanded to 11 items covering the full v1.0.0 scope: complete frontend redesign, folder upload, POST presigned URLs, bucket notifications, lifecycle execution, spec-compliant multipart ETag, Veeam B&R compatibility, Object Lock per-version enforcement, security fixes BUG-25/26/27, throughput metrics fix, and refresh token fix. (web/frontend/src/pages/about/index.tsx)

[1.0.0] - 2026-03-17¶

Added¶

Folder upload in bucket browser — the bucket browser now supports uploading entire folder trees via drag-and-drop (using DataTransfer.webkitGetAsEntry(), works in all browsers without any confirmation dialog) or via "Browse folder" button backed by the File System Access API showDirectoryPicker() (Chrome/Edge/Safari 15.2+; Firefox shows an unsupported message). The full relative path is preserved as the S3 key prefix, matching AWS S3 behavior. The upload modal has Files / Folder tabs. (web/frontend/src/pages/buckets/[bucket]/index.tsx, web/frontend/src/locales/en/buckets.json, web/frontend/src/locales/es/buckets.json)

Changed¶

Button consistency across all pages — all action buttons now use the shared Button component (consistent font, padding, border-radius, icon spacing via gap-2). Previously Settings, Logging Targets, Migrations, Security, Dashboard, and Buckets pages mixed raw <button> elements and manually-styled <Link> tags with the Button component, causing visual inconsistencies. Also removed mr-2 from icon children inside Button elements (redundant with the gap-2 base class that was causing double spacing). (web/frontend/src/pages/settings/index.tsx, LoggingTargets.tsx, web/frontend/src/pages/security/index.tsx, web/frontend/src/pages/cluster/Migrations.tsx, web/frontend/src/pages/index.tsx, web/frontend/src/pages/buckets/index.tsx)
Upload modal UX — the upload modal now shows a styled drag-and-drop zone with a branded "Browse files" button; after selecting files the drop zone collapses to a compact bar so the modal no longer grows downward. The selected file list is capped at a scrollable max-h-48 preview area. (web/frontend/src/pages/buckets/[bucket]/index.tsx)

Fixed¶

Object table header height jumps on row selection — when checking items in the bucket browser, the "Delete selected" button appeared conditionally inside the table header, pushing the layout downward. The header container now has a fixed height (h-14 shrink-0); the button is rendered conditionally inside the fixed container so the layout remains stable. (web/frontend/src/pages/buckets/[bucket]/index.tsx)

Security¶

BUG-27 (MEDIUM): Open redirect via success_action_redirect in POST presigned uploads — HandlePostObject redirected to any URL supplied in the success_action_redirect form field, including javascript: and data: schemes. Added a scheme whitelist check: only URLs starting with http:// or https:// are accepted; all other values return HTTP 400 InvalidArgument. (pkg/s3compat/presigned.go)
BUG-25 (CRITICAL): SSRF via webhook notification delivery — deliverNotification used http.DefaultClient with no network restrictions, allowing an attacker with bucket-owner privileges to configure a notification webhook pointing to internal services (e.g. http://169.254.169.254/, http://localhost:8081/). Replaced with a custom notifSSRFBlockingClient() that dials through a net.Dialer which rejects loopback, link-local, and RFC-1918 private addresses before any HTTP bytes are sent. The client is injectable via Handler.notifHTTPClient so tests can bypass the SSRF check using a plain client. (pkg/s3compat/notifications.go, pkg/s3compat/handler.go)
BUG-26 (HIGH): No validation on notification webhook endpoint URL — PutBucketNotification in bucket_ops.go accepted any string as a queue/topic/function ARN value without checking it was a valid HTTP URL. An attacker could store malformed or non-HTTP values that would later cause unhandled errors or unexpected behavior in the delivery goroutine. Added URL scheme validation: only http:// and https:// endpoints are accepted; others return InvalidArgument. (pkg/s3compat/bucket_ops.go)

Changed¶

Log levels: per-request S3 traces moved from Info to Debug — several high-frequency per-request log entries were emitted at Info level, causing excessive noise in production deployments running at the default Info log level. The following were changed to Debug: SigV4/V2 signature parsing steps (ValidateS3Signature called, Parsed SigV4 signature, Canonical request components, Delegating to SigV4/V2), per-request path resolution traces (GetObject: Using bucketPath, PutObject: Using bucketPath), GetObjectLockConfiguration step-by-step traces, quota check internals, and the S3 API request log in both S3RequestLog middleware and server.go. Meaningful operational events (object created/deleted, multipart completed, replication results, lifecycle expiry, Object Lock changes) remain at Info. (internal/middleware/verbose_logging.go, internal/server/server.go, internal/auth/manager.go, pkg/s3compat/handler.go, pkg/s3compat/object_ops.go, pkg/s3compat/multipart.go, internal/object/manager.go, internal/replication/, internal/lifecycle/worker.go)

UI — Complete Frontend Redesign¶

New floating layout — the application shell now uses a floating design: sidebar and main content area are inset from the browser edge (p-3 gap-3) with the page background color (slate-200 light / slate-900 dark) visible as a gap, creating a framed floating appearance. (web/frontend/src/components/layout/AppLayout.tsx)
Collapsible floating sidebar — the sidebar can now be collapsed to icon-only mode (w-16) or expanded (w-64) via a toggle button at the bottom. State is persisted to localStorage. In icon-only mode each item shows its icon with a native title tooltip and navigates directly on click. On mobile the sidebar slides in/out as a fixed overlay. New icons: PanelLeftClose / PanelLeftOpen for the toggle button. (web/frontend/src/components/layout/SidebarNav.tsx)
New light mode theme — white cards on slate background — replaced the previous all-gray light mode (#7d7d7d background, #e5e7eb cards) with a proper contrast scheme: slate-200 (#e2e8f0) page background, white (#ffffff) cards and sidebar, slate-900 foreground text, slate-600 muted text, slate-300 borders. This eliminates the "gray on gray" readability problem that affected all pages in light mode. Dark mode is unchanged. (web/frontend/src/index.css)
Design tokens — global replacement of hardcoded gray colors — a mass replacement across all pages and components converted hardcoded Tailwind gray pairs (e.g. text-gray-500 dark:text-gray-400, bg-white dark:bg-gray-800, border-gray-200 dark:border-gray-700, hover:bg-gray-50 dark:hover:bg-gray-700) to semantic CSS variable tokens (text-muted-foreground, text-foreground, bg-card, border-border, hover:bg-secondary). All pages, modals, and UI components are now fully theme-aware. (web/frontend/src/pages/**/*.tsx, web/frontend/src/components/**/*.tsx)
TopBar redesign — removed hardcoded white/dark background in favor of bg-transparent (inherits the parent card container), removed the bottom border divider, and changed all action buttons from bg-background (blended with gray page in light mode) to bg-card (clearly visible white/dark surface). Removed sticky positioning (not needed with the new layout). Dropdowns use bg-card border-border shadow-float. (web/frontend/src/components/layout/TopBar.tsx)
Button component — outline/secondary/ghost variants fixed for light mode — variant="outline" used bg-background which in light mode matched the medium gray page background, making buttons invisible. Changed to bg-card for outline and secondary variants so buttons are always clearly visible as white/dark surfaces. ghost variant now uses text-foreground hover:bg-secondary. (web/frontend/src/components/ui/Button.tsx)
Page headers standardized — all page titles now use text-2xl font-bold text-foreground and all description lines use text-sm text-muted-foreground mt-1, replacing the inconsistent mix of text-3xl, text-2xl, missing text-sm, and missing mt-1 across pages. A reusable PageHeader component was created for future use. (web/frontend/src/components/ui/PageHeader.tsx, all pages under web/frontend/src/pages/)
Table component — compact S3-style rows — reduced cell padding from px-6 py-4 to px-4 py-2 and header padding from px-6 py-3 to px-4 py-2, matching the compact single-line row style of AWS S3 console. Row hover changed from a gradient with left border accent to a simple hover:bg-secondary. Header background uses bg-secondary. Row dividers use divide-border. (web/frontend/src/components/ui/Table.tsx)
Bucket list — inline name cell — removed the w-9 h-9 gradient icon container that forced rows to be tall. The bucket name cell now shows a small inline icon, the bucket name link, and status badges (WORM, scan progress, integrity issues) all on a single line, matching AWS S3 list style. All gradient hover effects on TableRow across pages replaced with hover:bg-secondary. (web/frontend/src/pages/buckets/index.tsx)
Dashboard page — theme-aware — replaced hardcoded bg-white dark:bg-gray-800 rounded-xl border border-gray-200 cards with the Card component. Page title uses text-foreground, description uses text-muted-foreground. All chart empty states, legend items, and bucket list items use semantic tokens. (web/frontend/src/pages/index.tsx)
About page — transparent logo background in light mode — the logo container had bg-gradient-to-br from-gray-50 to-gray-100 in light mode. Changed to transparent; the dark mode gradient is preserved for logo visibility. (web/frontend/src/pages/about/index.tsx)
Content area padding reduced — the inner content wrapper padding was halved from p-4 md:p-6 2xl:p-8 to p-2 md:p-3 2xl:p-4, giving more usable screen space to page content. (web/frontend/src/components/layout/AppLayout.tsx)
i18n: added collapseSidebar and expandSidebar keys — the new sidebar collapse toggle button referenced these translation keys which were missing from both English and Spanish locale files. (web/frontend/src/locales/en/layout.json, web/frontend/src/locales/es/layout.json)

Fixed¶

Notification webhook tests failing after SSRF client introduction — the new notifSSRFBlockingClient() correctly blocked connections to 127.0.0.1 (loopback), which is where test webhook servers listen. Tests injecting webhook endpoints pointed at localhost always received a connection-refused error from the SSRF dial hook. Fixed by adding a notifHTTPClient *http.Client field to Handler; production code leaves it nil (falls back to SSRF client), and tests inject plainHTTPClient() (standard http.Client with no restrictions) to allow localhost webhook delivery. (pkg/s3compat/handler.go, pkg/s3compat/notifications.go, pkg/s3compat/notifications_test.go)

Dependencies¶

Downgraded Vite from v8 to v7 — vite@8.0.0 was installed by npm update but @tailwindcss/vite@4.x only declares peer support for ^5.2.0 || ^6 || ^7; the build failed with ERESOLVE unable to resolve dependency tree. Pinned to vite@^7.3.1. (web/frontend/package.json)
Downgraded @vitejs/plugin-react from v6 to v4 — @vitejs/plugin-react@6 imports vite/internal which does not exist in Vite 7, causing ERR_PACKAGE_PATH_NOT_EXPORTED at build time. Downgraded to @vitejs/plugin-react@^4, the last version compatible with Vite 7. (web/frontend/package.json)
Fixed vite.config.ts for Vite 7 compatibility — the config used two Vite 8-only APIs: minify: 'oxc' (OXC minifier, Rust-based, Vite 8 only) and rolldownOptions (Rolldown bundler API, Vite 8 only). Changed to minify: 'esbuild' and rollupOptions for Vite 7 compatibility. (web/frontend/vite.config.ts)
Updated jsdom from v28 to v29 — no breaking changes affecting the project; Vitest declares jsdom: '*' as a peer dependency so any version is accepted. (web/frontend/package.json)

Added¶

S3: POST presigned URLs (HTML form upload) — browsers can now upload files directly to a bucket via HTML <form enctype="multipart/form-data"> using S3-compatible POST policy signatures. The server validates the policy expiration, HMAC-SHA256 signature (V4) or V2 signature, bucket/key/content-type conditions, starts-with prefix conditions, and content-length-range constraints before storing the object. Supports success_action_redirect (303), success_action_status (200/201/204), and x-amz-meta-* form fields. (pkg/s3compat/presigned.go, internal/api/handler.go)
S3: bucket notifications now stored and dispatched as webhooks — PutBucketNotification and GetBucketNotification were previously no-ops; the configuration was accepted but discarded. Notification config is now persisted in the bucket metadata. After each mutating object operation (PutObject, DeleteObject, CopyObject, CompleteMultipartUpload), the bucket's notification rules are evaluated: event type is matched against configured patterns (supports trailing wildcard s3:ObjectCreated:*), object key is checked against prefix/suffix filters, and matching targets receive an S3-compatible JSON event payload via HTTP POST (non-blocking goroutine). Since MaxIOFS is not AWS, SNS/SQS/Lambda ARN values are treated as webhook HTTP endpoint URLs (same approach as MinIO). (internal/metadata/types.go, internal/bucket/types.go, internal/bucket/interface.go, internal/bucket/manager_impl.go, internal/bucket/adapter.go, pkg/s3compat/bucket_ops.go, pkg/s3compat/notifications.go, pkg/s3compat/handler.go, pkg/s3compat/object_ops.go, pkg/s3compat/multipart.go)
S3: per-bucket CORS rules are now applied to actual requests — bucket CORS configurations stored via PutBucketCORS were saved to the metadata store but never enforced on incoming requests. Added bucketCORSMiddleware to the S3 API handler that looks up the bucket's CORS rules, matches the request Origin against the configured allowed origins and methods, and sets the appropriate Access-Control-Allow-Origin, Access-Control-Allow-Methods, Access-Control-Allow-Headers, Access-Control-Expose-Headers, and Access-Control-Max-Age headers. The middleware runs before s3ClientMiddleware so that browser OPTIONS preflight requests (which carry no auth headers) are answered correctly without being redirected to the web console. (internal/api/handler.go)
S3: lifecycle Expiration.Days / Expiration.Date rules now executed — lifecycle rules with an Expiration action (delete objects older than N days, or older than a fixed date) were parsed and stored but the background lifecycle worker never evaluated them. Added processObjectExpiration to the worker; it lists all objects under the rule prefix, compares each object's LastModified against the configured cutoff (either time.Now() - Days or the literal Date), and calls DeleteObject for expired objects. On versioned buckets DeleteObject without a versionId creates a delete marker (correct S3 behavior). (internal/lifecycle/worker.go)
S3: lifecycle AbortIncompleteMultipartUpload rules now executed — lifecycle rules with an AbortIncompleteMultipartUpload action were stored but never evaluated. Added processAbortIncompleteMultipartUploads to the worker; it lists all in-progress multipart uploads for the bucket, compares each upload's Initiated timestamp against time.Now() - DaysAfterInitiation, and calls AbortMultipartUpload for stale uploads. (internal/lifecycle/worker.go)
S3: multipart upload ETag now uses the correct MD5(parts)-N format — CompleteMultipartUpload previously used only the MD5 of the concatenated part ETags as a hex string. The AWS S3 spec defines the multipart ETag as hex(MD5(raw_binary_MD5_part1 || … || raw_binary_MD5_partN))-N, where each part digest is the 16 raw bytes decoded from the hex ETag. Added computeMultipartETag to compute the spec-compliant value; the resulting ETag is stored in metadata and returned in the response. This enables clients such as aws s3 sync --checksum to correctly verify uploaded multipart objects. (internal/object/manager.go)

Fixed¶

Veeam: HEAD / returned 404, causing Veeam to enable multi-bucket mode — Veeam makes a HEAD / request to the S3 endpoint root before any bucket operations to verify it is talking to a valid S3 service. The router only registered GET / (ListBuckets) but not HEAD /, so Veeam received HTTP 404 with no S3 headers. This caused Veeam to classify the storage as generic S3-compatible and activate multi-bucket mode automatically. Fixed by registering HEAD / on the same route and returning HTTP 200 with S3-compatible headers. (internal/api/handler.go)
Veeam: HeadBucket and GetBucketLocation missing x-amz-bucket-region header — Veeam reads this header from both the HEAD /bucket and GET /bucket?location responses to determine the bucket region and decide whether to activate multi-bucket mode. Without the header, Veeam cannot confirm same-region access and defaults to enabling multi-bucket mode, which must then be disabled manually. Added x-amz-bucket-region: us-east-1 to both responses, consistent with MinIO behavior. (pkg/s3compat/handler.go)
Veeam / S3: GetBucketLocation returned "us-east-1" instead of empty string — the AWS S3 spec requires that buckets in the default region return an empty LocationConstraint, not the string "us-east-1". Fixed to return "". (pkg/s3compat/handler.go)
Veeam / S3: PutObjectLockConfiguration nil pointer panic when <Rule> element is absent — if a client sent a PutObjectLockConfiguration request without a <Rule>/<DefaultRetention> element (e.g., a bare <ObjectLockEnabled>Enabled</ObjectLockEnabled> body), the handler accessed newConfig.Rule.DefaultRetention.Mode directly, triggering a nil pointer dereference that was caught by the deferred panic handler and returned HTTP 500. Veeam interpreted this as "default retention is not supported". Added an explicit nil check: if Rule or DefaultRetention is absent the handler now returns MalformedXML 400 with a clear message. (pkg/s3compat/handler.go)
S3 Object Lock: PutObjectLockConfiguration returned wrong error code when Object Lock is not enabled on bucket — when called on a bucket that was not created with Object Lock enabled, the handler returned ObjectLockConfigurationNotFoundError (the same 404 code used by GetObjectLockConfiguration). The AWS S3 spec requires InvalidBucketState with HTTP 409 Conflict for this case. Fixed to return InvalidBucketState 409; also added InvalidBucketState to the writeError switch so it maps to the correct HTTP status. (pkg/s3compat/handler.go)
Metrics: real-time throughput cards (requests/s, bytes/s, objects/s) always showed zero — RecordThroughput was defined but never called anywhere; the three counters (requestCount, bytesProcessed, objectsProcessed) were always 0 so CalculateThroughput() returned zero for all three values every 5-second cycle. Fixed by calling RecordThroughput inside TracingMiddleware after every request: bytes = r.ContentLength (upload body) + response bytes written (tracked by extending responseWriter with a bytesWritten counter); objects = 1 for PutObject / GetObject / DeleteObject / CopyObject / MultipartUpload operations. Latency percentile cards were unaffected. (internal/middleware/tracing.go)
Cluster: duplicate buckets in list when same bucket is replicated across nodes — when a bucket was mirrored/replicated on multiple cluster nodes, both S3 API ListBuckets and the web console showed one entry per node (e.g. two entries for the same logical bucket). Users could mistake a replica for a duplicate and delete it. BucketAggregator.ListAllBuckets now deduplicates by (TenantID, Name) after aggregating all nodes; one canonical entry per logical bucket is returned, preferring the local node. The console handleListBuckets in cluster mode uses bucketAggregator.ListAllBuckets, so S3 clients and the web console share the same deduplicated list. (internal/cluster/bucket_aggregator.go, internal/server/console_api.go, internal/cluster/bucket_aggregator_test.go)
Cluster / quotas: tenant storage must not double-count replicated buckets — when a bucket is replicated across nodes, the same data exists on multiple nodes; tenant storage usage must be counted once, not per replica. (Phase 1: list dedup done; quota aggregation to exclude replica size is planned.)
Static website: error document not persisted when set — configuring only the index document did not persist correctly; setting both index and error document appeared to work but the error document field showed empty when revisiting the config. The errorDocument field is now optional; when provided it is persisted. (internal/metadata/types.go, internal/server/website_handlers.go, web/frontend/src/pages/buckets/[bucket]/settings.tsx, web/frontend/src/lib/api.ts)
Static website: 404 when accessing website endpoint for bucket without website config — accessing the website-style URL for a bucket that has no static website hosting configured now returns 403 Access Denied (S3-style XML) instead of 404, so unconfigured website endpoints do not leak bucket existence. (internal/server/server.go)
Clean URL shares (non-presigned) returning 403 — shared object URLs that do not use presigned query parameters (clean links stored in DB) were failing with 403. Share URLs are now generated without tenant_id in the path (bucket names are globally unique). GetShareByObject when tenantID is empty (clean URL) searches by bucket_name and object_key only. (internal/server/console_api.go, internal/share/sqlite.go)
TestAccessKeySyncManager_Start "enabled auto sync" flakiness on CI — setup (schema init, DB inserts) used a context with timeout, causing "context deadline exceeded" on slow CI. Setup now uses context.Background(); only the Start() call uses context.WithTimeout to assert non-blocking behavior. (internal/cluster/access_key_sync_test.go)
Refresh token never saved after 2FA login — verify2FA in api.ts discarded the refresh_token returned by POST /auth/2fa/verify; sessions after 2FA expired after 15 minutes. LoginResponse.refreshToken is now populated and forwarded to tokenManager.setTokens(). (web/frontend/src/lib/api.ts)
Refresh token never saved after OAuth/SSO login — oauth-complete.tsx only read token from the URL, ignoring refresh_token; OAuth/SSO sessions expired after 15 minutes. The page now reads and stores refresh_token. (web/frontend/src/pages/auth/oauth-complete.tsx)
Audit logs export only fetched the current page — exportToCSV used filteredLogs (capped to current page). Export now loops through all pages via the API. (web/frontend/src/pages/audit-logs/index.tsx)
Audit logs stats cards showed counts for current page only — success/failed counters were derived from the current page. Replaced with queries that use the total field from the API. (web/frontend/src/pages/audit-logs/index.tsx)
CSV export: timestamp column split by comma — formatTimestamp used toLocaleString() (e.g. "3/5/2026, 2:30:00 PM"), so the comma split the timestamp into two columns. Replaced with YYYY-MM-DD HH:MM:SS format. (web/frontend/src/pages/audit-logs/index.tsx)
Clean share (URL limpia) 403 — S3 API — unauthenticated GET to /{bucket}/{object} (clean share link, no presigned params) now resolves the share via GetShareByObject with empty tenant and, when found, serves the object instead of returning 403. Bucket read permission uses an allowedByShare flag so access is granted whenever the share was found (including global buckets where shareTenantID is empty; previously the check shareTenantID != "" wrongly denied those). (pkg/s3compat/handler.go; lookup by bucket+object when tenant empty in internal/share/sqlite.go.)
Website 403 HostId — when the static website endpoint returns 403 Access Denied (bucket without website config or bucket not found), the response <HostId> and header X-Amz-Id-2 now use the request host (r.Host) instead of a static value. (internal/server/server.go)
S3: X-Amz-Id-2 header sent as hex instead of base64 — MaxIOFS generated the host ID as a 64-character lowercase hex string (hex.EncodeToString), while AWS S3 and MinIO use base64-encoded binary (48 random bytes → 64-char base64). Some S3 clients validate or log this header; using non-standard encoding could cause parsing issues. Changed to base64.StdEncoding.EncodeToString with 48 bytes of random data. (internal/middleware/s3headers.go, pkg/s3compat/handler.go)
S3: responses missing Content-Length header — writeXMLResponse called w.WriteHeader(statusCode) before the response body was known, causing Go's HTTP layer to fall back to chunked transfer encoding (no Content-Length). AWS S3 and MinIO always send Content-Length. Fixed by buffering the full XML body first, setting Content-Length from the buffer size, then writing the header and body. (pkg/s3compat/handler.go)
Object Lock: default retention rule now optional at bucket creation — the console API and frontend previously required a retention mode (GOVERNANCE/COMPLIANCE) and a period (days/years) whenever Object Lock was enabled, making it impossible to create a bucket with Object Lock enabled but no bucket-level default retention. Veeam B&R and other clients that manage retention per-object reject buckets with a pre-configured default retention rule ("unable to use backup immutability: the default retention is not supported"). Fixed: mode and period are now optional — a bucket can be created with Object Lock enabled and no default retention rule. If either mode or period is provided, both must be specified (consistency validation remains). The bucket-level default retention was never required by the AWS S3 spec; it is a convenience feature that is incompatible with clients that set retention per-object. (internal/server/console_api.go, web/frontend/src/pages/buckets/create.tsx)
Veeam SOSAPI: capacity.xml reported 0 bytes when tenant has no storage quota configured — when a tenant user requested capacity.xml and the tenant had no quota set (MaxStorageBytes == 0), the handler calculated totalCapacity = 0 and availableCapacity = 0, causing Veeam to display the repository as full or with unknown capacity. Fixed: when MaxStorageBytes is 0 (no quota), the handler now falls back to the actual disk usage (same as global users), reporting the real available disk space. (pkg/s3compat/sosapi.go)
Object Lock: PutObjectLockConfiguration with no <Rule> element now clears the bucket-level default retention — AWS S3 allows sending a PutObjectLockConfiguration request with only <ObjectLockEnabled>Enabled</ObjectLockEnabled> and no <Rule> element to remove the bucket-level default retention policy without disabling Object Lock or affecting per-object locks. Previously MaxIOFS returned HTTP 400 MalformedXML for any request missing <Rule>/<DefaultRetention>, preventing administrators from clearing a pre-existing default retention rule on a bucket. Fixed: when <Rule> is absent the handler now clears ObjectLock.Rule and persists the updated configuration. Existing per-object lock metadata is unaffected. (pkg/s3compat/handler.go)
Veeam: "unable to use backup immutability: the default retention is not supported" — root cause: versioning Suspended on Object Lock bucket — Object Lock requires versioning to be permanently Enabled; AWS S3 does not allow suspending versioning once Object Lock is active. If the bucket's versioning status was Suspended (or unset) while Object Lock was enabled, GetBucketVersioning returned Suspended, causing Veeam to reject the bucket as incompatible. Fixed in two places: (1) GetBucketVersioning now overrides Suspended/unset to Enabled when Object Lock is active, matching AWS S3 behavior where Object Lock permanently locks versioning on; (2) PutBucketVersioning now returns InvalidBucketState (409) when attempting to suspend versioning on an Object Lock bucket. (pkg/s3compat/handler.go)
S3: HeadBucket missing x-amz-bucket-object-lock-enabled header — AWS S3 and MinIO include this header in the HeadBucket response when the bucket was created with Object Lock enabled. Veeam B&R uses it to determine whether the bucket supports immutability before proceeding with Object Lock configuration. The handler only called BucketExists (boolean check) and never exposed the Object Lock status. Changed to GetBucketInfo to retrieve full bucket metadata and set x-amz-bucket-object-lock-enabled: true when Object Lock is enabled. Also moved the Veeam HeadBucket debug log to after all headers are set (previously it logged at the start of the handler before the bucket lookup). (pkg/s3compat/handler.go)
Veeam immutable backup: HeadObject with ?versionId returned 404 — Veeam B&R verifies objects in an immutable repository by issuing HEAD /{bucket}/{object}?versionId={id} before starting a backup session (Cloud.StartBackupSession). HeadObject internally called GetObjectMetadata which ignores the versionId query parameter entirely, so it looked up the latest version regardless; when the latest version differed from the requested one the lookup failed and returned 404, causing Veeam to abort with "Failed to preprocess target Error: Not Found". Fixed by detecting the versionId query parameter in HeadObject and delegating to GetObject (which already supports version lookups) when one is provided, closing the reader immediately since HEAD does not return a body. (pkg/s3compat/handler.go)
Veeam immutable backup: PutObjectRetention with ?versionId returned 404 — after verifying objects, Veeam applies per-object retention by issuing PUT /{bucket}/{object}?retention&versionId={id}. The PutObjectRetention handler called SetObjectRetention which used GetObjectMetadata (no version support) to look up the object, returning 404 for any version-specific retention request. This caused the backup session to abort immediately after the object verification phase. Fixed in two layers: (1) SetObjectRetention in the object manager now accepts an optional versionID ...string parameter and uses metadataStore.GetObject with the specific version when provided; (2) the PutObjectRetention handler extracts the versionId query parameter and forwards it. (pkg/s3compat/object_ops.go, internal/object/manager.go)
Object Lock: enabling Object Lock on a bucket did not automatically enable versioning — AWS S3 requires versioning to be permanently Enabled on any bucket with Object Lock; enabling Object Lock atomically enables versioning and prevents it from ever being suspended. SetObjectLockConfig only saved the Object Lock configuration without touching the versioning state, leaving the bucket in an inconsistent state (Object Lock active, versioning unset). Separately, PutBucketVersioning already blocked Suspended on Object Lock buckets but the initial versioning activation on bucket creation was missing. Fixed: SetObjectLockConfig now sets Versioning: Enabled in the same UpdateBucket call whenever ObjectLockEnabled is true. (internal/bucket/manager_impl.go)
Object Lock: legal hold and retention not enforced on specific object versions — when SetObjectLegalHold or SetObjectRetention was called with a specific versionId, the updated lock metadata was written to the non-versioned metadata key instead of the per-version key. As a result, DeleteObject with a versionId read from the per-version key and found no legal hold or retention, allowing deletion of locked versions. Root cause: PutObjectVersion stored a lightweight ObjectVersion struct (no LegalHold/Retention fields) at the versioned key; the full ObjectMetadata (with lock fields) was only at the non-versioned key. Fixed by storing full ObjectMetadata at versioned keys, adding an IsLatest field, and routing PutObject writes to both the versioned and non-versioned keys when the version is the latest. (internal/metadata/types.go, internal/metadata/pebble_objects.go, internal/object/manager.go, internal/object/adapter.go)
Tests: TestS3Operation_MissingAuthentication expected 403 but received 307 redirect — the test sent requests with no S3 headers, so isS3Client() returned false and s3ClientMiddleware redirected to the console URL before the authentication check ran. Fixed by adding x-amz-content-sha256 to test requests so they pass the S3 client check and reach the auth logic that correctly returns 403 AccessDenied. (internal/api/handler_security_test.go)
Tests: PutObjectLockConfiguration XML missing required S3 namespace — several tests sent <ObjectLockConfiguration> XML without the xmlns="http://s3.amazonaws.com/doc/2006-03-01/" namespace. The Go xml.Unmarshal decoder enforces namespace matching because the struct declares the full qualified name, so all requests were rejected with "expected element in name space … but have no name space". Added the namespace to all affected test XML payloads. (pkg/s3compat/s3_test.go)
Tests: TestGetBucketLocation incorrectly checked response body for region string — GetBucketLocation correctly returns an empty <LocationConstraint> per the AWS S3 spec (default-region buckets return empty body). The test asserted the body contained "us-east-1", which always failed. Fixed to assert the x-amz-bucket-region response header instead. (pkg/s3compat/s3_test.go)
Tests: TestValidateShareAccess_TenantBucket mock returned wrong type — mockShareManagerFull.GetShareByObject returned map[string]interface{} but validateShareAccess casts the result to *share.Share; the type assertion always failed with "share manager returned invalid type". Fixed the mock to return a proper *share.Share. (pkg/s3compat/handler_coverage_test.go)

Pre-1.0 history — 0.2.0-dev through 1.0.0-rc1 (October 2025 – March 2026)¶

Twenty alpha, beta and release-candidate versions preceded the first stable release. They are summarised here rather than listed: none of them were supported, and every one has been superseded by 1.0.0. The full entries remain in git log and in this file's history.

0.2.0-dev → 0.2.5-alpha (Oct 2025) — the foundations: an S3-compatible API, the React console, SQLite metadata, the filesystem backend, and the beginnings of multi-tenancy, users and access keys.
0.3.x-beta (Oct–Nov 2025) — bucket tagging, a CORS editor, complete bucket policy operations, object versioning with delete markers, and lifecycle rules — each with a visual editor rather than raw XML.
0.4.x-beta (Nov 2025) — the audit log: a SQLite store with 20+ event types, filtering and CSV export in the console, and a configurable retention policy.
0.5.0-beta (Dec 2025) — bucket replication (realtime, scheduled and batch), the logging infrastructure (console, file, HTTP, syslog), themes, nightly multi-architecture builds, and the frontend test suite.
0.6.x-beta (Dec 2025 – Jan 2026) — clustering: node-to-node bucket replication over HMAC, tenant synchronisation, the cluster management UI, bucket-location caching with failover, and Debian packaging.
0.7.0-beta (Jan 2026) — the database migration system, AWS-format access keys (AKIA), bucket inventory reports, cluster bucket migration, RPM packages and the benchmark suite.
0.8.0-beta (Feb 2026) — object search and filters, bucket policy enforcement, presigned URL validation (SigV4 and SigV2), Object Lock default retention, and the cluster join protocol.
0.9.x-beta (Feb 2026) — identity providers (LDAP and OAuth SSO with auto-provisioning and group mapping), inter-node TLS, external syslog targets, tombstone-based deletion sync, and a round of tenant-isolation fixes across the user, access-key and bucket-permission handlers.
1.0.0-beta (Mar 2026) — the metadata engine moved from BadgerDB to Pebble, object integrity verification, SMTP notifications, disk and quota alerts, maintenance mode, and the cluster state-snapshot and stale-reconciler mechanisms. Object compression was removed.
1.0.0-rc1 (Mar 2026) — a full security audit and its remedies: AES-256-GCM replacing AES-256-CTR for object encryption, the cluster CA private key no longer crossing the network, missing authorization on cluster handlers, SSRF in webhooks, logging targets and replication endpoints, path traversal on Windows, plaintext replication credentials, a predictable HMAC nonce, an infinite proxy loop, OAuth CSRF validation, CORS allowlisting, and a set of frontend XSS and cookie-flag fixes.