docs
Try the demo Get started
Esc
↑↓ moveEnter openEsc close

Run it · 8 of 12

Checking the store, snapshots, and moving blobs

Looking after the inbuxa store: checking that no message has lost its content, taking a snapshot for a backup without stopping the server, and changing where blobs are kept without losing any.

Who needs this: Anyone who runs an inbuxa server and backs it up.

13 min read

On this page

The server keeps two kinds of data. The data store holds mailboxes, calendars, contacts, settings and every message's details. The blob store holds the content itself: message bodies, attachments, files and filter scripts. By default both are in one RocksDB directory, but the blob store can be a directory of its own, S3, Azure or a database.

A message whose details are in the data store and whose content isn't in the blob store can't be opened, and the server still looks healthy. Three tools deal with that:

  • a store check finds such messages;
  • a snapshot gives a backup tool a consistent copy without stopping the server;
  • a blob store change keeps mail readable when the blob store setting moves, until its content is copied over.

They're in 27.1-rc.16 and 2026.11.9.1 and later. In the console they're under Settings › Storage › Health & backups (inbuxa Admin 27.1-rc.11 and later), and over the API they're inbuxa:StoreCheck, inbuxa:StoreSnapshot and inbuxa:BlobStoreChange, under the urn:inbuxa:jmap capability. Each one needs its own permission, which the built-in Administrator role has:

Permission Allows
sysStoreCheck Check that the store has every blob its data needs
sysStoreSnapshot Take a snapshot of the store without stopping the server
sysBlobStoreChange See, plan, copy, apply, clean up after and dismiss changes of blob store

They cover every account's data, so a tenant administrator never gets them. Everything done with them is in the audit log; reading isn't.

Checking the store

A check reads and reports. It changes nothing, runs inside the server while mail flows, and works the same on one machine or a cluster, whatever the stores are.

There are two kinds:

  • Quick: every blob something needs is in the blob store.
  • Full: also reads each blob back whole and checks that its content hashes to its id. Blob ids are the hash of their content, so this proves the content is what was written, not just that something is stored under the id. It reads the whole store, so it takes longer.

It covers the blobs that mail, files, filter scripts, calendar events and contacts hold, queued outgoing mail, and deleted items kept for undelete. For mail, it also checks that each message holds a link to the blob its details name. Uploads nothing holds yet aren't covered; they expire on their own.

One check runs at a time across the server. By default it reads at most 1,000 blobs and 20 MiB a second, so a full check of a big store doesn't hold up mail. Set other limits when you start it; 0 means no limit.

Reading the results

A check ends done, stopped, failed (with the reason) or interrupted, when its node restarted or stopped answering for five minutes. It finds four kinds of problem:

Problem What it means What to do
Missing The blob store has nothing under that id If the blob store setting ever changed, look at blob store changes first. Otherwise restore the blob from a backup
Damaged (full only) What's stored isn't what was written, or doesn't decompress Restore the blob from a backup taken before the damage
Unlinked A message names a blob but holds no link to it, so the blob cleanup can remove it Back it up while it's still there. The check says whether it is
Unreadable The blob store answered with an error Check that the store is reachable, and run the check again

Each problem names the account, what needed the blob (a message, a file, a queued message and so on), its id and the blob id. A missing blob also says whether it was ever completely stored. One that was stored and then lost points at the storage or at something that deleted files. One that never finished being stored points at a crash while it was written.

A check keeps its first 1,000 problems. The counts cover all of them, and every one is also logged as a store.data-corruption event, with the account, document and blob id. The latest 20 checks are kept.

Over the API, create one to start it:

["inbuxa:StoreCheck/set", {
  "accountId": "<your account id>",
  "create": { "c": { "mode": "full", "blobsPerSecond": 500 } }
}, "0"]

get returns its status, progress (0 to 1), the counts and the problems. An update with "action": "stop" stops it.

Snapshots

A snapshot is a copy of the whole store at one moment, taken while the server runs. A backup tool copies the snapshot, rather than the live store, which changes under it. That's the difference from an export, which needs the server stopped for a consistent copy.

It needs RocksDB as the data store, with blobs either in it (the default) or in a filesystem blob store. With PostgreSQL, MySQL or FoundationDB, use the database's own backup tools; a snapshot is refused and says so. Blobs in S3, Azure, a database or a sharded set are refused too: the snapshot couldn't hold them, and the blob cleanup could remove some before your backup tool copied them.

Where it goes

Under snapshots/ in the RocksDB directory, one directory per snapshot, named for when it was taken (UTC):

/var/lib/inbuxa/data/snapshots/
├── 20261110T013000Z/
│   ├── data/            the data store, as of that moment
│   ├── blobs/           a filesystem blob store's files, if blobs are separate
│   └── snapshot.json    when, by whom, the version, its size, how to restore
└── latest -> 20261110T013000Z

The location is fixed; nobody can ask for a snapshot elsewhere. On the store's own filesystem, files are hard-linked rather than copied, so a snapshot is quick and at first takes almost no space. It grows as the live store changes after it.

The latest two are kept: completing one deletes the older ones. A directory only appears under its plain name once it's complete, so a backup tool that copies snapshots/latest never gets half a snapshot. Not in it: a search index kept outside the store (Meilisearch, Elasticsearch and so on), which you rebuild after a restore, and the in-memory store, which a restarted server doesn't need.

Leave snapshots/ out of other copies

A backup that copies the whole RocksDB directory would copy every snapshot too. Exclude snapshots/, or copy only snapshots/latest.

Taking one from a backup job

Take a snapshot, wait for it to be done, then copy snapshots/latest. Over the API, a create takes no properties:

["inbuxa:StoreSnapshot/set", {
  "accountId": "<your account id>",
  "create": { "s": {} }
}, "0"]

The answer has its name and path. get returns its status: running, then done, or failed with the error. One snapshot is taken at a time. A refusal comes back in notCreated with the reason: another data or blob store, or a blob store change that isn't settled.

Restoring

  1. Stop the server.
  2. Move the RocksDB directory's contents aside, except snapshots/, and copy the snapshot's data/ into it.
  3. With a separate filesystem blob store, do the same with its directory and the snapshot's blobs/.
  4. Start the server. If the search index is kept outside the store, reindex the accounts once it's running.

The same steps are in each snapshot's snapshot.json.

Changing the blob store

Changing the blob store setting, under Settings › Storage › Stores › Blobs (messages & files), moves nothing. Everything already stored stays where it was. The server keeps track of it so no mail goes missing:

  1. When a node starts with a new blob store setting, it records a blob store change: from where, to where, when and which node. It logs a warning that names both places. Where a store is means its kind and location (a directory and its depth, a bucket and prefix, a database), never its credentials.
  2. Until the change is settled, the server reads through: a blob the new store doesn't have is looked for in the old one, returned, and kept in the new one. Each such read is a store.blob-read-through event. New blobs only go to the new store.
  3. You copy the rest. The copy goes through every blob the server still needs and writes the ones the new store doesn't have. It checks each message's content against its id as it goes, and doesn't copy one that doesn't match.
  4. A copy that got everything ends copied, and the change is settled. Every node stops reading through it within a minute.

While a change isn't settled, the console shows a banner on every page, and a snapshot is refused, because it would leave out blobs only the old store has.

In a cluster, change the setting on every node, then start the copy once every node runs the new setting. Until then, nodes still on the old setting write to the old store, and reading through covers it.

Copying

In the console, Copy on the change. Over the API, an update:

["inbuxa:BlobStoreChange/set", {
  "accountId": "<your account id>",
  "update": { "<change id>": { "action": "copy", "bytesPerSecond": 0 } }
}, "0"]

It has the same limits as a store check: by default 1,000 blobs and 20 MiB a second, and 0 is no limit. One copy runs at a time. "action": "stop" stops one; what it copied stays copied.

A copy that couldn't get everything ends incomplete, and the change stays unsettled. It keeps the first 1,000 problems and logs every one as store.data-corruption:

Problem Meaning
Missing Neither store has it. Restore it from a backup
Damaged The old store's copy doesn't match its id, so it wasn't copied. Restore it from a backup
Unreadable A store answered with an error. Fix that and copy again

Copying again skips what's already there.

Planning a change

On a large store, the copy after a change runs for hours, and until it ends, every first read of an older blob goes to the old store. To avoid that, plan the change first, and copy everything over while the current store is still the blob store:

  1. Plan it. In the console, Plan a change on the Blob store changes page, with the new store's settings (the same fields as the blob store setting). The server opens the store and writes and deletes a test blob in it before it records the plan, so a wrong bucket or password shows up now, not after a restart.
  2. Copy ahead. Copy ahead on the planned change copies everything from the current store into the new one, with the same checks and limits as any copy. Mail keeps going to the current store meanwhile, and nothing reads from the new one. A blob deleted from the current store is deleted from the new one too.
  3. Make it the blob store. Once a copy ahead ends copied, this writes the planned settings, credentials included, to the blob store setting. Nobody types them twice, and the setting can't point somewhere else by mistake.
  4. Restart the nodes. As a node starts with the new setting, the planned change becomes the change, and reading through covers what arrived since the copy ahead.
  5. Copy again. This last pass checks every blob, but transfers only those written between the copy ahead and the restart, and settles the change. On S3 or Azure, checking a blob is still a request, so it takes a while at the same limits, just with little data.

One change can be planned at a time, and none while another change isn't settled: copying ahead reads the current store only, so it would miss what only an older store has. Dismissing a planned change forgets it and its settings. What was copied ahead stays in the new store, and removing it is up to you.

Over the API, a plan is a create with the new store's settings as target, an x:BlobStore value as the blob store setting takes it. get never returns them, only where the store is:

["inbuxa:BlobStoreChange/set", {
  "accountId": "<your account id>",
  "create": { "plan": { "target": {
    "@type": "FileSystem", "path": "/srv/inbuxa/blobs", "depth": 2
  } } }
}, "0"]

A planned change has "stage": "planned". Copy ahead with "action": "copy" and apply with "action": "apply", as updates. With inbuxa-cli:

id=$(inbuxa-cli store plan new-store.json)
inbuxa-cli store copy "$id"
inbuxa-cli store apply "$id"
# restart every node, then:
inbuxa-cli store copy "$id"

Planning is in 27.1-rc.18 and later, and in inbuxa Admin 27.1-rc.11 and later.

Changing it back

Setting the blob store back to where it was before an unsettled change settles that change by itself, marked moved back: its old store is the current one again. The move back is a new change, and that's the one to copy, if anything was written in between.

Dismissing a change

Dismissing a change forgets it, and the server stops reading through to its old store. Nothing is deleted from either store. Dismiss an unsettled change only if you copied the old store's blobs over some other way, or don't need them: anything only the old store has will be missing.

Nothing is deleted from the old store unless you clean it up. Dismissing a change forgets its old store too, so clean up first if you mean to.

Cleaning up the old store

Once a change is copied, every blob the server holds is in the new store, and the old store still has a copy of each one. A directory or a bucket of its own you could delete by hand, but not the data store, or a bucket prefix shared with other things: the blobs sit among things that must stay. A cleanup has the server delete them from the old store, each one only after checking its copy in the new store.

Wait for your backups first. A backup of the data store taken before the change still has the old blob store setting, so once restored, it reads its blobs from the old store. After a cleanup, such a backup won't restore in full. Keep the old store until those backups have aged out. If you must restore one after a cleanup, point its blob store setting at the new store: the server records that as a change, and every blob it held when the cleanup ran is there.

In the console, Clean up the old store on a copied change. Preview goes through everything and deletes nothing: it counts what would be deleted and the space that frees, and lists any blob that would stay. Delete from the old store does it, once you've ticked that backups from before the change have aged out.

For each blob the server holds, the cleanup reads the new store's copy and checks it: message content against its id, anything else byte for byte against the old store's copy. Only then is it deleted from the old store. A blob that doesn't check out stays there, is logged as store.data-corruption, and is listed:

Problem Meaning
Not in the current store The new store doesn't have it. A full store check lists what it belongs to
Damaged in the current store The new store's copy doesn't match its id
Different in the current store The new store's copy isn't the same as the old store's
Unreadable A store answered with an error. Fix that and clean up again

A cleanup that kept some ends some kept. Put those blobs right and clean up again: what's already deleted is skipped. One that kept nothing ends done, and the server forgets the old store, so the change can't be cleaned up again. A cleanup can be stopped, and what it deleted stays deleted.

A cleanup is refused when:

  • the change isn't copied, or was moved back: its old store is the current one again;
  • the change was settled before 27.1-rc.19, which didn't keep the old store's settings;
  • the current blob store isn't the change's new store;
  • a node still has its blob store in the old place. Restart it with the new setting first. A node not heard from in ten minutes doesn't count;
  • the two stores overlap: two settings that reach the same place, or a sharded store with the other among its members;
  • a planned change would copy ahead into the old store;
  • a copy, a copy ahead or another cleanup is running.

What stays. Only the blobs the server knows about are deleted. Anything else in the old store stays: blobs nothing held any more when the change was made, a directory store's empty folders, whatever else shares a bucket or prefix. Once a cleanup is done, deleting what's left of a directory or a bucket of its own is up to you.

Getting the space back depends on the old store. The server doesn't compact or vacuum for you:

  • the data store on RocksDB frees it as it compacts, over the following hours;
  • on PostgreSQL, a vacuum (or autovacuum, in time) lets the server reuse it, and only VACUUM FULL returns it to the operating system, locking the table while it runs;
  • on MySQL, OPTIMIZE TABLE returns it;
  • a bucket with versioning keeps the deleted blobs as old versions until its own lifecycle rules remove them;
  • a snapshot taken before the change still holds the old store's blobs, so their space comes back only once that snapshot is deleted. The cleanup says how many there are.

Over the API, a preview is an update with dryRun, and the cleanup itself needs confirm set to the change's id:

["inbuxa:BlobStoreChange/set", {
  "accountId": "<your account id>",
  "update": { "<change id>": { "action": "cleanUp", "dryRun": true } }
}, "0"]
["inbuxa:BlobStoreChange/set", {
  "accountId": "<your account id>",
  "update": { "<change id>": { "action": "cleanUp", "confirm": "<change id>" } }
}, "0"]

It has the same limits as a copy, and "action": "stop" stops it. The change's cleanup holds the latest run, preview or not. With inbuxa-cli:

inbuxa-cli store clean-up "$id" --dry-run
inbuxa-cli store clean-up "$id"

Cleaning up is in 27.1-rc.19 and later, and in inbuxa Admin 27.1-rc.12 and later.