mapache

Fast, encrypted, deduplicating backup tool. Written in Rust.

Install

curl -fsSL https://github.com/jlantxa/mapache/raw/main/tools/install.sh | sh

Or grab a binary from the releases page.

Usage

# Initialize a repository
mapache init -r /path/to/repo

# Create a snapshot
mapache snapshot ~/Documents -r /path/to/repo

# Launch the TUI
mapache tui -r /path/to/repo

# List snapshots
mapache log -c -r /path/to/repo

# Restore
mapache restore --target /tmp/restore -r /path/to/repo

Features

• Parallel backup and restore — Multi-threaded pipeline with configurable concurrency for readers and packers.
• Encryption — AES-256-GCM-SIV with Argon2id key derivation. Data is never stored or transmitted in the clear.
• Deduplication — Content-defined chunking (FastCDC) across all snapshots and machines. Only new data is stored.
• Backends — Local filesystem, SFTP, and S3-compatible object storage.
• Terminal UI — Interactive TUI with dashboard, snapshot and restore wizards, file explorer, and diff viewer.
• Bundles — Self-contained encrypted .mapache files for transfer or cold storage.
• FUSE mount — Browse snapshot contents as a mounted filesystem (Unix).
• Retention policies — Hourly, daily, weekly, monthly and yearly rules. Filter by host and tag.
• Multi-client — Non-exclusive locking allows multiple hosts to back up to the same repository.
• Portable — Linux, macOS, Windows and Android. Single self-contained binary for each platform (statically linked on Linux and Windows).

Mapache User Manual

mapache — a fast, secure, deduplicating, incremental backup tool.

Table of Contents

1. Introduction
2. Installation
3. Quick Start
4. Concepts
5. Configuration
6. Hooks
7. Repository Management
8. Taking Snapshots
9. Restoring Data
10. Exploring Snapshots
11. Retention Policies
12. Maintenance
13. Security & Key Management
14. Bundle Files
15. FUSE Mount
16. Snapshot Lifecycle
17. Terminal User Interface
18. Shell Completions
19. Environment Variables
20. Global Options Reference
21. Full Command Reference

1. Introduction

Mapache (Spanish for “raccoon”) is a high-performance backup tool written in Rust. It uses a content-addressable repository model: every file, directory, and piece of metadata is decomposed into binary objects identified by their cryptographic hash (BLAKE3). This enables global deduplication — if multiple files or snapshots share the same content, even across different machines, the data is stored only once.

Key properties:

• Deduplication — FastCDC (Content-Defined Chunking) splits files into variable-size chunks; only new chunks are stored.
• Encryption — Mandatory AES-GCM-SIV authenticated encryption. Data is never stored or transmitted in cleartext.
• Compression — Zstd compression with adjustable levels, or no compression at all for already-compressed data (video, photos).
• Snapshots — Every backup is a point-in-time snapshot. Snapshots are logically independent (no “full vs. incremental” chain) but share underlying data blobs.
• Portable — Single self-contained binary with no runtime dependencies (statically linked on Linux and Windows; macOS binaries link Apple’s system libraries).
• Backends — Local filesystem, SFTP, and S3-compatible object storage.
• Verifiable — Built-in integrity verification at every level.

Project Status

Mapache is feature-complete and production-oriented, but relatively new. Users should validate their own backup/restore workflows before relying on it for mission-critical data.

2. Installation

Quick Install (Linux, macOS, Windows)

curl -fsSL https://github.com/jlantxa/mapache/raw/main/tools/install.sh | sh

Downloads the latest release binary for your platform and installs it to /usr/local/bin/mapache. Set VERSION or INSTALL_DIR to customize.

Pre-built Binaries

Download binaries from the Releases page. Linux and Windows binaries are statically linked and require no dependencies; macOS binaries link Apple’s system libraries.

Install from crates.io

cargo install mapache

Builds the latest published version from source and links it against your system’s libraries (glibc on Linux, system frameworks on macOS). Requires the Rust toolchain and a C toolchain, but no other build dependencies. To install a specific version:

cargo install mapache --version 0.7.1

macOS: the mount feature needs FUSE on your machine, which is up to you to install — brew install --cask macfuse. This is the same deal as Linux, where mounting needs the system’s fuse3 package. macFUSE is required both to build and to run; without it, install without mount: cargo install mapache --no-default-features. The pre-built binaries include mount and only need macFUSE at runtime.

Build from Source

Requires the Rust toolchain.

# Build release binary
cargo build --release

# Install to cargo bin path
cargo install --path crates/mapache

For a statically-linked build:

make release-static

Feature Flags

• mount (default) — FUSE mount support (Unix). FUSE is a user-provided prerequisite: Linux needs the fuse3 package (the fusermount helper is used at runtime); macOS needs macFUSE (brew install --cask macfuse), both to build and to run.

To build without FUSE:

cargo build --release --no-default-features

To build with all features:

cargo build --release --all-features

Docker

A Docker image with all cross-compilation toolchains is available at tools/docker/. See the README or CONTRIBUTING for usage.

3. Quick Start

Initialize a Repository

# Local directory
mapache init -r /backup/myrepo

# SFTP server
mapache init -r sftp://user@host/backup

# S3 bucket
mapache init -r s3://my-bucket/backups

You will be prompted for a username and password. Do not lose this password — it is the only way to access your data.

Create a Snapshot

mapache snapshot ~/Documents -r /backup/myrepo

# Or pipe data directly via --stdin
tar czf - ~/Documents | mapache snapshot --stdin -r /backup/myrepo

List Snapshots

mapache log -c -r /backup/myrepo

Restore a Snapshot

# Restore the latest snapshot
mapache restore --target /tmp/restore -r /backup/myrepo

Apply Retention and Clean Up

# Keep last 7 daily snapshots, then garbage-collect
mapache forget --keep-daily 7 --clean -r /backup/myrepo

4. Concepts

Repository

A repository is a directory (local or remote) that stores all backup data. It contains objects, pack files, indices, keys, snapshots, and locks.

repo/
├── index/          # Pack index files (blob → pack mapping)
├── keys/           # User key files
├── locks/          # Runtime lock files
├── manifest        # Repository metadata (ID, version, creation time)
├── objects/        # Pack files (00/..ff/ fanout)
└── snapshots/      # Snapshot metadata files

Content-Addressable Storage

Every blob (chunk of data or tree metadata) is identified by the BLAKE3 hash of its raw content. This means:

• Identical content produces the same hash → deduplication.
• Any change in content produces a different hash → integrity verification.

Zero-filled regions are deduplicated separately from chunking: mapache recognizes all-zero content and stores only a marker (a Zero blob with length 0 in the pack footer) instead of the bytes. Any number of zero regions — in one file or across many — share the same marker and consume no pack space.

Pack Files

Blobs are not stored individually. They are grouped into pack files (default 16 MiB each) for I/O efficiency. A pack file contains multiple encoded (compressed + encrypted) blobs and a footer that describes their offsets and sizes.

Index

The index maps blob IDs to their physical location (pack ID + offset + length). It is the source of truth — any blob not in the index is considered non-existent and subject to garbage collection.

Index Loading Modes

By default (and with --index-mode eager) all index files are loaded into RAM for the fastest lookups. With --index-mode lazy, mapache keeps only the most recently used indices resident — a “hot pool” bounded by runtime.lru-max-blobs (default 1,000,000 blobs) — and tracks the rest as lightweight cold entries, loading them from disk on demand. Lazy mode reduces RAM usage on large repositories at the cost of re-reading cold indices; it can be set per-command with --index-mode or globally in the [global] config section.

Snapshots

A snapshot is a point-in-time record of your files. It contains:

• The root tree ID (linking to all file/directory metadata)
• Timestamp, hostname, username
• Paths, tags, description
• Summary statistics (processed bytes, dedup counts, diff counts)

Snapshots are logically independent. Deleting an old snapshot does not affect newer ones (shared blobs are reference-counted by the index).

Chunking

Mapache uses FastCDC (Content-Defined Chunking) with:

• Minimum chunk: 512 KiB
• Average chunk: 1 MiB
• Maximum chunk: 8 MiB
• Normalization: L2

Chunker parameters are stable across versions to ensure consistent chunk boundaries.

Encryption

All data (packs, indices, snapshots, manifest) is encrypted with AES-GCM-SIV using a randomly-generated 256-bit master key. User passwords are derived via Argon2id to produce wrapping keys that encrypt the master key.

Error Correction (ECC)

Mapache supports optional Reed-Solomon erasure coding to protect pack files against bit-rot and silent data corruption. When enabled, each pack file gets a parity-only sidecar (.ecc file) that stores redundant parity shards.

Key properties:

• No data duplication — sidecars store only parity shards, not the original data. A 50% ECC overhead adds ~50% of the data size in parity storage.
• Per-stripe processing — data is split into stripes of up to 200 shards. Each stripe is independently encoded, limiting memory to ~800 KB regardless of pack size.
• Transparent to hashing — the BLAKE3 hash is computed on the original pack content. ECC sidecars are invisible to the content-addressable layer.
• Compressed and encrypted — like all repository objects, ECC sidecars are compressed and encrypted on disk.
• Repair on verify — verify --repair detects corruption via parity mismatch and reconstructs data using the remaining intact shards.

ECC is configured per-repository at initialization time with --ecc <PERCENT>.

5. Configuration

Environment Variables

TOML Config File

Mapache can read settings from a TOML configuration file specified via --with-config <PATH>. The file supports [global], [snapshot], [restore], [forget], [runtime], and [hooks] sections.

Generate the canonical template with:

mapache config template                             # Print to stdout
mapache config template --output ./mapache.toml     # Write to file

All values are commented out; uncomment and adjust as needed.

Example configuration:

[global]
repo = "s3://my-bucket/backups"
no-cache = false
ssh-privatekey = "~/.ssh/id_ed25519"
ssh-known-hosts = "~/.ssh/known_hosts"
auth-file = "~/.mapache/auth"
pack-size = 16.0
key-file = "~/.mapache/repo.key"
quiet = false
compression = "fast"
retry-lock = "5m"
limit-upload = "10 MiB/s"
limit-download = "50 MiB/s"
index-mode = "eager"

[snapshot]
paths = ["/home/user/Documents"]
exclude = ["**/node_modules", "**/.git"]
tags = "work,important"
one-file-system = false
fail-on-skipped = false
skip-if-unchanged = false
readers = 4
packers = 4

[restore]
strategy = "newer"
sparse = true
verify = true

[forget]
keep-daily = 7
keep-weekly = 4
keep-monthly = 6
keep-yearly = 2
keep-min = 3

[runtime]
restore-pack-prefetch = 4
restore-blob-concurrency = 8
restore-max-open-files = 128
min-pack-size-factor = 0.05
gc-repack-concurrency = 4
blobs-per-index-file = 65535
index-flush-timeout-secs = 600
lru-max-blobs = 1000000
s3-multipart-threshold = 134217728
s3-multipart-part-size = 134217728

[hooks.snapshot.pre]
command = "echo 'Starting snapshot'"

[hooks.snapshot.post]
command = "echo 'Snapshot finished: $MAPACHE_RESULT'"

CLI flags override config file values. Config file values supplement CLI lists (e.g., exclude paths from both sources are combined).

Authentication

Authentication can be provided via:

1. Interactive prompt (default) — prompts for username and password
2. --auth-file <PATH> — file with username on the first line and password on the second line
3. MAPACHE_USERNAME / MAPACHE_PASSWORD environment variables
4. --key-file <PATH> — path to an external key file (still requires username and password for authentication)

When authenticating interactively, mapache allows 3 password attempts before failing.

6. Hooks

Mapache supports running custom shell scripts before and after certain commands. This is useful for notifications, logging, mounting/unmounting filesystems, or any pre/post processing.

Configuration

Hooks are defined in the TOML config file under [hooks.<command>] sections:

[hooks.snapshot.pre]
command = "echo 'Starting snapshot'"
timeout = 300

[hooks.snapshot.post]
command = "/usr/local/bin/notify.sh"

Each hook can specify:

Pre and Post Hooks

• Pre hooks run before the command starts. If the hook exits with a non-zero status, the command is aborted immediately.
• Post hooks run after the command finishes, regardless of success or failure. Non-zero exit status only logs a warning — the command is not affected.

Supported Commands

Hooks can be configured for: snapshot, restore, forget, clean, and verify.

CLI Override

The --pre-hook and --post-hook flags can be passed on the command line to override the TOML-configured hook for a single invocation. An empty string falls through to the TOML hook. When a CLI override is used, the timeout from the TOML config is not applied.

mapache snapshot /data --pre-hook "echo starting"
mapache snapshot /data --post-hook ""   # falls through to TOML hook

Environment Variables

The following variables are passed to every hook script:

For security, the MAPACHE_USERNAME and MAPACHE_PASSWORD environment variables are stripped from the hook environment and are never accessible to hook scripts.

Timeout

When a timeout is configured and the hook exceeds it, the child process is killed immediately. For pre hooks, this is treated as a failure and the command is aborted. For post hooks, the timeout is only logged as a warning and does not affect the command result.

Shell Execution

Hook commands are executed via sh -c on Unix and cmd /C on Windows.

Dry Run

When a command is run with --dry-run, hooks are not executed.

Example

[hooks.snapshot.pre]
command = "df -h /backup > /tmp/disk-space-before.txt"

[hooks.snapshot.post]
command = """
  if [ "$MAPACHE_RESULT" = "success" ]; then
    curl -s -X POST https://hooks.example.com/backup-ok \
      -d "repo=$MAPACHE_REPOSITORY&cmd=$MAPACHE_COMMAND"
  fi
"""

[hooks.clean.pre]
command = "/usr/local/bin/maintenance-start.sh"
timeout = 60

[hooks.clean.post]
command = "/usr/local/bin/maintenance-end.sh"

7. Repository Management

init — Initialize a Repository

mapache init -r <URL>
mapache init --format 2 -r <URL>   # Explicit v2 (default)
mapache init --format 1 -r <URL>   # Legacy v1 (deprecated)
mapache init --ecc 50 -r <URL>     # Enable ECC with 50% overhead
mapache init --calibrate-kdf -r <URL>  # Tune Argon2id for this hardware

Creates a new empty repository at the given location. Supported URL schemes:

• Local: /path/to/repo or local:/path/to/repo
• SFTP: sftp://user@host:port/path
• S3: s3://bucket-name/path

SSH key support: mapache supports Ed25519 and ECDSA keys for SFTP authentication. RSA keys are not supported — use ssh-keygen -t ed25519 to generate a compatible key pair.

You will be prompted for a username and password. A manifest file is created with a unique repository ID and the current mapache version. Do not lose your password — there is no password recovery mechanism.

Note: Format v1 is deprecated and will be unsupported in a future release. New repositories should use v2 (the default). To migrate an existing v1 repository, see migrate.

unlock — Remove Stale Locks

mapache unlock -r <URL>
mapache unlock --force -r <URL>

Locks are created when mapache opens a repository and automatically released on clean shutdown. If a process crashes, stale locks may remain. By default, unlock removes only expired locks. Use --force to remove all locks (including non-expired ones).

cache — Manage Local Cache

mapache cache
mapache cache --delete <PREFIX> [<PREFIX>...]
mapache cache --clear

Mapache caches metadata (tree blobs, snapshots, indices) locally to speed up subsequent operations on remote backends. The cache command lists cache directories by repository ID, deletes specific caches by prefix, or clears all caches.

copy — Transfer Snapshots Between Repositories

mapache copy --all --from <URL> -r <URL>
mapache copy --all --from <URL> --dry-run -r <URL>
mapache copy --from <URL> --snapshot ID --host HOST -r <URL>
mapache copy --from <URL> --tags WORK --snapshot ID -r <URL>

Copies snapshots (and their data) from one repository to another. Useful for migration, creating replicas, or consolidating backups. At least one of --all, --snapshot, --host or --tags must be given to select which snapshots to copy.

The copy operation:

1. Connects to both source (--from) and destination (-r) repositories
2. Lists all snapshots in the source, applying the given filters (--all, --snapshot, --host, --tags)
3. Skips snapshots already present in the destination (by ID)
4. Walks every referenced tree to discover all required data blobs
5. Filters out blobs already present in the destination (blob-level dedup)
6. Transfers only missing blobs, re-packing them at the destination
7. Writes snapshot metadata to the destination

Snapshots are re-encoded under the destination repository’s master key, so their file IDs are re-derived at the destination (blob IDs stay identical, as they hash the plaintext). Any parent/amends links between copied snapshots are remapped to the new IDs.

Source and destination must use the same format version (v1 and v2 cannot be mixed). Use mapache migrate to upgrade a v1 repository to v2.

Supports --json for machine-readable progress events.

sync — Replicate a Repository

mapache sync --target <URL> -r <URL>
mapache sync --target <URL> --delete -r <URL>
mapache sync --target <URL> --dry-run -r <URL>

Synchronizes a repository to a different backend. Useful for creating replicas or migrating between storage providers.

The sync operation:

1. diffs source vs destination
2. deletes obsolete files (destination-only)
3. copies missing or changed files
4. synchronizes the manifest last to ensure destination validity.

Source and destination must use the same format version (v1 and v2 cannot be mixed): sync replicates the raw storage layout, so version-mixed replicas would be unreadable. Use mapache migrate to upgrade a v1 repository to v2.

rebuild-index — Disaster Recovery

mapache rebuild-index
mapache rebuild-index --dry-run

Rebuilds the master index from scratch by scanning all existing pack files and parsing their footers. Used for disaster recovery when the index is lost or corrupted but all pack files are intact.

The command will:

1. List all pack files in the repository
2. Parse each pack footer to discover blobs
3. Build a new master index
4. Persist the new index
5. Delete old/replaced index files

migrate — Migrate Repository Format

mapache migrate -r <URL>
mapache migrate --dry-run -r <URL>

Migrates a repository from format v1 to v2. The migration performs four steps:

1. Re-encrypt packs — Every pack file is re-encrypted with the nonce placed at the end (v2) instead of the start (v1). This changes each pack’s ID (the hash depends on the ciphertext).
2. Re-encrypt snapshots — Snapshot metadata files are re-encrypted with the new nonce position.
3. Rebuild index — A new binary index (v2 format) is built from the re-encrypted packs, replacing the old JSON index files.
4. Update manifest — The manifest version is updated to 2, completing the migration atomically.

The original files are kept while the new packs, snapshots, and index are written. Updating the manifest is the migration’s commit point; only after it succeeds are the old files removed. This makes the migration safe to interrupt: before the commit point, the original repository remains intact, and after it, the repository uses the new format.

Important: Back up your repository before migrating. While the migration is designed to be safe, having a backup is always recommended for destructive operations.

8. Taking Snapshots

Basic Usage

mapache snapshot PATH [PATH...] -r <URL>

One or more paths can be specified. Mapache backs up each path, deduplicating shared content across all of them.

Common Options

With --one-file-system, each explicitly supplied source path starts its own filesystem boundary. Entries on another device are not traversed. The option requires filesystem device IDs: the device ID from lstat on Unix, the volume serial number on Windows. Snapshot creation fails if they cannot be determined for a source path.

Exclusions

Exclusions support glob patterns (**, *, ?) as well as literal paths. Non-glob paths are normalized to absolute paths; glob patterns are kept as-is for matching. They can be specified inline or in a file:

# Inline exclusions
mapache snapshot ~/project --exclude "~/project/target","~/project/.git"

# Exclusion file (one path per line)
mapache snapshot ~/project --exclude-file ~/.mapache-exclude

Tags

Tags are comma-separated labels used for filtering snapshots in log, forget, and other commands.

mapache snapshot ~/Documents --tags "work,important"

Incremental Backups

By default, mapache uses the latest snapshot as the parent for incremental diffing. Only changed/new files are processed; unchanged files are skipped.

# Explicitly set the parent snapshot
mapache snapshot ~/Documents --parent latest

# Force a full scan (ignore parent)
mapache snapshot ~/Documents --no-parent

# Use a specific snapshot as parent
mapache snapshot ~/Documents --parent abc1234f

Skip If Unchanged

When --skip-if-unchanged is set, mapache compares the new snapshot’s root tree with the parent’s. If they match, no new snapshot file is created.

Parallelism

• --readers: Controls how many files are read and chunked concurrently.
• --packers: Controls how many threads compress, encrypt, and write pack files concurrently.

Higher values improve throughput on fast storage but consume more memory.

Dry Run

Using --dry-run shows what would be backed up without writing any data:

mapache snapshot ~/Documents --dry-run -r /backup/myrepo

Hooks are also skipped during dry runs (see Hooks).

Stdin Backup

Use --stdin to back up data from a pipe or redirected file as a single virtual file at /stdin:

# Backup a database dump
pg_dump mydb | mapache snapshot --stdin -r /backup/myrepo

# Backup a tar archive
tar czf - /important/data | mapache snapshot --stdin -r /backup/myrepo --tags "archive"

# Redirect a file
mapache snapshot --stdin -r /backup/myrepo < /path/to/largefile.bin

The data is chunked, deduplicated, compressed, and encrypted identically to regular files. The --stdin flag is mutually exclusive with paths, excludes, and --parent.

Important: --stdin requires non-interactive authentication (stdin is consumed by backup data, so the password cannot be prompted interactively). Use either --auth-file or MAPACHE_USERNAME/MAPACHE_PASSWORD env vars:

# Option 1: auth file
echo "myusername" > ~/.mapache-auth
echo "mypassword" >> ~/.mapache-auth
mapache snapshot --stdin -r /backup/myrepo --auth-file ~/.mapache-auth

# Option 2: environment variables
MAPACHE_USERNAME=myusername MAPACHE_PASSWORD=mypassword \
  mapache snapshot --stdin -r /backup/myrepo

Snapshot Output

After a successful snapshot, mapache displays:

• Changes since parent (new/changed/deleted/unchanged files and dirs)
• Data and metadata sizes (raw vs compressed)
• The new snapshot ID

9. Restoring Data

Basic Usage

mapache restore [SNAPSHOT] --target <DIR> -r <URL>

Restores a snapshot to the given target directory. The SNAPSHOT argument can be a snapshot ID prefix or latest (default).

Options

Conflict Strategies

The same strategy applies to conflicting symlinks: skip keeps the existing symlink with a warning, overwrite replaces it, fail aborts, and newer compares modification times.

Path Filtering

Restore supports both include and exclude filtering:

# Restore only specific paths
mapache restore --target /tmp/restore --include "docs/,src/main.rs"

# Exclude certain paths
mapache restore --target /tmp/restore --exclude "*.log,*.tmp"

# Use filter files
mapache restore --target /tmp/restore --include-file restore-list.txt

Strip Prefix

When restoring a subset of paths, --strip-prefix removes the common ancestor directory:

# Snapshot contains: /home/user/project/src/main.rs
# Without --strip-prefix: /tmp/restore/home/user/project/src/main.rs
# With --strip-prefix:    /tmp/restore/src/main.rs

Delete Mode

--delete removes files in the target directory that do not exist in the snapshot. Use with caution. --no-preserve-root extends deletion to the root level of the restored tree.

Sparse Files

--sparse creates sparse files instead of preallocating space. This is faster on some filesystems but may cause fragmentation.

Verification

By default, restore decides whether to overwrite a file based on the chosen strategy alone. Use --verify to enable content-level verification: the restorer reads each existing local file, hashes every blob, and compares the result against the repository index.

When --verify is active with overwrite or newer, the restorer performs incremental restore: each blob is individually compared against the local file content. Only blobs whose hash differs from the on-disk data are downloaded and written; unchanged blobs are skipped. This significantly speeds up repeated restores of large files where only a small fraction of the content changed.

If verification fails (e.g. unreadable file), the restorer falls back to the strategy’s default behavior.

Dry Run

mapache restore --target /tmp/restore --dry-run

Simulates the full restore process (including conflict detection, path filtering, and deletion) without touching the filesystem. Hooks are also skipped (see Hooks).

Restore Output

Displays progress, a summary of errors and warnings, and total duration.

10. Exploring Snapshots

log — List Snapshots

mapache log -r <URL>
mapache log -c -r <URL>           # Compact listing
mapache log --all -r <URL>        # Show active + dropped
mapache log --dropped -r <URL>    # Show only dropped
mapache log --tags "work" -r <URL> # Filter by tags
mapache log SNAPSHOT_ID -r <URL>  # Show specific snapshot

Full listing shows: ID, date, size, root, hostname, username, tags, paths, and description. Compact mode (-c) shows a summary table.

ls — Browse Snapshot Contents

mapache ls -r <URL>                        # List root of latest snapshot
mapache ls SNAPSHOT_ID -r <URL>            # List root of specific snapshot
mapache ls --path /home/user/docs -r <URL> # List specific path
mapache ls -l -r <URL>                     # Long listing (permissions, size, mtime)
mapache ls -H -r <URL>                     # Human-readable sizes
mapache ls -R -r <URL>                     # Recursive listing

find — Search Snapshots

mapache find PATTERN -r <URL>              # Search in all snapshots
mapache find PATTERN --snapshot latest -r <URL>  # Search in latest only
mapache find PATTERN --snapshot ID -r <URL>      # Search in specific snapshot
mapache find PATTERN --json -r <URL>       # JSON output

Searches file and directory names by pattern across one or all snapshots. When the pattern does not contain a /, the search is recursive and matches in any subdirectory. Use a leading / (e.g. find /file.txt) to restrict the search to the root directory of each snapshot.

Supports --json for machine-readable output. Exits with a non-zero code on error.

diff — Compare Snapshots

mapache diff SOURCE_ID TARGET_ID -r <URL>

Shows the differences between two snapshots. Symbol legend:

Output includes a summary table and a size comparison.

stats — Repository Statistics

mapache stats -r <URL>
mapache stats --full -r <URL>   # Parse pack footers (expensive)

Shows:

• Packs: count, total size, optional footer analysis (blob count, encoded vs raw sizes, dangling blobs)
• Index: count, size, indexed blobs, raw/encoded sizes
• Snapshots: count, total snapshot metadata size
• Snapshot reference summary: referenced blobs (data + tree), raw/encoded sizes, compression ratios, total restorable size
• Unreferenced blobs (shown only if >0): blobs tracked in the index but not referenced by any snapshot, with their total reclaimable encoded size
• Keys: count, total size
• Manifest: size
• Total repository size

--full parses every pack footer, giving physical blob statistics and dangling blob detection (expensive on large repositories).

dump — Print File Contents

mapache dump SNAPSHOT_ID --path /path/to/file -r <URL>
mapache dump latest --path /path/to/file -r <URL>

Prints the contents of a single file from a snapshot to stdout. The SNAPSHOT_ID argument accepts ID prefixes or latest.

cat — Print Repository Objects

mapache cat manifest -r <URL>
mapache cat snapshot:ID -r <URL>
mapache cat pack:ID -r <URL>
mapache cat blob:ID -r <URL>
mapache cat tree:ID -r <URL>
mapache cat index:ID -r <URL>
mapache cat key:ID -r <URL>
mapache cat lock:ID -r <URL>

Prints a decoded representation of the selected repository object, which is useful for debugging and manual inspection. Structured objects are printed as JSON; blob:ID prints the blob as UTF-8 text and fails if its contents are not valid UTF-8. This is not the raw on-disk representation.

11. Retention Policies

forget — Remove Snapshots

mapache forget [SNAPSHOT_IDS...] -r <URL>     # Remove specific snapshots
mapache forget --keep-daily 7 -r <URL>         # Apply retention rules
mapache forget --keep-last 5 --clean -r <URL>  # Keep + GC
mapache forget --force SNAPSHOT_ID -r <URL>    # Permanently delete

Retention Rules

By default, forget marks snapshots as dropped (renames to .dropped). Use --force for immediate deletion.

A snapshot is kept if any rule applies. Rules are inclusive, not exclusive.

Filtering

# Only consider snapshots with specific tags
mapache forget --keep-daily 7 --tags "work"

# Only consider snapshots from specific hosts
mapache forget --keep-daily 7 --host my-server

Garbage Collection Integration

# Run GC after forget
mapache forget --keep-daily 7 --clean

# Custom GC tolerance
mapache forget --keep-daily 7 --clean --tolerance 10

Dry Run

mapache forget --keep-daily 7 --dry-run

Shows which snapshots would be kept and removed without making changes. Hooks are also skipped during dry runs (see Hooks).

12. Maintenance

clean — Garbage Collection

mapache clean -r <URL>               # Full GC with 0% tolerance
mapache clean --tolerance 10 -r <URL> # Tolerate 10% garbage per pack
mapache clean --no-repack -r <URL>    # Only remove unreferenced packs
mapache clean --dry-run -r <URL>      # Simulate

The garbage collector:

1. Scans all snapshots to determine which blobs are still referenced.
2. Identifies:
   • Unused packs — packs where no blobs are referenced (removed).
   • Obsolete packs — packs with blobs that will be repacked.
   • Small packs — packs smaller than the minimum size threshold (repacked).
   • Tolerated packs — packs with garbage below the tolerance threshold (kept as-is).
3. Repacks small packs and obsolete-pack survivors into new pack files.
4. Removes orphaned index entries.

--no-repack skips repacking (equivalent to tolerance = 100%).

The garbage collector can be safely interrupted with SIGINT/SIGTERM (Ctrl+C). The shutdown signal is checked at safe checkpoints between GC phases, so no data corruption occurs. This also applies to forget --clean.

Hooks are skipped when running with --dry-run (see Hooks).

verify — Integrity Verification

mapache verify -r <URL>                    # Logical consistency check
mapache verify --read-packs -r <URL>       # Full physical verification
mapache verify --read-packs --sample 10%   # Verify 10% of packs
mapache verify --read-packs --parallel 8   # Parallel verification
mapache verify --repair                    # Repair corrupt packs via ECC
mapache verify --fail-early                # Stop on first error
mapache verify --with-cache                # Use local cache
mapache verify --dump-pack-blobs packs.txt # Dump every blob descriptor to file

Logical Verification (default)

• Checks that every snapshot references known tree blobs.
• Checks that every indexed blob points to an existing pack file.
• Verifies snapshot tree traversal integrity.

Metadata Verification

Index, snapshot, and manifest files are always checked: each one is read back and validated against its content hash. With --repair and ECC enabled, a corrupt metadata file is repaired from its parity sidecar when possible.

Physical Verification (--read-packs)

• Reads, decrypts, and hashes every blob in every pack (or a sample).
• Detects bit-rot (pack file hash mismatch) and individual blob corruption.
• Reports corrupt blobs and the files/snapshots they affect.

ECC Repair (--repair)

When ECC is enabled on the repository, --repair checks each pack against its parity sidecar. The sidecar is decoded (decompressed + decrypted) before use. Corrupt data shards are reconstructed from the remaining intact shards using Reed-Solomon erasure coding. Repair reports all results but never aborts early — it attempts to repair every pack regardless of individual failures.

Seven outcome cases per pack:

Sampling

--sample <PCT> verifies only a random percentage of packs. Useful for periodic integrity checks on large repositories.

Dump pack blobs (--dump-pack-blobs)

--dump-pack-blobs <FILE> writes every blob descriptor from every pack footer to a plain-text file. Each line contains <blob_id_hex> <blob_type> <pack_id_hex> for one descriptor. The flag runs with bounded memory (one pack at a time), so it is safe on large repositories.

Useful for auditing what each pack physically contains, detecting phantom descriptors, or identifying unreferenced blobs offline with sort | uniq -w 64.

repack — Re-encode All Blobs

mapache repack -r <URL> --compression best

Re-encodes all reachable blobs using the current compression and pack-size settings while preserving chunk IDs and snapshots. This is the way to change the compression of an existing repository after the fact (e.g. --compression none for content that does not compress, best for archival).

Old packs are removed after the replacement index is persisted. Repack requires an exclusive lock and may take significant time on large repositories.

rechunk — Recalculate Snapshot Chunks

mapache rechunk -r <URL>

Reconstructs every file from its snapshot blobs, recalculates its chunk boundaries with the current chunker, and writes new blobs using the current compression and pack-size settings.

Existing blobs are not deleted: blobs superseded by the rechunk remain in place until a later clean reclaims the space. Rechunk requires an exclusive lock and may take significant time on large repositories.

13. Security & Key Management

Mapache uses a dual-layer encryption model:

1. Master Key — A random 256-bit AES-GCM-SIV key stored encrypted inside key files. This key encrypts all repository data.
2. User Keys — Derived from username + password via Argon2id KDF. Each user key encrypts the master key. Multiple users can access the same repository.

Key files can be stored inside the repository (keys/ directory) or exported as standalone files.

key list — List Keys

mapache key list -r <URL>

Shows all key files in the repository: username, key ID, creation date.

key add — Add a New User

mapache key add -r <URL>                     # Store key in repository
mapache key add --path ~/my-backup-key -r <URL>  # Export key file locally

Requires an existing user’s credentials to decrypt the master key, then prompts for the new user’s username and password.

key delete — Remove a User

mapache key delete KEY_ID -r <URL>          # Requires confirmation
mapache key delete KEY_ID --yes -r <URL>    # Skip the confirmation prompt

Deletes a key file by ID prefix. The affected user will no longer be able to access the repository. Deletion requires confirmation unless --yes is given.

key change-password — Change Password

mapache key change-password -r <URL>

Prompts for current credentials, then for a new password. Generates a new key file and deletes the old one.

key export — Export a Key File

mapache key export KEY_ID -r <URL>           # Exports to ./keyfile
mapache key export KEY_ID --output ~/safe/keyfile -r <URL>

Exports a key file from the repository to a local path. This file can be used later with --key-file to authenticate without username/password.

Security Best Practices

• Never lose your password — there is no recovery mechanism.
• Export key files to a secure location for disaster recovery.
• Use --key-file to authenticate without typing passwords (useful for scripts).
• The repository is a black box — all data is encrypted. An attacker with full read access to the storage cannot decrypt any content without the master key.

Renewing the Master Key

Mapache does not rotate the master key in place. If the master key (or a user password that wraps it) is suspected to be compromised, the sanctioned procedure is to recreate the repository under a fresh master key and copy the data over:

1. Initialize a new repository with a new master key: mapache init -r <NEW_URL>
2. Copy every snapshot (this re-encrypts all data under the new master key): mapache copy --all -r <NEW_URL> --from <OLD_URL>
3. Verify the new repository: mapache verify -r <NEW_URL> and mapache log -r <NEW_URL>.
4. Destroy the old repository and all copies of its key files.

Note that any data already extracted under the old master key remains readable; recreation only protects future access. Snapshot IDs are re-derived in the new repository (they hash the encrypted bytes, which differ under the new key), but all content is preserved.

14. Bundle Files

The bundle command creates, extracts, imports, exports, or mounts self-contained .mapache bundle files. Bundles are encrypted, deduplicated archives that can be transferred to other systems.

Create a Bundle

# Create a bundle from files/directories
mapache bundle -b PATH [PATH...] -o bundle.mapache

# With exclusions
mapache bundle -b ~/project -o project.mapache -e "*.log" -e "tmp/"

# Custom compression
mapache bundle -b ~/Documents -o docs.mapache --compression best

# Use a directory as the bundle root (its children become top-level items)
mapache bundle -b ~/Photos -o photos.mapache --as-root

# Enable ECC protection (10% overhead)
mapache bundle -b ~/Documents -o docs.mapache --ecc 10

Extract a Bundle

mapache bundle -x bundle.mapache -o /tmp/extract

If -o is omitted, extracts to the current directory. If the bundle was created with ECC (--ecc), corrupted blobs are automatically repaired during extraction using Reed-Solomon erasure coding — no manual intervention needed.

Export a Snapshot to a Bundle

# Export the latest snapshot
mapache bundle --export-snapshot latest -o backup.mapache -r /path/to/repo

# Export by snapshot ID prefix
mapache bundle --export-snapshot a1b2c3d4 -o backup.mapache -r /path/to/repo

The exported bundle contains all data from the snapshot. The bundle’s blob IDs match the repository, so importing the bundle into the same (or another) repository will deduplicate automatically against existing blobs.

Import a Bundle as a Snapshot

# Import a bundle as a snapshot into the repository
mapache bundle -i bundle.mapache -r /path/to/repo

# Import multiple bundles at once (each becomes a separate snapshot)
mapache bundle -i bundle1.mapache bundle2.mapache -r /path/to/repo

Blobs already present in the repository are skipped, making this efficient for cross-repo transfers. Each imported snapshot appears in log with a description recording which bundle file it came from.

Mount a Bundle (FUSE, Unix only)

mapache bundle -m bundle.mapache /mnt/bundle
mapache bundle -m bundle.mapache /mnt/bundle -c      # Auto-create mountpoint
mapache bundle -m bundle.mapache /mnt/bundle --metadata-only

Bundle-specific options:

Bundles are password-protected (AES-GCM-SIV). The password is independent from the repository password.

15. FUSE Mount

Mount a Repository (Unix only, mount feature)

mapache mount /mnt/backup -r <URL>
mapache mount /mnt/backup -r <URL> --allow-other
mapache mount /mnt/backup -r <URL> --metadata-only
mapache mount /mnt/backup -r <URL> --create-mountpoint
mapache mount /mnt/backup -r <URL> --cache-size-mib 128

Mounts the entire repository as a FUSE filesystem. All snapshots are listed at the root; each snapshot directory contains the backed-up files.

Mount a Bundle

mapache mount bundle.mapache /mnt/bundle
mapache mount bundle.mapache /mnt/bundle -b    # Force bundle mode

If the path has a .mapache extension, mapache auto-detects it as a bundle.

Mount Options

Press Ctrl+C to unmount.

16. Snapshot Lifecycle

amend — Modify a Snapshot

# Modify tags
mapache amend --tags "updated,important" -r <URL>

# Modify description
mapache amend --description "Backup before upgrade" -r <URL>

# Clear metadata
mapache amend --clear-tags -r <URL>
mapache amend --clear-description -r <URL>

# Amend specific snapshots (one or more IDs)
mapache amend <SNAPSHOT_ID> -r <URL>
mapache amend <SNAPSHOT_ID_1> <SNAPSHOT_ID_2> -r <URL>

# Add exclusions post-hoc (rebuilds tree)
mapache amend --exclude "*.log" -r <URL>

# Apply to all snapshots
mapache amend --all --tags "archived" -r <URL>

# Keep the old snapshot file
mapache amend --keep-old -r <URL>

When exclusions are added, mapache rebuilds the snapshot tree (creating new tree blobs) without re-chunking file contents. The old snapshot file is deleted unless --keep-old is set.

recall — Recover Dropped Snapshots

mapache recall SNAPSHOT_ID -r <URL>

Restores a snapshot that was previously dropped (marked with .dropped extension) by forget. The snapshot is reactivated and will appear in log listings again.

17. Terminal User Interface

mapache tui -r <URL>

Launches an interactive terminal user interface with screens for:

• Dashboard — overview with version, snapshot count, latest snapshot and session lock status (shared, exclusive, released or off). Lock status comes from the in-memory session handle, without additional repository queries.
• File explorer — browse snapshot contents with inline detail panel
• Snapshot detail — inspect individual snapshots
• Snapshot creation wizard
• Restore wizard
• Forget/retention management
• Diff screen — navigable tree of changes between snapshots (</> to browse adjacent pairs, / to filter, u to toggle unchanged files)
• Find screen — real-time glob search across all snapshots with progress, results table, and direct navigation to explorer/restore
• Stats screen (6) — repository statistics; asks on entry whether to collect quick or full stats (f afterwards adds the pack footers, --full, without re-reading other data) and r to refresh
• Clean screen (7) — run garbage collection with a tolerance percentage, an optional --no-repack/--dry-run mode, a live progress view and a summary of what was removed
• Verify screen (8) — run repository verification with optional --read-packs (and parallel worker count), percentage sampling, ECC --repair, and --fail-early, showing live per-phase progress and a pass/fail report with the integrity counters

Press ? (or F1) anywhere for a list of key bindings, Ctrl+C to quit, and use the mouse wheel to scroll.

Quitting during snapshot, restore, clean or verify requests cancellation and waits for the operation to finish before releasing the session lock. Clean requires exclusive access while executing GC; it refuses to run if another repository lock is active, then restores the previous session mode afterwards. Stats refresh reloads the index; leaving, refreshing or changing stats mode cancels and waits for the previous collection.

Configuration forms group options into Basic and Advanced sections. Tab/Shift+Tab or the arrow keys move between fields, skipping section headers; the form scrolls to keep the focused field visible and displays its help below. Numeric fields accept typed digits as well as left/right adjustments.

Snapshot paths, restore path selections, tags, host filters and include/exclude lists appear as chips. Enter opens an empty entry editor; type a value and press Enter again to add it. Backspace in an empty editor removes the last chip, and Esc discards an unfinished entry. Each entry is a single value, so commas inside paths and glob patterns are preserved. Restore preloads paths selected in the file explorer; removing all path chips restores the entire snapshot. Non-empty Include patterns override the path selection. The restore target remains a single destination directory.

Submitting invalid options focuses the offending field and shows the error (for example, Paths required). Esc leaves an unchanged form immediately; for changed values, it displays a warning and requires a second Esc to discard them. Any other key dismisses that warning. While editing, Esc first closes the editor; text-field edits are retained, while unfinished numeric/chip edits are discarded.

In retention, confirmed discard restores the last applied rule and Force values; applying a valid form establishes the new saved state.

Snapshot configuration also exposes Exclude file, No scan, One filesystem, Fail on skipped, Skip unchanged, Parent and With atime. Parent accepts a snapshot ID or latest; it cannot be combined with No parent. Dry run simulates the snapshot with an isolated index and a write-discarding backend. Progress is marked [DRY RUN], and the summary does not report a saved snapshot.

Restore supports Include file and Exclude file (one pattern per line), Quit on error and Sparse. File patterns are combined with the form’s include/exclude lists using the same preparation as the CLI. Delete removes target items absent from the snapshot after restoring and requires an extra confirmation, except during a dry run. The target root stays protected unless No preserve root is explicitly enabled; this option requires Delete.

Forget retention’s Keep min keeps at least the given number of matching snapshots after applying its rules. Force permanently deletes selected snapshot metadata rather than staging it for removal; these snapshots cannot be recalled. Keep-tags, host and tag filters remain available as chips.

Requires a supported terminal.

18. Shell Completions

mapache completion --shell bash --path /etc/bash_completion.d/
mapache completion --shell zsh --path /usr/share/zsh/site-functions/
mapache completion --shell fish --path ~/.config/fish/completions/
mapache completion --shell powershell --path ~/Documents/PowerShell/

Generates shell completion scripts for the given shell. Supported shells include: bash, zsh, fish, powershell, elvish, and nushell.

19. Environment Variables

If MAPACHE_REPOSITORY is set, the -r flag can be omitted for all commands.

If MAPACHE_USERNAME and MAPACHE_PASSWORD are set, interactive authentication prompts are skipped.

20. Global Options Reference

These options are available on most commands (exceptions: bundle, cache, completion do not use repository connection).

Compression Levels

With none, data and tree blobs are stored uncompressed, which saves CPU and avoids wasting space on content that does not compress. Each blob stores a single compressed/uncompressed marker, so restoring, verifying, or migrating the repository requires no extra configuration. Metadata (footers, index, snapshots) is always compressed regardless of this setting. none is also available in the TOML config as compression = "none".

Note: none requires repository format v2. The v1 format always stores blobs zstd-compressed and has no per-blob compression marker, so using --compression none (or compression = "none") against a v1 repository fails — migrate it to v2 first (mapache migrate -r <URL>).

Bandwidth Limit Format

1000        -> 1000 bytes/s
1K / 1KB   -> 1000 bytes/s  (or 1024 for K)
1M / 1MB   -> 1,000,000 bytes/s  (or 1,048,576 for M)
1G / 1GB   -> 1,000,000,000 bytes/s  (or 1,073,741,824 for G)
1.5M       -> 1,572,864 bytes/s
10MiB/s    -> 10,485,760 bytes/s

Duration Format

30s     -> 30 seconds
5m      -> 5 minutes
5m30s   -> 5 minutes 30 seconds
1h      -> 1 hour
1d      -> 1 day
2w      -> 2 weeks
3m      -> 3 months (30-day months)
4y      -> 4 years (365-day years)

21. Full Command Reference

mapache init

Initialize a new backup repository.

mapache init -r <URL>
  --format <1|2>         Repository format version (default: 2)
  --ecc <0-100>          Enable ECC with given overhead percentage (0 disables)
  --calibrate-kdf        Benchmark Argon2id parameters for this hardware

mapache snapshot

Create a new backup snapshot.

mapache snapshot [PATHS...] -r <URL>
  --as-root               Use a single directory as the snapshot root
  --exclude <PATH,...>    Paths to exclude (repeatable, or comma-separated)
  --exclude-file <PATH>   File with exclude paths (one per line)
  --tags <TAGS>           Comma-separated tags
  --description <TEXT>    Snapshot description
  --no-parent             Force complete analysis (no parent diff)
  --no-scan               Skip the initial file system scan
  --one-file-system       Do not descend into a different filesystem
  --fail-on-skipped       Fail if any source item is skipped
  --skip-if-unchanged     Skip saving if no changes detected
  --parent <ID|latest>    Use specific parent snapshot
  --readers <N>           Parallel file readers (default: 4)
  --packers <N>           Writer threads (default: 4)
  --dry-run               Simulate without making changes
  --stdin                 Read data from stdin as file /stdin
  --with-atime            Store file access times
  --pre-hook <CMD>        Shell command to run before snapshot (overrides config)
  --post-hook <CMD>       Shell command to run after snapshot (overrides config)

mapache restore

Restore a snapshot to a target path.

mapache restore [SNAPSHOT] --target <PATH> -r <URL>
  --target <PATH>         Restore destination (required)
  --include <PATH,...>    Paths to include (repeatable, or comma-separated)
  --include-file <PATH>   File with include paths (one per line)
  --exclude <PATH,...>    Paths to exclude (repeatable, or comma-separated)
  --exclude-file <PATH>   File with exclude paths (one per line)
  --strip-prefix          Strip longest common prefix from restored paths
  --strategy <MODE>       Conflict resolution: fail|skip|overwrite|newer
  --delete                Delete files in target not in snapshot
  --no-preserve-root      Allow --delete at root level
  --quit-on-error         Exit immediately on restore error
  --sparse                Create sparse files
  --verify                Verify content by hashing each blob (incremental restore)
  --batch-size <N>        Max files per batch (limits memory; default: no limit)
  --pre-hook <CMD>        Shell command to run before restore (overrides config)
  --post-hook <CMD>       Shell command to run after restore (overrides config)
  --dry-run               Simulate restoration

mapache forget

Remove snapshots and apply retention policies.

mapache forget [SNAPSHOT_IDS...] -r <URL>
  --force                 Immediately delete (don't just mark)
  --tags <TAGS>           Filter by tags
  --host <HOST>           Filter by host (repeatable)
  --keep-last <N|all>     Keep last N snapshots (or all)
  --keep-within <DUR>     Keep snapshots within duration
  --keep-hourly <N|all>   Keep N hourly snapshots
  --keep-daily <N|all>    Keep N daily snapshots
  --keep-weekly <N|all>   Keep N weekly snapshots
  --keep-monthly <N|all>  Keep N monthly snapshots
  --keep-yearly <N|all>   Keep N yearly snapshots
  --keep-tags <TAGS>      Always keep snapshots with these tags
  --keep-min <N|all>      Minimum snapshots to keep
  --dry-run               Show what would be removed
  --clean                 Run garbage collector after
  -t, --tolerance <%>     GC garbage tolerance (0–100)
  --pre-hook <CMD>        Shell command to run before forget (overrides config)
  --post-hook <CMD>       Shell command to run after forget (overrides config)

mapache log

Show snapshots in the repository.

mapache log [SNAPSHOT_ID] -r <URL>
  --dropped               Show only dropped snapshots
  --all                   Show active and dropped snapshots
  -c, --compact           Compact table listing
  --tags <TAGS>           Filter by tags

mapache ls

List nodes/files in a snapshot.

mapache ls [SNAPSHOT] -r <URL>
  --path <PATH>           Path within snapshot
  -l, --long              Long listing format
  -H, --human-readable    Human-readable sizes
  -R, --recursive         Recursive listing

mapache find

Find files and directories in the repository.

mapache find <TARGET> -r <URL>
  --snapshot <ID|latest>  Search in specific snapshot (all if omitted)
  --json                  Output results in JSON format

Exits with a non-zero code on error.

mapache diff

Show differences between snapshots.

mapache diff <SOURCE_ID> <TARGET_ID> -r <URL>

mapache dump

Print the contents of a file from a snapshot to stdout.

mapache dump [SNAPSHOT] -r <URL>
  --path <PATH>           Path to the file inside the snapshot (required)

mapache stats

Show repository statistics.

mapache stats -r <URL>
  --full                  Parse pack footers for physical stats (expensive)

mapache clean

Run garbage collection.

mapache clean -r <URL>
  -t, --tolerance <%>     Garbage tolerance (0–100, default: 0)
  --no-repack             Skip repacking
  --dry-run               Simulate without making changes
  --pre-hook <CMD>        Shell command to run before clean (overrides config)
  --post-hook <CMD>       Shell command to run after clean (overrides config)

mapache verify

Verify repository integrity.

mapache verify -r <URL>
  --read-packs            Read, decrypt, hash all data (slow)
  --repair                Repair corrupt packs via ECC sidecars
  -p, --parallel <N>      Parallel pack reading (default: 4)
  --with-cache            Use local cache (disabled by default)
  --fail-early            Stop on first error
  --sample <%>            Verify random sample of packs (e.g., 10.5%)
  --dump-pack-blobs <FILE> Dump all blob descriptors to a plain-text file
  --pre-hook <CMD>        Shell command to run before verify (overrides config)
  --post-hook <CMD>       Shell command to run after verify (overrides config)

mapache key

Manage repository key files.

mapache key list -r <URL>
mapache key add -r <URL> [--path <PATH>] [--calibrate-kdf]
mapache key delete <KEY_ID> -r <URL> [--yes]
mapache key change-password -r <URL> [--calibrate-kdf]
mapache key export <KEY_ID> -r <URL> [-o <PATH>]

When --calibrate-kdf is used, mapache benchmarks the current hardware and selects Argon2id memory/iterations/parallelism tuned to a 500 ms target. Memory is auto-detected (10% of RAM, clamped to 32–64 MiB). Use this when creating a key on a powerful server that will later be used for restores.

mapache ecc

Manage Reed-Solomon ECC sidecars for the repository.

mapache ecc enable <PERCENT> -r <URL>      # Enable ECC and generate sidecars
mapache ecc disable -r <URL>               # Disable ECC and remove sidecars
mapache ecc set-percent <PERCENT> -r <URL> # Change overhead and regenerate
mapache ecc regenerate -r <URL>            # Regenerate all sidecars

ecc requires repository format v2. Use mapache migrate to upgrade v1 repositories before enabling ECC.

mapache bundle

Create, extract, import, export, or mount .mapache bundle files.

mapache bundle -b <INPUT...> -o <OUTPUT.mapache> [OPTIONS]
mapache bundle -x <INPUT.mapache> [-o <DIR>] [OPTIONS]
mapache bundle --export-snapshot <SNAPSHOT> -o <OUTPUT.mapache> -r <URL>
mapache bundle -i <INPUT.mapache...> -r <URL>
mapache bundle -m <INPUT.mapache> <MOUNTPOINT> [OPTIONS]

  -a, --bundle            Create mode
  -x, --extract           Extract mode
  -i, --import            Import mode: one or more bundle files (requires -r)
  --export-snapshot <SNAPSHOT>  Export mode: export snapshot (latest, prefix, or ID; requires -r)
  -m, --mount             Mount mode (FUSE, Unix only)
  -o, --output <PATH>     Bundle file (-a, --export-snapshot) or destination (-x)
  -e, --exclude <GLOB>    Exclude globs (bundle mode)
  --as-root               Use a single directory as the bundle root (bundle mode)
  --readers <N>           Parallel readers (default: 4)
  --calibrate-kdf         Benchmark Argon2id parameters for this hardware (create mode)
  -c, --create            Create mountpoint (mount mode)
  --allow-other           Allow other users (mount mode)
  --metadata-only         Don't load file contents (mount mode)
  --cache-size-mib <MIB>  Data cache size (mount mode, default: 256)

mapache mount

Mount repository or bundle as FUSE filesystem (Unix only).

mapache mount <MOUNTPOINT> -r <URL>
  -b, --bundle            Force bundle mount
  --allow-other           Allow other users
  -c, --create-mountpoint Create mountpoint if missing
  --metadata-only         Don't load file contents
  --cache-size-mib <MIB>  Data cache size (default: 64)

Exits with a non-zero code on error.

mapache cat

Print repository objects.

mapache cat manifest -r <URL>
mapache cat <type>:<ID> -r <URL>
  Types: manifest, pack, blob, tree, index, key, snapshot, lock

mapache amend

Modify an existing snapshot.

mapache amend [SNAPSHOT...] -r <URL>
  -a, --all               Apply to all snapshots
  --keep-old              Keep the old snapshot file
  --tags <TAGS>           Replace tags
  --clear-tags            Remove all tags
  --description <TEXT>    Replace description
  --clear-description     Remove description
  --exclude <PATH,...>    Exclude paths (rebuilds tree)
  --exclude-file <PATH>   File with exclude paths

mapache recall

Recover a dropped (forgotten) snapshot.

mapache recall <SNAPSHOT_ID> -r <URL>

mapache rebuild-index

Rebuild the index by scanning all packs.

mapache rebuild-index -r <URL>
  --dry-run               Simulate without making changes

mapache migrate

Migrate repository format from v1 to v2.

mapache migrate -r <URL>
  --dry-run               Simulate without making changes

mapache repack

Re-encode all reachable blobs with current settings.

mapache repack -r <URL>

mapache rechunk

Recalculate chunk boundaries for every snapshot file and write new blobs using the current settings.

mapache rechunk -r <URL>

mapache copy

Transfer snapshots between repositories.

mapache copy --all --from <URL> -r <URL>
  --from <URL>                Source repository (required)
  --from-ssh-privatekey <PATH>  SSH key for source repository
  --from-ssh-known-hosts <PATH>  SSH known hosts file for source
  --all                       Copy all snapshots (cannot be combined with the filters below)
  --snapshot <PREFIX>         Copy only snapshots with this ID prefix (repeatable)
  --host <HOST>               Copy only snapshots from this host (repeatable)
  --tags <TAGS>               Copy only snapshots with this tag (repeatable)
  --dry-run                   Show what would be copied without transferring

mapache sync

Synchronize a repository to another backend.

mapache sync --target <URL> -r <URL>
  --delete                Delete files in destination not in source
  --dst-ssh-privatekey <PATH>  SSH key for destination
  --dst-ssh-known-hosts <PATH>  SSH known hosts file for destination
  --dry-run               Simulate without making changes

mapache unlock

Remove stale locks.

mapache unlock -r <URL>
  -f, --force             Remove all locks (including non-expired)

mapache cache

Manage local cache directories.

mapache cache
  --delete <PREFIX>...    Delete cache folders by ID prefix (repeatable)
  --clear                 Delete all cache folders

mapache completion

Generate shell completion scripts.

mapache completion --shell <SHELL> --path <DIR>
  --shell <SHELL>         Shell type (bash, zsh, fish, powershell, elvish, nushell)
  --path <DIR>            Output directory

mapache config

Manage repository configuration.

mapache config template [--output <PATH>]
  --output <PATH>         Write template to file instead of stdout

mapache tui

Launch terminal user interface.

mapache tui -r <URL>

Design (Repository Format v2)

Note: Mapache uses self-contained, independent format specifications. This document describes one such format from first principles; no other format version is referenced here.

Introduction

The mapache repository

Mapache implements a repository as its central location for data storage. The mapache repository is a collection of content-addressable files and binary objects distributed in a centralized layout in the repository directory. Content-addressable means that, with rare exceptions, all files and objects are indexed not by a name or an arbitrary identifier, but the hash of its binary contents. When mapache writes a snapshot file to the repository, it will first calculate the hash of its content and that hash will be the name of the file.

This way of indexing data offers two interesting properties, provided that a strong enough hash is used to avoid collisions:

1. Two identical objects will have the same ID, therefore if an object O with content C and hash H exists, all binary streams with hash H are guaranteed to have content C. This allows us to skip writing a second copy of the identical data and grants us the certainty that content C exists already in the repository.

2. If the contents of an object changed due to error or external manipulation, its hash would not correspond to its contents. If the hash were also manipulated to match the content, this new ID would no longer represent the original data. This allows us to detect errors or external manipulation of the data.

In the context of mapache, an object is a stream of binary data represented by its hash, or ID, regardless of how it is stored. An object can be stored as a single file, with its filename being the ID of the object, or bundled with other objects in a file with any known arbitrary format. In any case, the filename will identify its contents as a whole by their hash.

The mapache repository is a tree directory that contains files with different purposes:

repo
├── index
│   └── 771f7c9d05241c55a87e004b148310953c1565ef10f55daf02a68ba0ea4e0166
├── keys
│   └── cea3212843e488b168f6bb21b757d6cb4882a559e10ec4b33f305e982c4c3c8e
├── locks
├── manifest
├── objects
│   ├── 00
│   ├── 01
│   ├── ..
│   ├── ff
│   │   └── ff890e7aeb656e7f57004e05009fa66576448a150c6471d38e11e4e6d0f3227b
└── snapshots
    └── 1d0e210a91f77744f87420387e52a108b87fd900a1d8462f3b9def4b99a47b9e
    └── 84afca27f7c134131cf8d33a2ae70114889c3c31dc281b7fffad998e7ddb569a.dropped

Storage

Mapache is a de-duplicating, incremental backup program. It implements a content-defined chunking algorithm to split files into small chunks. Every chunk is identified by the hash of its raw data, or ID. This algorithm is able to detect content boundaries, so when you modify the file, the algorithm can detect the new data added as a series of new chunks. All these file chunks, or data blobs are stored in the objects directory, which works as a blob library. Since we can identify every blob by its ID, a single blob is stored only once. If two files share a blob, this blob is saved only once in the repository. When a file is saved along multiple snapshots, mapache will chunk the file and only store the new blobs that do not exist already in that blob library.

A snapshot is akin to a still photograph of the file system at a point of time. The content of files will be deduplicated and encoded into blobs in the object storage. However, if we only stored data blobs, we would not be able to reconstruct the original file system tree, the metadata, permissions, file times and attributes. Mapache stores the tree metadata as blobs in the object storage. File nodes contain the list of data blobs that constitute its contents, symlinks contain the target path, directory nodes have a reference to its own subtree, etc. These metadata blobs are also deduplicated so that if two trees are exactly identical, there only exists one copy in the object storage. This deduplication reduces the amount of data necessary to store the snapshot metadata, since most of the time a snapshot only introduces a few modified or new trees. The snapshot file will reference the root tree, and contain metadata like a timestamp, host, tags, description, etc.

Blobs in the object storage are not stored as individual files. We could do that, but it would be highly inefficient. A generic repository could potentially contain millions of blobs, exhausting the available file system inodes and requiring too many I/O operations to store all the blobs. Instead, blobs are packed in pack files. A pack file contains multiple blobs and some metadata describing them. Pack files can be configured to have any size up to 4 GiB. In order to find blobs efficiently within the pack files, the mapache repository maintains an index. This index, composed of one or more index files, describes the packs and their contents and allows mapache to find a blob by mapping the blob ID to the pack ID, its offset, length and type.

The index is the central piece of how mapache works. If a blob ID is not referenced by the index, the blob does not exist for mapache and will be subject to elimination by the garbage collector, even if there is a pack file that contains it. This allows mapache to append new data atomically. If a backup is interrupted, all blobs and packs not referenced by a persisted index will be left dangling, as if they were never added. Restarting the backup will not resume the process from where it was interrupted, but all indexed blobs will not be rewritten. To avoid losing all progress, mapache may periodically persist all finalized indices to file.

Chunking

Mapache uses a custom implementation of FastCDC. FastCDC is a Content-Defined Chunking algorithm proposal that offers a speedup of 3x - 12x with respect to Rabin-based CDC with comparable deduplication ratio.

Compression and encryption

A good chunker can do wonders to reduce the amount of data needed to store all the files of a snapshot, but we can go one step further. Mapache uses zstd to compress almost all the data stored in the repository, including data and tree blobs. The compression level can be configured, affecting the compression ratio and the total time needed to take a snapshot.

Compression is a per-repository policy, not a per-blob algorithm choice: the whole repository uses a single algorithm (currently zstd). If additional algorithms are added in the future, the algorithm will be declared in the manifest. Individual blobs are only marked as compressed or uncompressed (see the pack footer format).

Compression can be disabled with the --compression none flag. In that mode, data and tree blobs are stored without compression (still encrypted), which is useful for content that is already compressed (video, photos, archives). Metadata (footers, index, snapshots) is always compressed with the default level, regardless of this flag. none requires the per-blob compression marker in the pack footer.

Security is a non-negotiable aspect of mapache by design. Everything except for a handful of bytes is encrypted. Encryption cannot be disabled or opted-out. Mapache implements AES-GCM-SIV. This is a modern cipher which implements encryption and authentication of data with a key. AES-GCM-SIV not only encrypts the data, it adds an authentication layer that allows mapache to detect when an object has been altered or manipulated.

Each encrypted blob has its AES-GCM-SIV nonce appended at the end ([ciphertext | tag | nonce]) instead of the start.

Nonces are derived deterministically from the plaintext content: the nonce is the first 12 bytes of the BLAKE3 hash of the plaintext — the same hash that determines the blob’s content ID. Identical plaintexts therefore always produce identical ciphertext, which is safe for AES-GCM-SIV: a nonce can only repeat together with identical plaintext, so there is no collision risk even at scale. This also makes encryption reproducible: the same input byte-for-byte always yields the same encoded blob.

Master key and key files

All the encrypted content in the mapache repository is encrypted with a unique 256-bit master key. This key is randomly generated when the repository is created. Users do not access the repository with the master key directly. Instead, they are asked for a username and a password. The password is used to derive a 256-bit personal key using Argon2. The personal key is solely used to encrypt personal copies of the master key in a KeyFile. Every user has a KeyFile that they use to open the repository. These KeyFiles can be stored in the keys directory or provided externally.

Mapache does not implement master key rotation, and it is not planned. Every KeyFile holds a personal copy of the master key, so any user can obtain it; a rotation would have to be performed by someone holding that key, and a malicious user could hijack the repository in the process. To change the master key or the set of users, create a new repository and copy the snapshots over.

Hashing

All content IDs (hashes) used by mapache to identify objects in the repository are generated with a high-performance cryptographic hashing algorithm. The current implementation uses BLAKE3, which produces 256-bit hashes.

Threat model

Mapache assumes that the host system is secure. However, the repository itself may reside on an untrusted medium where an attacker has the ability to read, modify, or delete the repository files.

Security Guarantees

Mapache provides strong guarantees for the security of the data, but it cannot prevent file system operations by an external attacker:

• Confidentiality: An attacker should not be able to decipher the stored data.
• Integrity (Verifiability): Mapache can detect if the data has been modified or corrupted.

Protection Mechanisms

Mapache uses a dual-layer approach to ensure data integrity and confidentiality:

• Authenticated Encryption with Associated Data (AEAD): All data is protected using an AEAD cipher (AES-GCM-SIV). This means the information is both encrypted (ensuring confidentiality) and authenticated. The authentication layer allows Mapache to detect if a file has been altered or manipulated externally.
• Content-Addressability: Objects are identified by the cryptographic hash (ID) of their content. This provides a secondary and independent verification mechanism to ensure the content has not been corrupted.

Why both layers are needed

The AEAD authenticates the ciphertext with the master key. Because every user of the repository holds the master key (via their KeyFile), the AEAD can only guarantee that the ciphertext has not been modified or corrupted since it was written — it cannot attest that the decrypted plaintext is the content its ID claims. A user of the repository, or an attacker who has compromised the master key, can encrypt any content and produce a valid, authenticated ciphertext under any ID, and the AEAD will accept it.

Content-addressability is the layer that closes this gap. The ID (the BLAKE3 hash of the plaintext) is fixed independently in the pack footer and the index, and mapache re-hashes every blob immediately after decryption on the read path by default. A blob whose decrypted content does not match its ID — whether by accidental corruption, a wrong pack or offset, or deliberate substitution by a key-holding party — fails the hash check. This verification is mandatory on every read; it is not deferred to an optional verify workflow.

Attacker Limitations

Due to the mandatory encryption and authentication:

• An attacker who reads the data will only obtain unintelligible, encrypted information.
• An attacker who modifies or renames files will cause the corresponding integrity check (both AEAD and hash verification) to fail, rendering the data unusable by Mapache.

Therefore, the most effective action an attacker on the untrusted medium can take is the destruction (deletion) of the repository files. Mapache is unable to prevent files in the repository from being modified, renamed, or deleted by an external entity with access to the storage medium.

Repository format

Manifest

The manifest file describes the repository: its ID, creation time, and optional ECC parameters. The ID is a 256-bit identifier generated randomly at the time of creation.

The manifest file is compressed and encrypted with the master key.

The manifest is serialized as JSON:

{
  "version": 2,
  "id": "b7468f6331331302b06c63b98a14e50107f9cc26683afd064c0af84eec53b3e7",
  "created_time": "2025-11-20T13:06:52.266751300+01:00",
  "hash_algorithm": "blake3-256",
  "compression_algorithm": "zstd",
  "encryption_algorithm": "aes-256-gcm-siv",
  "ecc": {
    "data_shards": 100,
    "parity_shards": 2
  }
}

The algorithm fields are immutable repository-wide selections. Readers reject values they do not support instead of guessing from the repository version. Repositories created before these fields existed use the v2 defaults shown above.

The ecc field is optional. When present, it indicates that ECC sidecars are enabled for pack, index, and snapshot files in this repository. data_shards (K) and parity_shards (P) are the Reed-Solomon parameters stored for forward compatibility: if the formula that maps user configuration to K/P changes in the future, old repositories still decode correctly using the stored values.

JSON is used instead of a binary format to allow extending the manifest with new fields without breaking format compatibility.

Snapshots

The snapshot file contains metadata about a snapshot: timestamp, user name, host name, paths, root tree, summary, etc. The file name (ID) is the hash of the encoded content.

{
  "timestamp": "2025-11-20T19:00:00.375479500+01:00",
  "tree": "b266241241162f84e91165d96d03de0719b385c5b2918cec02b5324869e6145c",
  "root": "I:\\mapache",
  "paths": [
    "I:\\mapache\\core"
  ],
  "hostname": "cocoon",
  "username": "mapache",
  "version": "0.7.0",
  "tags": ["daily", "critical"],
  "description": "Full backup of project directory",
  "summary": {
    "processed_items_count": 94,
    "processed_bytes": 744155,
    "raw_bytes": 744155,
    "encoded_bytes": 197650,
    "meta_raw_bytes": 42694,
    "meta_encoded_bytes": 18639,
    "total_raw_bytes": 786849,
    "total_encoded_bytes": 216289,
    "data_blobs": 42,
    "meta_blobs": 12,
    "ecc_bytes": 10240,
    "new_files": 79,
    "deleted_files": 0,
    "changed_files": 0,
    "new_dirs": 15,
    "deleted_dirs": 0,
    "changed_dirs": 0,
    "unchanged_files": 0,
    "unchanged_dirs": 0,
    "amends": null
  }
}

All fields except timestamp, tree, root, and summary are optional and omitted from the JSON when not set. The parent field (not shown) references the previous snapshot ID when this snapshot was created with --parent. The amends field references a snapshot ID when this snapshot was created with --amend.

Packs and blobs

Mapache stores backup data in the objects directory. A fanout of 1 byte is used to reduce the number of files per directory: 256 directories with names 00 to ff contain the pack files. A packfile with ID dd2856e... will be stored in objects/dd/dd2856e9f7ac9b495e7ebce1370f68a2e60824cee9546b817f7cbbb4d571bd4b.

All blobs are stored in pack files, which group multiple blobs together to optimize storage and I/O operations. The pack file name (ID) is the hash of its content. Since a pack file contains already pre-encoded blobs and an encoded footer, no further encoding is applied.

The layout of a pack file is the following.

Encoded pack 1 Encoded pack 2 … Encoded pack N PACK FOOTER

The blobs are encoded and appended to the pack. Their ID is the hash of their raw content. The pack footer describes the blobs contained in the pack footer. This footer includes a 4-byte field which is the length of the total footer. The pack footer is encoded, except for the length field.

Each encrypted blob has its AES-GCM-SIV nonce appended at the end ([ct | tag | nonce]). This eliminates an extra allocation and memory copy during encryption — the ciphertext is encrypted in-place and the tag and nonce are appended to the same buffer. This is transparent to the pack format — the footer stores encoded length and raw length as before.

Trees

Mapache stores file contents (data blobs) as well as tree metadata (tree blobs). Tree and data blobs are stored in separate packs. The data blobs contain chunks of the file contents. The tree blob contain metadata about the file system tree nodes. Data and tree blobs are stored in pack files in the same way. Each tree is stored as a single blob. A tree node can be a file, directory, symlink, etc. Directories have a tree field which points to its subtree. File nodes have a blobs field which contains a list of its data blobs in order. Symlinks have a field symlink_info which contains metadata about the target path and node type of the target.

Tree format

Tree blobs use JSON. JSON is chosen over a binary format for trees because it is forward-compatible: serde ignores unknown fields, so new metadata attributes can be added to tree nodes without breaking existing parsers. Index and packs use binary formats where compactness is critical.

Node example (directory):

{
  "name": "tests",
  "type": "directory",
  "metadata": {
    "size": 0,
    "created_time": {
      "secs_since_epoch": 1763560510,
      "nanos_since_epoch": 935069400
    },
    "modified_time": {
      "secs_since_epoch": 1763560510,
      "nanos_since_epoch": 937075900
    }
  },
  "tree": "093ca4db9e88163d1e9e815e01d9b3a6e7e7980d21a674e433eb75fdb2f133cd"
}

Full tree example (directory with nodes):

{
  "nodes": [
    {
      "name": "integration_tests",
      "type": "directory",
      "metadata": {
        "size": 4096,
        "created_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 935069400
        },
        "modified_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        }
      },
      "tree": "9a426139fa0315872bb3da4ee6ed6f0955f7425846efd0c22c3d040d7f9dc9d7"
    },
    {
      "name": "tests.rs",
      "type": "file",
      "metadata": {
        "size": 834,
        "created_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        },
        "modified_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        }
      },
      "blobs": [
        "6772c0355a4f84a095343951e371677fc88ba92820bc4e12ab78c9de61ae3ea2"
      ]
    }
  ]
}

Pack footer format

The pack footer begins with an 8-byte header (MPFT, version 2, and zero reserved flags), followed by a variable-length list of metadata blob entries, each 41 bytes long, followed by a fixed-size trailer. Each entry contains an ID (32 bytes), a blob type byte (u8), and both the encoded length and raw length (u32) of the associated data blob. The trailer is a single u32 field which stores the total length of the entire pack footer, allowing a parser to efficiently skip directly to the file’s data section. All data except the footer length field is zstd-compressed then encrypted.

┌─────────┬────────┬────────┬─────┬────────┬─────────────────────┐
│ MPFT v2 │ Blob 1 │ Blob 2 │ ... │ Blob N │ Footer length (u32) │
└─────────┴────────┴────────┴─────┴────────┴─────────────────────┘

┌──────────────────────────────────────────────────────────────────────────┐
│                    Pack footer blob entry (41 bytes)                     │
├──────────────────────────┬────────┬────────┬─────────────────────────────┤
│ Field                    │ Size   │ Offset │ Notes                       │
├──────────────────────────┼────────┼────────┼─────────────────────────────┤
│ ID (256-bit raw hash)    │ 32     │ 0      │ Hash of the raw blob data   │
├──────────────────────────┼────────┼────────┼─────────────────────────────┤
│ Type + compression (u8)  │ 1      │ 32     │ Low 7 bits: blob type;      │
│                          │        │        │ high bit: 0 = uncompressed, │
│                          │        │        │ 1 = zstd-compressed         │
├──────────────────────────┼────────┼────────┼─────────────────────────────┤
│ Encoded length (u32)     │ 4      │ 33     │ Length of the encoded blob  │
├──────────────────────────┼────────┼────────┼─────────────────────────────┤
│ Raw length (u32)         │ 4      │ 37     │ Length before encoding      │
└──────────────────────────┴────────┴────────┴─────────────────────────────┘
  ^ (41 bytes)

The compression marker is a single bit packed into the high bit of the blob type byte. Because the compression algorithm is repo-wide (declared in the manifest), a per-blob boolean is sufficient. The low 7 bits hold the blob type:

When the high bit is set (0x80 | type), the blob’s payload is zstd-compressed and the encoded length is the compressed size; when clear, the payload is stored as-is (encoded length == raw length).

Zero blobs (0x03) carry no payload in the pack: their encoded length is 0 (the same zero region is deduplicated across snapshots) and their raw length is the number of zero bytes the blob represents. A restore reconstructs exactly that many zero bytes for the position the file node assigns to the blob, ordered among the sibling blobs by the tree. No data is ever stored for zero blobs.

The descriptor list is padded to a multiple of 64 entries (FOOTER_BLOB_MULTIPLE) before serialization. Missing entries are filled with Padding descriptors (type 0x02) whose other fields are random noise. As a result, every pack footer is a whole number of 64-entry blocks. Readers must skip Padding descriptors; they reference no data.

Index

The index files, located in the repo/index directory, are the repository’s central lookup mechanism and source of truth for all stored data.

These files contain metadata about the repository’s pack files. Crucially, they act as a map, linking the ID (cryptographic hash) of every content-addressable tree and data blob to its specific physical location, including the Pack ID, offset, and length within that pack.

The index determines what data exists in the repository; any pack or blob data not referenced by an index is considered non-existent and subject to garbage collection, which is key to mapache’s atomic append mechanism. The filename of the index file itself is the hash of its encoded content.

Index files use a compact binary format. This reduces index size and improves parsing speed.

Index binary format

Top-level:

Pack encoding (repeated per pack):

Blob encoding (45 bytes each, repeated per blob):

The binary format reduces an index entry to 45 bytes (vs ~150+ bytes for JSON), and a full index file to roughly 45 bytes per blob plus overhead.

Error correction (ECC)

Mapache supports optional Reed-Solomon erasure coding to protect pack files against bit-rot and silent data corruption. ECC is configured per-repository via the --ecc <PERCENT> flag during init (0–100% overhead, 0 disables).

The ecc crate is a pure Reed-Solomon implementation with no knowledge of overhead percentages or repository concepts. It exposes a K/P-based API: ecc_encode(data, k, p) and ecc_decode(data, ecc_payload). Mapache decides the K and P values (currently K=100, P=overhead) and stores them in the manifest. This separation allows changing the K/P strategy (presets, adaptive overhead, etc.) without modifying the ecc crate.

Sidecar layout

ECC sidecars store CRC32 checksums and parity shards — the original pack data is never duplicated on disk. The sidecar file is compressed, encrypted, and stored alongside the pack with the same ID and a .ecc extension:

objects/
├── 00
│   └── 00ab12...cd pack file
│   └── 00ab12...cd.ecc  ← ECC sidecar (CRC32s + parity)
├── 01
│   └── 01cd34...ef pack file
└── ff
    └── ff5678...ab pack file

MECP wire format

Sidecars use the MECP (Mapache ECC Parity) format:

Header (54 bytes):
┌──────────┬─────────┬──────────┬───────────┬───────────────┬──────────────┬────────────────┐
│ MAGIC    │ VERSION │ reserved │ K (u16)   │ original_len  │ stripe_count │
│ b"MECP"  │ u8      │ 0        │ P (u16)   │ u64 LE        │ u32 LE       │ BLAKE3 hash     │
└──────────┴─────────┴──────────┴───────────┴───────────────┴──────────────┴────────────────┘
  4 bytes    1 byte    1 byte     2+2 bytes    8 bytes         4 bytes       32 bytes

Per stripe:
┌──────────────────┬──────────────────┬─────────────────────┐
│ CRC32 data shards│ CRC32 parity     │ raw parity shards   │
│ (K × 4 bytes)    │ (P × 4 bytes)    │ (P × SHARD_SIZE)    │
└──────────────────┴──────────────────┴─────────────────────┘

Each stripe stores a CRC32 checksum (IEEE 802.3) for every data and parity shard. This enables per-shard integrity verification: during decode, each shard is independently checked against its CRC32. Corrupt shards are marked as erasures and reconstructed by Reed-Solomon. This converts error correction (floor(P/2) limit) into erasure recovery (up to P shards per stripe).

CRC32 uses a 256-entry lookup table generated at compile time via const fn (zero runtime overhead, no external dependencies).

K and P are stored in the header for forward compatibility: if the formula that maps overhead to K/P changes in the future, old sidecars still decode correctly using the stored values.

The header also stores the BLAKE3-256 hash of the exact encoded file bytes protected by the sidecar. Repair refuses to use a sidecar whose binding does not match the file being repaired.

Each shard is SHARD_SIZE = 4096 bytes (matching the OS page size and mapache block size). The last shard in each stripe may be zero-padded.

Striped encoding

For large packs, data is split into stripes of K data shards each (K=100 by default). Each stripe is independently encoded, so the maximum memory per stripe is ~400 KB (100 shards × 4096 bytes). This allows ECC processing of multi-gigabyte packs without loading the entire file into memory.

Data (1 GB pack, 5% overhead):
┌─────────────────────────┬─────────────────────────┬─────┐
│ stripe 0 (100 shards)   │ stripe 1 (100 shards)   │ ... │
│ → 5 parity shards       │ → 5 parity shards       │     │
└─────────────────────────┴─────────────────────────┴─────┘

The last stripe may have fewer data bytes but uses the same K and P, so the ReedSolomon codec is built once and reused across all stripes.

Repair

During verify --repair, mapache reads the pack file and its .ecc sidecar. The sidecar is decoded (decompressed + decrypted) before use. For each stripe, the CRC32 of every data shard (read from the pack) and every parity shard (read from the sidecar) is verified against the stored checksums. Corrupt shards are marked as erasures. If erasures ≤ P, Reed-Solomon reconstructs the missing shards from the remaining intact ones. If erasures > P, the stripe cannot be repaired and an error is returned.

The hash (BLAKE3) is always computed on the original pack content without ECC. The sidecar is transparent to the content-addressable storage layer. Like all repository objects, the sidecar is compressed and encrypted on disk.

Garbage collection

When GC deletes a pack file, it also deletes the corresponding .ecc sidecar. Orphaned .ecc files (where the base pack no longer exists) are detected and cleaned up automatically.

Locks

Lock files are the locking and synchronization mechanism at the repository level. When mapache opens a repository, it may need to acquire an exclusive or non-exclusive lock. Non-exclusive locks allow concurrent access with other non-exclusive locks. They are used for constructive operations which do not affect other users. Exclusive locks are used to get single, exclusive access to the repository for destructive operations like forget or clean.

A lock is considered expired if its timestamp is older than a predetermined threshold. Expired locks are not considered when trying to acquire a new lock and get deleted by the garbage collection mechanism. During normal operation, locks get periodically refreshed. If the process that holds a lock terminates abruptly, it might not release its lock. The expiration time serves as a way to remove those locks which did not get deleted.

The lock name is a unique 256-bit ID generated randomly at the time of creation and has nothing to do with its content. A lock that is refreshed will conserve its ID, even though the timestamp will change.

{
  "id": "3468acbec193d65560c8ff16fea30e2c566d48ddf1234aefa9b1bac09c1cf71f",
  "timestamp": "2025-11-20T00:00:00.390485800+01:00",
  "exclusive": false,
  "hostname": "cocoon",
  "username": "mapache",
  "pid": 4216,
  "context": ["mapache", "snapshot", "/"],
  "creation_time": "2025-11-20T00:00:00.390485800+01:00"
}

Lock files may also contain context (a list of command-line arguments identifying the holding process) and creation_time (when the lock was first acquired). Both fields are optional: files written by older versions may omit them.

Keys

KeyFiles serve as keys to the repository. They contain metadata about the user, creation time, parameters for the password derivation function and the encrypted master key, using the password and a salt. KeyFiles can be stored in the keys directory or outside of the repository. KeyFiles are compressed but not encrypted. The KeyFile name (ID) is the hash of the encoded (compressed) content.

{
  "created": "2025-11-20T19:00:00.663953700+01:00",
  "username": "mapache",
  "kdf": {
    "algorithm": "argon2id",
    "m": 19456,
    "t": 2,
    "p": 1
  },
  "salt": "VRKrdGuf5T8iW3b5VYgJ+bMTMrYKRphbxkq2qW6C7VQ=",
  "encrypted_key": "bkwxbjuLbV094UiRFoO5t2YlV3ZhsCQdn3z2A+YyPSNewGFUo+AwL+L7Jae406yW4OHcHzFwN3wv7Vlj"
}

The kdf field is a tagged enum (KdfConfig) that stores both the algorithm name and its parameters. The algorithm discriminator allows extending to additional KDFs in the future without changing the on-disk format. Currently the only supported value is "argon2id".

Note: The top-level m, t, p fields are deprecated and will be removed in a future version. New keyfiles always write the kdf object. Old keyfiles without kdf are read by constructing the kdf object from the flat fields.

Local cache

Mapache maintains a local cache to speedup some commands and minimize the number of remote I/O operations. Certain metadata like tree blobs, snapshots and indices are cached locally to avoid having to download them from a remote backend. The cache is a mirror copy of the mapache repository layour, but it only contains those files that are eligible for caching. These cache directories are stored in the usual cache directory of the OS.

Mapache provides a cache command to list and manage cache folders, including a cleanup option.

Bundle files

A bundle file (.mapache) is a self-contained, encrypted archive that stores the same blob types used inside a repository: tree blobs, metadata blobs, and data blobs. The bundle uses its own independent encryption key derived from the bundle password (not the repository key).

Bundle layout

┌──────────────────────────────────┐
│ BundleHeader (58 bytes, plain)   │
├──────────────────────────────────┤
│ Encrypted blob data sections...  │
├──────────────────────────────────┤
│ Encrypted ECC section (optional) │
├──────────────────────────────────┤
│ Encrypted BundleIndex            │
├──────────────────────────────────┤
│ Encrypted Manifest               │
├──────────────────────────────────┤
│ Encrypted BundleTrailer          │
├──────────────────────────────────┤
│ Trailer size (u32, LE, plain)    │
└──────────────────────────────────┘

The header and trailer size are stored unencrypted to allow efficient random access. All other sections are encrypted with AES-256-GCM-SIV using a key derived from the bundle password via Argon2id.

Blob data sections are compressed with zstd and then encrypted by default. Compression can be disabled with --compression none, in which case blobs are only encrypted (still protected by AES-256-GCM-SIV). The per-blob compression marker in the index tracks whether each blob is compressed, so bundles with mixed compression can be read correctly.

ECC in bundles

When --ecc <PERCENT> is passed during bundle creation, the data section is protected by inline Reed-Solomon erasure coding. The ECC section is placed between the blob data and the index:

• Data section is split into K=100 data shards of 4096 bytes each.
• P=percent parity shards are computed per stripe using Reed-Solomon.
• The ECC payload (header + CRC32s + parity shards) is encrypted with AES-256-GCM-SIV before being written to the bundle.
• The BundleTrailer gains two extra fields: ecc_offset (u64) and ecc_len (u32), increasing the trailer from 68 to 80 bytes.

During extraction, if a blob fails to decrypt (corruption), the reader automatically reads the full data section + ECC section, verifies each shard via CRC32, and runs Reed-Solomon decode to repair corrupt shards. The repaired data is then re-read and decoded successfully.

Bundle index

The bundle index maps blob IDs to their physical location within the bundle file. Each index entry is 49 bytes:

┌──────────────────────────────────┬───────┬──────────────────┬────────┬────────────┐
│ Blob ID (32 bytes)               │ Type  │ Offset           │ Length │ Raw Length │
│                                  │+Comp  │ (u64, LE)        │(u32,LE)│ (u32, LE)  │
│                                  │ (u8)  │                  │        │            │
└──────────────────────────────────┴───────┴──────────────────┴────────┴────────────┘
   32 bytes                          1        8 bytes          4 bytes    4 bytes

The type byte uses the same encoding as pack footers: the low 7 bits hold the blob type (0x00=Data, 0x01=Tree, 0x02=Padding, 0x03=Zero), and the high bit indicates whether the blob is zstd-compressed (1) or uncompressed (0).

Export and import

The --export-snapshot command walks the snapshot tree, loads each referenced blob from the repository and writes it into the bundle preserving the original blob IDs. When the same bundle is later imported into a repository (or a different one), each blob is checked against the destination’s master index. Blobs that already exist are skipped; only new blobs are written and packed. The imported snapshot tree is then persisted as a new snapshot.

This design means:

1. Cross-repo deduplication — importing a bundle into a repository that already contains the same data stores no additional blobs. This is useful for merging backups from different machines or migrating between repositories.
2. No format differences — bundles and repositories store identical blob types. Export does not re-encode or re-encrypt data; it simply relocates the raw blobs into a single portable file.

Design (Repository Format v1)

Note: This document describes repository format v1, which is deprecated and will be unsupported in a future release. To migrate a v1 repository, run mapache migrate --repo <path>.

Introduction

The mapache repository

Mapache implements a repository as its central location for data storage. The mapache repository is a collection of content-addressable files and binary objects distributed in a centralized layout in the repository directory. Content-addressable means that, with rare exceptions, all files and objects are indexed not by a name or an arbitrary identifier, but the hash of its binary contents. When mapache writes a snapshot file to the repository, it will first calculate the hash of its content and that hash will be the name of the file.

This way of indexing data offers two interesting properties, provided that a strong enough hash is used to avoid collisions:

1. Two identical objects will have the same ID, therefore if an object O with content C and hash H exists, all binary streams with hash H are guaranteed to have content C. This allows us to skip writing a second copy of the identical data and grants us the certainty that content C exists already in the repository.

2. If the contents of an object changed due to error or external manipulation, its hash would not correspond to its contents. If the hash were also manipulated to match the content, this new ID would no longer represent the original data. This allows us to detect errors or external manipulation of the data.

In the context of mapache, an object is a stream of binary data represented by its hash, or ID, regardless of how it is stored. An object can be stored as a single file, with its filename being the ID of the object, or bundled with other objects in a file with any known arbitrary format. In any case, the filename will identify its contents as a whole by their hash.

The mapache repository is a tree directory that contains files with different purposes:

repo
├── index
│   └── 771f7c9d05241c55a87e004b148310953c1565ef10f55daf02a68ba0ea4e0166
├── keys
│   └── cea3212843e488b168f6bb21b757d6cb4882a559e10ec4b33f305e982c4c3c8e
├── locks
├── manifest
├── objects
│   ├── 00
│   ├── 01
│   ├── ..
│   ├── ff
│   │   └── ff890e7aeb656e7f57004e05009fa66576448a150c6471d38e11e4e6d0f3227b
└── snapshots
    └── 1d0e210a91f77744f87420387e52a108b87fd900a1d8462f3b9def4b99a47b9e
    └── 84afca27f7c134131cf8d33a2ae70114889c3c31dc281b7fffad998e7ddb569a.dropped

Storage

Mapache is a de-duplicating, incremental backup program. It implements a content-defined chunking algorithm to split files into small chunks. Every chunk is identified by the hash of its raw data, or ID. This algorithm is able to detect content boundaries, so when you modify the file, the algorithm can detect the new data added as a series of new chunks. All these file chunks, or data blobs are stored in the objects directory, which works as a blob library. Since we can identify every blob by its ID, a single blob is stored only once. If two files share a blob, this blob is saved only once in the repository. When a file is saved along multiple snapshots, mapache will chunk the file and only store the new blobs that do not exist already in that blob library.

A snapshot is akin to a still photograph of the file system at a point of time. The content of files will be deduplicated and encoded into blobs in the object storage. However, if we only stored data blobs, we would not be able to reconstruct the original file system tree, the metadata, permissions, file times and attributes. Mapache stores the tree metadata as blobs in the object storage. File nodes contain the list of data blobs that constitute its contents, symlinks contain the target path, directory nodes have a reference to its own subtree, etc. These metadata blobs are also deduplicated so that if two trees are exactly identical, there only exists one copy in the object storage. This deduplication reduces the amount of data necessary to store the snapshot metadata, since most of the time a snapshot only introduces a few modified or new trees. The snapshot file will reference the root tree, and contain metadata like a timestamp, host, tags, description, etc.

Blobs in the object storage are not stored as individual files. We could do that, but it would be highly inefficient. A generic repository could potentially contain millions of blobs, exhausting the available file system inodes and requiring too many I/O operations to store all the blobs. Instead, blobs are packed in pack files. A pack file contains multiple blobs and some metadata describing them. Pack files can be configured to have any size up to 4 GiB. In order to find blobs efficiently within the pack files, the mapache repository maintains an index. This index, composed of one or more index files, describes the packs and their contents and allows mapache to find a blob by mapping the blob ID to the pack ID, its offset, length and type.

The index is the central piece of how mapache works. If a blob ID is not referenced by the index, the blob does not exist for mapache and will be subject to elimination by the garbage collector, even if there is a pack file that contains it. This allows mapache to append new data atomically. If a backup is interrupted, all blobs and packs not referenced by a persisted index will be left dangling, as if they were never added. Restarting the backup will not resume the process from where it was interrupted, but all indexed blobs will not be rewritten. To avoid losing all progress, mapache may periodically persist all finalized indices to file.

Chunking

Mapache uses a custom implementation of FastCDC. FastCDC is a Content-Defined Chunking algorithm proposal that offers a speedup of 3x - 12x with respect to Rabin-based CDC with comparable deduplication ratio.

Compression and encryption

A good chunker can do wonders to reduce the amount of data needed to store all the files of a snapshot, but we can go one step further. Mapache uses zstd to compress almost all the data stored in the repository, including data and tree blobs. The compression level can be configured, affecting the compression ratio and the total time needed to take a snapshot.

Security is a non-negotiable aspect of mapache by design. Everything except for a handful of bytes is encrypted. Encryption cannot be disabled or opted-out. Mapache implements AES-GCM-SIV. This is a modern cipher which implements encryption and authentication of data with a key. AES-GCM-SIV not only encrypts the data, it adds an authentication layer that allows mapache to detect when an object has been altered or manipulated.

Master key and key files

All the encrypted content in the mapache repository is encrypted with a unique 256-bit master key. This key is randomly generated when the repository is created. Users do not access the repository with the master key directly. Instead, they are asked for a username and a password. The password is used to derive a 256-bit personal key using Argon2. The personal key is solely used to encrypt personal copies of the master key in a KeyFile. Every user has a KeyFile that they use to open the repository. These KeyFiles can be stored in the keys directory or provided externally.

Mapache does not implement master key rotation at the moment, but it may do so in the future.

Hashing

All content IDs (hashes) used by mapache to identify objects in the repository are generated with a high-performance cryptographic hashing algorithm. The current implementation uses BLAKE3, which produces 256-bit hashes.

Threat model

Mapache assumes that the host system is secure. However, the repository itself may reside on an untrusted medium where an attacker has the ability to read, modify, or delete the repository files.

Security Guarantees

Mapache provides strong guarantees for the security of the data, but it cannot prevent file system operations by an external attacker:

• Confidentiality: An attacker should not be able to decipher the stored data.
• Integrity (Verifiability): Mapache can detect if the data has been modified or corrupted.

Protection Mechanisms

Mapache uses a dual-layer approach to ensure data integrity and confidentiality:

• Authenticated Encryption with Associated Data (AEAD): All data is protected using an AEAD cipher (AES-GCM-SIV). This means the information is both encrypted (ensuring confidentiality) and authenticated. The authentication layer allows Mapache to detect if a file has been altered or manipulated externally.
• Content-Addressability: Objects are identified by the cryptographic hash (ID) of their content. This provides a secondary and independent verification mechanism to ensure the content has not been corrupted.

Attacker Limitations

Due to the mandatory encryption and authentication:

• An attacker who reads the data will only obtain unintelligible, encrypted information.
• An attacker who modifies or renames files will cause the corresponding integrity check (both AEAD and hash verification) to fail, rendering the data unusable by Mapache.

Therefore, the most effective action an attacker on the untrusted medium can take is the destruction (deletion) of the repository files. Mapache is unable to prevent files in the repository from being modified, renamed, or deleted by an external entity with access to the storage medium.

Repository format

The mapache repository stores metadata in JSON format.

Manifest

The manifest file describes the repository: its ID, creation time and version number. The ID is a 256-bit identifier generated randomly at the time of creation.

The manifest file is compressed and encrypted with the master key.

{
  "version": 1,
  "id": "b7468f6331331302b06c63b98a14e50107f9cc26683afd064c0af84eec53b3e7",
  "created_time": "2025-11-20T13:06:52.266751300+01:00"
}

Snapshots

The snapshot file contains metadata about a snapshot: timestamp, user name, host name, paths, root tree, summary, etc. The file name (ID) is the hash of the encoded content.

{
  "timestamp": "2025-11-20T19:00:00.375479500+01:00",
  "tree": "b266241241162f84e91165d96d03de0719b385c5b2918cec02b5324869e6145c",
  "root": "I:\\mapache",
  "paths": [
    "I:\\mapache\\core"
  ],
  "hostname": "cocoon",
  "username": "mapache",
  "summary": {
    "processed_items_count": 94,
    "processed_bytes": 744155,
    "raw_bytes": 744155,
    "encoded_bytes": 197650,
    "meta_raw_bytes": 42694,
    "meta_encoded_bytes": 18639,
    "total_raw_bytes": 786849,
    "total_encoded_bytes": 216289,
    "new_files": 79,
    "deleted_files": 0,
    "changed_files": 0,
    "new_dirs": 15,
    "deleted_dirs": 0,
    "changed_dirs": 0,
    "unchanged_files": 0,
    "unchanged_dirs": 0
  }
}

Packs and blobs

Mapache stores backup data in the objects directory. A fanout of 1 byte is used to reduce the number of files per directory: 256 directories with names 00 to ff contain the pack files. A packfile with ID dd2856e... will be stored in objects/dd/dd2856e9f7ac9b495e7ebce1370f68a2e60824cee9546b817f7cbbb4d571bd4b.

All blobs are stored in pack files, which group multiple blobs together to optimize storage and I/O operations. The pack file name (ID) is the hash of its content. Since a pack file contains already pre-encoded blobs and an encoded footer, no further encoding is applied.

The layout of a pack file is the following.

Encoded pack 1 Encoded pack 2 … Encoded pack N PACK FOOTER

The blobs are encoded and appended to the pack. Their ID is the hash of their raw content. The pack footer describes the blobs contained in the pack footer. This footer includes a 4-byte field which is the length of the total footer. The pack footer is encoded, except for the length field.

Trees

Mapache stores file contents (data blobs) as well as tree metadata (tree blobs). Tree and data blobs are stored in separate packs. The data blobs contain chunks of the file contents. The tree blob contain metadata about the file system tree nodes in JSON format. Data and tree blobs are stored in pack files in the same way. Each tree is stored as a single blob. A tree node can be a file, directory, symlink, etc. Directories have a tree field which points to its subtree. File nodes have a blobs field which contains a list of its data blobs in order. Symlinks have a field symlink_info which contains metadata about the target path and node type of the target.

{
  "name": "tests",
  "type": "directory",
  "metadata": {
    "size": 0,
    "created_time": {
      "secs_since_epoch": 1763560510,
      "nanos_since_epoch": 935069400
    },
    "modified_time": {
      "secs_since_epoch": 1763560510,
      "nanos_since_epoch": 937075900
    }
  },
  "tree": "093ca4db9e88163d1e9e815e01d9b3a6e7e7980d21a674e433eb75fdb2f133cd"
}

{
  "nodes": [
    {
      "name": "integration_tests",
      "type": "directory",
      "metadata": {
        "size": 4096,
        "created_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 935069400
        },
        "modified_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        }
      },
      "tree": "9a426139fa0315872bb3da4ee6ed6f0955f7425846efd0c22c3d040d7f9dc9d7"
    },
    {
      "name": "tests.rs",
      "type": "file",
      "metadata": {
        "size": 834,
        "created_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        },
        "modified_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        }
      },
      "blobs": [
        "6772c0355a4f84a095343951e371677fc88ba92820bc4e12ab78c9de61ae3ea2"
      ]
    }
  ]
}

Pack footer format

The pack footer consists of a variable-length list of metadata blob entries, each 41 bytes long, followed by a fixed-size trailer. Each entry contains an ID (32 bytes), the blob type (u8), and both the encoded length and raw length (u32) of the associated data blob. The trailer is a single u32 field which stores the total length of the entire pack footer, allowing a parser to efficiently skip directly to the file’s data section. All data except the footer length field is encrypted.

┌────────┬────────┬─────┬────────┬─────────────────────┐
│ Blob 1 │ Blob 2 │ ... │ Blob N │ Footer length (u32) │
└────────┴────────┴─────┴────────┴─────────────────────┘

┌──────────────────────────────────────────────────────────────────────────┐
│                    Pack footer blob entry (41 bytes)                     │
├──────────────────────────────────────────────────────────────────────────┤
│ Field                    │ Size   │ Offset │ Notes                       │
├──────────────────────────┼────────┼────────┼─────────────────────────────┤
│ ID (256-bit raw hash)    │ 32     │ 0      │ Hash of the raw blob data   │
├──────────────────────────┼────────┼────────┼─────────────────────────────┤
│ Blob type (u8)           │ 1      │ 32     │                             │
├──────────────────────────┼────────┼────────┼─────────────────────────────┤
│ Encoded length (u32)     │ 4      │ 33     │ Length of the encoded blob  │
├──────────────────────────┼────────┼────────┼─────────────────────────────┤
│ Raw length (u32)         │ 4      │ 37     │ Length before encoding      │
└──────────────────────────┴────────┴────────┴─────────────────────────────┘
  ^ (41 bytes)

Index

The index files, located in the repo/index directory, are the repository’s central lookup mechanism and source of truth for all stored data.

These files contain metadata about the repository’s pack files. Crucially, they act as a map, linking the ID (cryptographic hash) of every content-addressable tree and data blob to its specific physical location, including the Pack ID, offset, and length within that pack.

The index determines what data exists in the repository; any pack or blob data not referenced by an index is considered non-existent and subject to garbage collection, which is key to mapache’s atomic append mechanism. The filename of the index file itself is the hash of its encoded content.

{
  "packs": [
    {
      "id": "c2334c5e29563850cb254b63fba125bb1b60e860b1886fae7bd4f5e5a637f4c0",
      "blobs": [
        {
          "id": "22bb6896db82b2fcc2d8057d6e16298e8318a04d1e0dd5b6c84fadc46a234f4c",
          "type": "Data",
          "offset": 25763,
          "length": 3265,
          "raw_length": 12108
        },
        {
          "id": "99ce77389066f2982e8350eefdbf21ccef026d94fc4827dbc24cd52d77e25e7f",
          "type": "Data",
          "offset": 73520,
          "length": 2462,
          "raw_length": 7515
        }
      ]
    },
    {
      "id": "9a2ad285d1dd3d9fd7ca7b8fc8bcbcbd62ef9871bbf1f74757ad392a58e4d71e",
      "blobs": [
        {
          "id": "ffcdcfc7ac3832168c952f4e396588b7cb523e009c0abf776c46806468146985",
          "type": "Tree",
          "offset": 3188,
          "length": 897,
          "raw_length": 3514
        },
        {
          "id": "23f87643e00c9f7c498c057a58ed2f38af22402eaab9e7b58a8bfa505b11756b",
          "type": "Tree",
          "offset": 5206,
          "length": 863,
          "raw_length": 3475
        },
      ]
    }
  ]
}

Locks

Lock files are the locking and synchronization mechanism at the repository level. When mapache opens a repository, it may need to acquire an exclusive or non-exclusive lock. Non-exclusive locks allow concurrent access with other non-exclusive locks. They are used for constructive operations which do not affect other users. Exclusive locks are used to get single, exclusive access to the repository for destructive operations like forget or clean.

A lock is considered expired if its timestamp is older than a predetermined threshold. Expired locks are not considered when trying to acquire a new lock and get deleted by the garbage collection mechanism. During normal operation, locks get periodically refreshed. If the process that holds a lock terminates abruptly, it might not release its lock. The expiration time serves as a way to remove those locks which did not get deleted.

The lock name is a unique 256-bit ID generated randomly at the time of creation and has nothing to do with its content. A lock that is refreshed will conserve its ID, even though the timestamp will change.

{
  "id": "3468acbec193d65560c8ff16fea30e2c566d48ddf1234aefa9b1bac09c1cf71f",
  "timestamp": "2025-11-20T00:00:00.390485800+01:00",
  "exclusive": false,
  "hostname": "cocoon",
  "username": "mapache",
  "pid": 4216
}

Keys

KeyFiles serve as keys to the repository. They contain metadata about the user, creation time, parameters for the password derivation function and the encrypted master key, using the password and a salt. KeyFiles can be stored in the keys directory or outside of the repository. KeyFiles are compressed but not encrypted. The KeyFile name (ID) is the hash of the encoded (compressed) content.

{
  "created": "2025-11-20T19:00:00.663953700+01:00",
  "username": "mapache",
  "m": 19456,
  "t": 2,
  "p": 1,
  "salt": "VRKrdGuf5T8iW3b5VYgJ+bMTMrYKRphbxkq2qW6C7VQ=",
  "encrypted_key": "bkwxbjuLbV094UiRFoO5t2YlV3ZhsCQdn3z2A+YyPSNewGFUo+AwL+L7Jae406yW4OHcHzFwN3wv7Vlj"
}

Local cache

Mapache maintains a local cache to speedup some commands and minimize the number of remote I/O operations. Certain metadata like tree blobs, snapshots and indices are cached locally to avoid having to download them from a remote backend. The cache is a mirror copy of the mapache repository layour, but it only contains those files that are eligible for caching. These cache directories are stored in the usual cache directory of the OS.

Mapache provides a cache command to list and manage cache folders, including a cleanup option.

Bundle files

A bundle file (.mapache) is a self-contained, encrypted archive that stores the same blob types used inside a repository: tree blobs, metadata blobs, and data blobs. The bundle uses its own independent encryption key derived from the bundle password (not the repository key).

Export and import

The --export-snapshot command walks the snapshot tree, loads each referenced blob from the repository and writes it into the bundle preserving the original blob IDs. When the same bundle is later imported into a repository (or a different one), each blob is checked against the destination’s master index. Blobs that already exist are skipped; only new blobs are written and packed. The imported snapshot tree is then persisted as a new snapshot.

This design means:

1. Cross-repo deduplication — importing a bundle into a repository that already contains the same data stores no additional blobs. This is useful for merging backups from different machines or migrating between repositories.
2. No format differences — bundles and repositories store identical blob types. Export does not re-encode or re-encrypt data; it simply relocates the raw blobs into a single portable file.