mapache

Fast, encrypted, deduplicating backup tool. Written in Rust.

Install

curl -fsSL https://github.com/jlantxa/mapache/raw/main/tools/install.sh | sh

Or grab a binary from the releases page.

Usage

# Initialize a repository
mapache init -r /path/to/repo

# Create a snapshot
mapache snapshot ~/Documents -r /path/to/repo

# Launch the TUI
mapache tui -r /path/to/repo

# List snapshots
mapache log -c -r /path/to/repo

# Restore
mapache restore --target /tmp/restore -r /path/to/repo

Features

• Parallel backup and restore — Multi-threaded pipeline with configurable concurrency for readers and packers.
• Encryption — AES-256-GCM-SIV with Argon2id key derivation. Data is never stored or transmitted in the clear.
• Deduplication — Content-defined chunking (FastCDC) across all snapshots and machines. Only new data is stored.
• Backends — Local filesystem, SFTP, and S3-compatible object storage.
• Terminal UI — Interactive TUI with dashboard, snapshot and restore wizards, file explorer, and diff viewer.
• Bundles — Self-contained encrypted .mapache files for transfer or cold storage.
• FUSE mount — Browse snapshot contents as a mounted filesystem (Unix).
• Retention policies — Hourly, daily, weekly, monthly and yearly rules. Filter by host and tag.
• Multi-client — Non-exclusive locking allows multiple hosts to back up to the same repository.
• Portable — Linux, macOS, Windows and Android. Single statically-linked binary for each platform.

Mapache User Manual

mapache — a fast, secure, deduplicating, incremental backup tool.

Table of Contents

1. Introduction
2. Installation
3. Quick Start
4. Concepts
5. Configuration
6. Hooks
7. Repository Management
8. Taking Snapshots
9. Restoring Data
10. Exploring Snapshots
11. Retention Policies
12. Maintenance
13. Security & Key Management
14. Bundle Files
15. FUSE Mount
16. Snapshot Lifecycle
17. Terminal User Interface
18. Shell Completions
19. Environment Variables
20. Global Options Reference
21. Full Command Reference
22. Troubleshooting & FAQ

1. Introduction

Mapache (Spanish for “raccoon”) is a high-performance backup tool written in Rust. It uses a content-addressable repository model: every file, directory, and piece of metadata is decomposed into binary objects identified by their cryptographic hash (BLAKE3). This enables global deduplication — if multiple files or snapshots share the same content, even across different machines, the data is stored only once.

Key properties:

• Deduplication — FastCDC (Content-Defined Chunking) splits files into variable-size chunks; only new chunks are stored.
• Encryption — Mandatory AES-GCM-SIV authenticated encryption. Data is never stored or transmitted in cleartext.
• Compression — Zstd compression with adjustable levels.
• Snapshots — Every backup is a point-in-time snapshot. Snapshots are logically independent (no “full vs. incremental” chain) but share underlying data blobs.
• Portable — Single statically-linked binary with zero runtime dependencies.
• Backends — Local filesystem, SFTP, and S3-compatible object storage.
• Verifiable — Built-in integrity verification at every level.

Project Status

Mapache is feature-complete and production-oriented, but relatively new. Users should validate their own backup/restore workflows before relying on it for mission-critical data.

2. Installation

Quick Install (Linux, macOS, Windows)

curl -fsSL https://github.com/jLantxa/mapache/raw/main/tools/install.sh | sh

Downloads the latest release binary for your platform and installs it to /usr/local/bin/mapache. Set VERSION or INSTALL_DIR to customize.

Pre-built Binaries

Download binaries from the Releases page. All binaries are statically linked and require no dependencies.

Build from Source

Requires the Rust toolchain.

# Build release binary
cargo build --release

# Install to cargo bin path
cargo install --path core

For a statically-linked build:

make release-static

Feature Flags

• fuse (default on Linux) — FUSE mount support. Requires libfuse headers at build time.

To build without FUSE:

cargo build --release --no-default-features

To build with all features:

cargo build --release --all-features

Docker

A multi-stage Dockerfile is included in the repository root.

3. Quick Start

Initialize a Repository

# Local directory
mapache init -r /backup/myrepo

# SFTP server
mapache init -r sftp://user@host/backup

# S3 bucket
mapache init -r s3://my-bucket/backups

You will be prompted for a username and password. Do not lose this password — it is the only way to access your data.

Create a Snapshot

mapache snapshot ~/Documents -r /backup/myrepo

# Or pipe data directly via --stdin
tar czf - ~/Documents | mapache snapshot --stdin -r /backup/myrepo

List Snapshots

mapache log -c -r /backup/myrepo

Restore a Snapshot

# Restore the latest snapshot
mapache restore --target /tmp/restore -r /backup/myrepo

Apply Retention and Clean Up

# Keep last 7 daily snapshots, then garbage-collect
mapache forget --keep-daily 7 --clean -r /backup/myrepo

4. Concepts

Repository

A repository is a directory (local or remote) that stores all backup data. It contains objects, pack files, indices, keys, snapshots, and locks.

repo/
├── index/          # Pack index files (blob → pack mapping)
├── keys/           # User key files
├── locks/          # Runtime lock files
├── manifest        # Repository metadata (ID, version, creation time)
├── objects/        # Pack files (00/..ff/ fanout)
└── snapshots/      # Snapshot metadata files

Content-Addressable Storage

Every blob (chunk of data or tree metadata) is identified by the BLAKE3 hash of its raw content. This means:

• Identical content produces the same hash → deduplication.
• Any change in content produces a different hash → integrity verification.

Pack Files

Blobs are not stored individually. They are grouped into pack files (default 16 MiB each) for I/O efficiency. A pack file contains multiple encoded (compressed + encrypted) blobs and a footer that describes their offsets and sizes.

Index

The index maps blob IDs to their physical location (pack ID + offset + length). It is the source of truth — any blob not in the index is considered non-existent and subject to garbage collection.

Snapshots

A snapshot is a point-in-time record of your files. It contains:

• The root tree ID (linking to all file/directory metadata)
• Timestamp, hostname, username
• Paths, tags, description
• Summary statistics (processed bytes, dedup counts, diff counts)

Snapshots are logically independent. Deleting an old snapshot does not affect newer ones (shared blobs are reference-counted by the index).

Chunking

Mapache uses FastCDC (Content-Defined Chunking) with:

• Minimum chunk: 512 KiB
• Average chunk: 1 MiB
• Maximum chunk: 8 MiB
• Normalization: L2

Chunker parameters are stable across versions to ensure consistent chunk boundaries.

Encryption

All data (packs, indices, snapshots, manifest) is encrypted with AES-GCM-SIV using a randomly-generated 256-bit master key. User passwords are derived via Argon2id to produce wrapping keys that encrypt the master key.

5. Configuration

Environment Variables

TOML Config File

Mapache can read settings from a TOML configuration file specified via --with-config <PATH>. The file supports [global], [snapshot], [restore], [forget], [runtime], and [hooks] sections.

Generate the canonical template with:

mapache config template                             # Print to stdout
mapache config template --output ./mapache.toml     # Write to file

All values are commented out; uncomment and adjust as needed.

Example configuration:

[global]
repo = "s3://my-bucket/backups"
no-cache = false
ssh-privatekey = "~/.ssh/id_ed25519"
ssh-known-hosts = "~/.ssh/known_hosts"
auth-file = "~/.mapache-auth"
pack-size = 16
key-file = "~/.mapache-key"
quiet = false
verbosity = 1
compression = "balanced"
retry-lock = "5m"
limit-upload = "10MB/s"
limit-download = "50MB/s"

[snapshot]
exclude = [".git/", "node_modules/", "target/"]
readers = 4
packers = 4
skip-if-unchanged = true

[restore]
strategy = "newer"
sparse = true
batch-size = 100000        # Limit memory by restricting files per batch (optional)

[forget]
keep-daily = 7
keep-weekly = 4
keep-monthly = 6
keep-yearly = 2
keep-min = 3

[runtime]
restore-pack-prefetch = 4
restore-blob-concurrency = 8
restore-max-open-files = 128
blobs-per-index-file = 65535
index-flush-timeout-secs = 600
s3-multipart-threshold = 134217728
s3-multipart-part-size = 134217728

[hooks.snapshot.pre]
command = "echo 'Starting snapshot'"

[hooks.snapshot.post]
command = "echo 'Snapshot finished: $MAPACHE_RESULT'"

CLI flags override config file values. Config file values supplement CLI lists (e.g., exclude paths from both sources are combined).

Authentication

Authentication can be provided via:

1. Interactive prompt (default) — prompts for username and password
2. --auth-file <PATH> — file with username on the first line and password on the second line
3. MAPACHE_USERNAME / MAPACHE_PASSWORD environment variables
4. --key-file <PATH> — path to an external key file (still requires username and password for authentication)

When authenticating interactively, mapache allows 3 password attempts before failing.

6. Hooks

Mapache supports running custom shell scripts before and after certain commands. This is useful for notifications, logging, mounting/unmounting filesystems, or any pre/post processing.

Configuration

Hooks are defined in the TOML config file under [hooks.<command>] sections:

[hooks.snapshot.pre]
command = "echo 'Starting snapshot'"
timeout = 300

[hooks.snapshot.post]
command = "/usr/local/bin/notify.sh"

Each hook can specify:

Pre and Post Hooks

• Pre hooks run before the command starts. If the hook exits with a non-zero status, the command is aborted immediately.
• Post hooks run after the command finishes, regardless of success or failure. Non-zero exit status only logs a warning — the command is not affected.

Supported Commands

Hooks can be configured for: snapshot, restore, forget, clean, and verify.

CLI Override

The --pre-hook and --post-hook flags can be passed on the command line to override the TOML-configured hook for a single invocation. An empty string falls through to the TOML hook. When a CLI override is used, the timeout from the TOML config is not applied.

mapache snapshot /data --pre-hook "echo starting"
mapache snapshot /data --post-hook ""   # falls through to TOML hook

Environment Variables

The following variables are passed to every hook script:

For security, the MAPACHE_USERNAME and MAPACHE_PASSWORD environment variables are stripped from the hook environment and are never accessible to hook scripts.

Timeout

When a timeout is configured and the hook exceeds it, the child process is killed immediately. For pre hooks, this is treated as a failure and the command is aborted. For post hooks, the timeout is only logged as a warning and does not affect the command result.

Shell Execution

Hook commands are executed via sh -c on Unix and cmd /C on Windows.

Dry Run

When a command is run with --dry-run, hooks are not executed.

Example

[hooks.snapshot.pre]
command = "df -h /backup > /tmp/disk-space-before.txt"

[hooks.snapshot.post]
command = """
  if [ "$MAPACHE_RESULT" = "success" ]; then
    curl -s -X POST https://hooks.example.com/backup-ok \
      -d "repo=$MAPACHE_REPOSITORY&cmd=$MAPACHE_COMMAND"
  fi
"""

[hooks.clean.pre]
command = "/usr/local/bin/maintenance-start.sh"
timeout = 60

[hooks.clean.post]
command = "/usr/local/bin/maintenance-end.sh"

7. Repository Management

init — Initialize a Repository

mapache init -r <URL>

Creates a new empty repository at the given location. Supported URL schemes:

• Local: /path/to/repo or local:/path/to/repo
• SFTP: sftp://user@host:port/path
• S3: s3://bucket-name/path

SSH key support: mapache supports Ed25519 and ECDSA keys for SFTP authentication. RSA keys are not supported — use ssh-keygen -t ed25519 to generate a compatible key pair.

You will be prompted for a username and password. A manifest file is created with a unique repository ID and the current mapache version. Do not lose your password — there is no password recovery mechanism.

unlock — Remove Stale Locks

mapache unlock -r <URL>
mapache unlock --force -r <URL>

Locks are created when mapache opens a repository and automatically released on clean shutdown. If a process crashes, stale locks may remain. By default, unlock removes only expired locks. Use --force to remove all locks (including non-expired ones).

cache — Manage Local Cache

mapache cache list
mapache cache --delete <PREFIX> [<PREFIX>...]
mapache cache --clear

Mapache caches metadata (tree blobs, snapshots, indices) locally to speed up subsequent operations on remote backends. The cache command lists cache directories by repository ID, deletes specific caches by prefix, or clears all caches.

copy — Transfer Snapshots Between Repositories

mapache copy --from <URL> -r <URL>
mapache copy --from <URL> --dry-run -r <URL>
mapache copy --from <URL> --snapshot ID --host HOST -r <URL>

Copies snapshots (and their data) from one repository to another. Useful for migration, creating replicas, or consolidating backups.

The copy operation:

1. Connects to both source (--from) and destination (-r) repositories
2. Lists all snapshots in the source, applying any filters (--snapshot, --host, --tags)
3. Skips snapshots already present in the destination (by ID)
4. Walks every referenced tree to discover all required data blobs
5. Filters out blobs already present in the destination (blob-level dedup)
6. Transfers only missing blobs, re-packing them at the destination
7. Writes snapshot metadata to the destination

Supports --json for machine-readable progress events.

sync — Replicate a Repository

mapache sync --target <URL> -r <URL>
mapache sync --target <URL> --delete -r <URL>
mapache sync --target <URL> --dry-run -r <URL>

Synchronizes a repository to a different backend. Useful for creating replicas or migrating between storage providers.

The sync operation:

1. diffs source vs destination
2. deletes obsolete files (destination-only)
3. copies missing or changed files
4. synchronizes the manifest last to ensure destination validity.

rebuild-index — Disaster Recovery

mapache rebuild-index
mapache rebuild-index --dry-run

Rebuilds the master index from scratch by scanning all existing pack files and parsing their footers. Used for disaster recovery when the index is lost or corrupted but all pack files are intact.

The command will:

1. List all pack files in the repository
2. Parse each pack footer to discover blobs
3. Build a new master index
4. Persist the new index
5. Delete old/replaced index files

8. Taking Snapshots

Basic Usage

mapache snapshot PATH [PATH...] -r <URL>

One or more paths can be specified. Mapache backs up each path, deduplicating shared content across all of them.

Common Options

Exclusions

Exclusions support glob patterns (**, *, ?) as well as literal paths. Non-glob paths are normalized to absolute paths; glob patterns are kept as-is for matching. They can be specified inline or in a file:

# Inline exclusions
mapache snapshot ~/project --exclude "~/project/target","~/project/.git"

# Exclusion file (one path per line)
mapache snapshot ~/project --exclude-file ~/.mapache-exclude

Tags

Tags are comma-separated labels used for filtering snapshots in log, forget, and other commands.

mapache snapshot ~/Documents --tags "work,important"

Incremental Backups

By default, mapache uses the latest snapshot as the parent for incremental diffing. Only changed/new files are processed; unchanged files are skipped.

# Explicitly set the parent snapshot
mapache snapshot ~/Documents --parent latest

# Force a full scan (ignore parent)
mapache snapshot ~/Documents --no-parent

# Use a specific snapshot as parent
mapache snapshot ~/Documents --parent abc1234f

Skip If Unchanged

When --skip-if-unchanged is set, mapache compares the new snapshot’s root tree with the parent’s. If they match, no new snapshot file is created.

Parallelism

• --readers: Controls how many files are read and chunked concurrently.
• --packers: Controls how many threads compress, encrypt, and write pack files concurrently.

Higher values improve throughput on fast storage but consume more memory.

Dry Run

Using --dry-run shows what would be backed up without writing any data:

mapache snapshot ~/Documents --dry-run -r /backup/myrepo

Hooks are also skipped during dry runs (see Hooks).

Stdin Backup

Use --stdin to back up data from a pipe or redirected file as a single virtual file at /stdin:

# Backup a database dump
pg_dump mydb | mapache snapshot --stdin -r /backup/myrepo

# Backup a tar archive
tar czf - /important/data | mapache snapshot --stdin -r /backup/myrepo --tags "archive"

# Redirect a file
mapache snapshot --stdin -r /backup/myrepo < /path/to/largefile.bin

The data is chunked, deduplicated, compressed, and encrypted identically to regular files. The --stdin flag is mutually exclusive with paths, excludes, and --parent.

Important: --stdin requires non-interactive authentication (stdin is consumed by backup data, so the password cannot be prompted interactively). Use either --auth-file or MAPACHE_USERNAME/MAPACHE_PASSWORD env vars:

# Option 1: auth file
echo "myusername" > ~/.mapache-auth
echo "mypassword" >> ~/.mapache-auth
mapache snapshot --stdin -r /backup/myrepo --auth-file ~/.mapache-auth

# Option 2: environment variables
MAPACHE_USERNAME=myusername MAPACHE_PASSWORD=mypassword \
  mapache snapshot --stdin -r /backup/myrepo

Snapshot Output

After a successful snapshot, mapache displays:

• Changes since parent (new/changed/deleted/unchanged files and dirs)
• Data and metadata sizes (raw vs compressed)
• The new snapshot ID

9. Restoring Data

Basic Usage

mapache restore [SNAPSHOT] --target <DIR> -r <URL>

Restores a snapshot to the given target directory. The SNAPSHOT argument can be a snapshot ID prefix or latest (default).

Options

Conflict Strategies

Path Filtering

Restore supports both include and exclude filtering:

# Restore only specific paths
mapache restore --target /tmp/restore --include "docs/,src/main.rs"

# Exclude certain paths
mapache restore --target /tmp/restore --exclude "*.log,*.tmp"

# Use filter files
mapache restore --target /tmp/restore --include-file restore-list.txt

Strip Prefix

When restoring a subset of paths, --strip-prefix removes the common ancestor directory:

# Snapshot contains: /home/user/project/src/main.rs
# Without --strip-prefix: /tmp/restore/home/user/project/src/main.rs
# With --strip-prefix:    /tmp/restore/src/main.rs

Delete Mode

--delete removes files in the target directory that do not exist in the snapshot. Use with caution. --no-preserve-root extends deletion to the root level of the restored tree.

Sparse Files

--sparse creates sparse files instead of preallocating space. This is faster on some filesystems but may cause fragmentation.

Verification

By default, restore decides whether to overwrite a file based on the chosen strategy alone. Use --verify to enable content-level verification: the restorer reads each existing local file, hashes every blob, and compares the result against the repository index.

When --verify is active with overwrite or newer, the restorer performs incremental restore: each blob is individually compared against the local file content. Only blobs whose hash differs from the on-disk data are downloaded and written; unchanged blobs are skipped. This significantly speeds up repeated restores of large files where only a small fraction of the content changed.

If verification fails (e.g. unreadable file), the restorer falls back to the strategy’s default behavior.

Dry Run

mapache restore --target /tmp/restore --dry-run

Simulates the full restore process (including conflict detection, path filtering, and deletion) without touching the filesystem. Hooks are also skipped (see Hooks).

Restore Output

Displays progress, a summary of errors and warnings, and total duration.

10. Exploring Snapshots

log — List Snapshots

mapache log -r <URL>
mapache log -c -r <URL>           # Compact listing
mapache log --all -r <URL>        # Show active + dropped
mapache log --dropped -r <URL>    # Show only dropped
mapache log --tags "work" -r <URL> # Filter by tags
mapache log SNAPSHOT_ID -r <URL>  # Show specific snapshot

Full listing shows: ID, date, size, root, hostname, username, tags, paths, and description. Compact mode (-c) shows a summary table.

ls — Browse Snapshot Contents

mapache ls -r <URL>                        # List root of latest snapshot
mapache ls SNAPSHOT_ID -r <URL>            # List root of specific snapshot
mapache ls --path /home/user/docs -r <URL> # List specific path
mapache ls -l -r <URL>                     # Long listing (permissions, size, mtime)
mapache ls -H -r <URL>                     # Human-readable sizes
mapache ls -R -r <URL>                     # Recursive listing

find — Search Snapshots

mapache find PATTERN -r <URL>              # Search in all snapshots
mapache find PATTERN --snapshot latest -r <URL>  # Search in latest only
mapache find PATTERN --snapshot ID -r <URL>      # Search in specific snapshot
mapache find PATTERN --json -r <URL>       # JSON output

Searches file and directory names by pattern across one or all snapshots. When the pattern does not contain a /, the search is recursive and matches in any subdirectory. Use a leading / (e.g. find /file.txt) to restrict the search to the root directory of each snapshot.

Supports --json for machine-readable output. Exits with a non-zero code on error.

diff — Compare Snapshots

mapache diff SOURCE_ID TARGET_ID -r <URL>

Shows the differences between two snapshots. Symbol legend:

Output includes a summary table and a size comparison.

stats — Repository Statistics

mapache stats -r <URL>
mapache stats --full -r <URL>   # Parse pack footers (expensive)

Shows:

• Packs: count, total size, optional footer analysis (blob count, encoded vs raw sizes, dangling blobs)
• Index: count, size, indexed blobs, raw/encoded sizes
• Snapshots: count, total snapshot metadata size
• Snapshot reference summary: referenced blobs (data + tree), raw/encoded sizes, compression ratios, total restorable size
• Keys: count, total size
• Manifest: size
• Total repository size

--full parses every pack footer, giving physical blob statistics and dangling blob detection (expensive on large repositories).

dump — Print File Contents

mapache dump SNAPSHOT_ID --path /path/to/file -r <URL>
mapache dump latest --path /path/to/file -r <URL>

Prints the contents of a single file from a snapshot to stdout. The SNAPSHOT_ID argument accepts ID prefixes or latest.

cat — Print Raw Repository Objects

mapache cat manifest -r <URL>
mapache cat snapshot:ID -r <URL>
mapache cat pack:ID -r <URL>
mapache cat blob:ID -r <URL>
mapache cat tree:ID -r <URL>
mapache cat index:ID -r <URL>
mapache cat key:ID -r <URL>
mapache cat lock:ID -r <URL>

Prints the raw JSON content of repository objects. Useful for debugging and manual inspection.

11. Retention Policies

forget — Remove Snapshots

mapache forget [SNAPSHOT_IDS...] -r <URL>     # Remove specific snapshots
mapache forget --keep-daily 7 -r <URL>         # Apply retention rules
mapache forget --keep-last 5 --clean -r <URL>  # Keep + GC
mapache forget --force SNAPSHOT_ID -r <URL>    # Permanently delete

Retention Rules

By default, forget marks snapshots as dropped (renames to .dropped). Use --force for immediate deletion.

A snapshot is kept if any rule applies. Rules are inclusive, not exclusive.

Filtering

# Only consider snapshots with specific tags
mapache forget --keep-daily 7 --tags "work"

# Only consider snapshots from specific hosts
mapache forget --keep-daily 7 --host my-server

Garbage Collection Integration

# Run GC after forget
mapache forget --keep-daily 7 --clean

# Custom GC tolerance
mapache forget --keep-daily 7 --clean --tolerance 10

Dry Run

mapache forget --keep-daily 7 --dry-run

Shows which snapshots would be kept and removed without making changes. Hooks are also skipped during dry runs (see Hooks).

12. Maintenance

clean — Garbage Collection

mapache clean -r <URL>               # Full GC with 0% tolerance
mapache clean --tolerance 10 -r <URL> # Tolerate 10% garbage per pack
mapache clean --no-repack -r <URL>    # Only remove unreferenced packs
mapache clean --dry-run -r <URL>      # Simulate

The garbage collector:

1. Scans all snapshots to determine which blobs are still referenced.
2. Identifies:
   • Unused packs — packs where no blobs are referenced (removed).
   • Obsolete packs — packs with blobs that will be repacked.
   • Small packs — packs smaller than the minimum size threshold (repacked).
   • Tolerated packs — packs with garbage below the tolerance threshold (kept as-is).
3. Repacks small packs and obsolete-pack survivors into new pack files.
4. Removes orphaned index entries.

--no-repack skips repacking (equivalent to tolerance = 100%).

The garbage collector can be safely interrupted with SIGINT/SIGTERM (Ctrl+C). The shutdown signal is checked at safe checkpoints between GC phases, so no data corruption occurs. This also applies to forget --clean.

Hooks are skipped when running with --dry-run (see Hooks).

verify — Integrity Verification

mapache verify -r <URL>                    # Logical consistency check
mapache verify --read-packs -r <URL>       # Full physical verification
mapache verify --read-packs --sample 10%   # Verify 10% of packs
mapache verify --read-packs --parallel 8   # Parallel verification
mapache verify --fail-early                # Stop on first error
mapache verify --with-cache                # Use local cache

Logical Verification (default)

• Checks that every snapshot references known tree blobs.
• Checks that every indexed blob points to an existing pack file.
• Verifies snapshot tree traversal integrity.

Physical Verification (--read-packs)

• Reads, decrypts, and hashes every blob in every pack (or a sample).
• Detects bit-rot (pack file hash mismatch) and individual blob corruption.
• Reports corrupt blobs and the files/snapshots they affect.

Sampling

--sample <PCT> verifies only a random percentage of packs. Useful for periodic integrity checks on large repositories.

rechunk — Re-process All Snapshots

mapache rechunk -r <URL>

Re-processes all snapshots using the current chunker parameters. Useful after chunker algorithm changes or when migrating between versions. Creates new blob versions and updates snapshot trees. Old blobs remain until cleaned by GC.

Requires an exclusive lock. May take significant time on large repositories.

13. Security & Key Management

Mapache uses a dual-layer encryption model:

1. Master Key — A random 256-bit AES-GCM-SIV key stored encrypted inside key files. This key encrypts all repository data.
2. User Keys — Derived from username + password via Argon2id KDF. Each user key encrypts the master key. Multiple users can access the same repository.

Key files can be stored inside the repository (keys/ directory) or exported as standalone files.

key list — List Keys

mapache key list -r <URL>

Shows all key files in the repository: username, key ID, creation date.

key add — Add a New User

mapache key add -r <URL>                     # Store key in repository
mapache key add --path ~/my-backup-key -r <URL>  # Export key file locally

Requires an existing user’s credentials to decrypt the master key, then prompts for the new user’s username and password.

key delete — Remove a User

mapache key delete KEY_ID -r <URL>

Deletes a key file by ID prefix. The affected user will no longer be able to access the repository.

key change-password — Change Password

mapache key change-password -r <URL>

Prompts for current credentials, then for a new password. Generates a new key file and deletes the old one.

key export — Export a Key File

mapache key export KEY_ID -r <URL>           # Exports to ./keyfile
mapache key export KEY_ID --output ~/safe/keyfile -r <URL>

Exports a key file from the repository to a local path. This file can be used later with --key-file to authenticate without username/password.

Security Best Practices

• Never lose your password — there is no recovery mechanism.
• Export key files to a secure location for disaster recovery.
• Use --key-file to authenticate without typing passwords (useful for scripts).
• The repository is a black box — all data is encrypted. An attacker with full read access to the storage cannot decrypt any content without the master key.

14. Bundle Files

The bundle command creates, extracts, or mounts self-contained .mapache bundle files. Bundles are encrypted, deduplicated archives that can be transferred to other systems.

Create a Bundle

# Create a bundle from files/directories
mapache bundle -a PATH [PATH...] -o bundle.mapache

# With exclusions
mapache bundle -a ~/project -o project.mapache -e "*.log" -e "tmp/"

# Custom compression
mapache bundle -a ~/Documents -o docs.mapache --compression best

Extract a Bundle

mapache bundle -x bundle.mapache -o /tmp/extract

If -o is omitted, extracts to the current directory.

Mount a Bundle (FUSE, Unix only)

mapache bundle -m bundle.mapache /mnt/bundle
mapache bundle -m bundle.mapache /mnt/bundle -c      # Auto-create mountpoint
mapache bundle -m bundle.mapache /mnt/bundle --metadata-only

Bundle-specific options:

Bundles are password-protected (AES-GCM-SIV). The password is independent from the repository password.

15. FUSE Mount

Mount a Repository (Unix only, fuse feature)

mapache mount /mnt/backup -r <URL>
mapache mount /mnt/backup -r <URL> --allow-other
mapache mount /mnt/backup -r <URL> --metadata-only
mapache mount /mnt/backup -r <URL> --create-mountpoint
mapache mount /mnt/backup -r <URL> --cache-size-mib 128

Mounts the entire repository as a FUSE filesystem. All snapshots are listed at the root; each snapshot directory contains the backed-up files.

Mount a Bundle

mapache mount bundle.mapache /mnt/bundle
mapache mount bundle.mapache /mnt/bundle -b    # Force bundle mode

If the path has a .mapache extension, mapache auto-detects it as a bundle.

Mount Options

Press Ctrl+C to unmount.

16. Snapshot Lifecycle

amend — Modify a Snapshot

# Modify tags
mapache amend --tags "updated,important" -r <URL>

# Modify description
mapache amend --description "Backup before upgrade" -r <URL>

# Clear metadata
mapache amend --clear-tags -r <URL>
mapache amend --clear-description -r <URL>

# Add exclusions post-hoc (rebuilds tree)
mapache amend --exclude "*.log" -r <URL>

# Apply to all snapshots
mapache amend --all --tags "archived" -r <URL>

# Keep the old snapshot file
mapache amend --keep-old -r <URL>

When exclusions are added, mapache rebuilds the snapshot tree (creating new tree blobs) without re-chunking file contents. The old snapshot file is deleted unless --keep-old is set.

recall — Recover Dropped Snapshots

mapache recall SNAPSHOT_ID -r <URL>

Restores a snapshot that was previously dropped (marked with .dropped extension) by forget. The snapshot is reactivated and will appear in log listings again.

17. Terminal User Interface

mapache tui -r <URL>

Launches an interactive terminal user interface with screens for:

• Dashboard — overview of the repository with version info and key stats
• File explorer — browse snapshot contents with inline detail panel
• Snapshot detail — inspect individual snapshots
• Snapshot creation wizard
• Restore wizard
• Forget/retention management
• Diff screen — navigable tree of changes between snapshots (</> to browse adjacent pairs, / to filter, u to toggle unchanged files)
• Find screen — real-time glob search across all snapshots with progress, results table, and direct navigation to explorer/restore

Requires a supported terminal.

18. Shell Completions

mapache completion --shell bash --path /etc/bash_completion.d/
mapache completion --shell zsh --path /usr/share/zsh/site-functions/
mapache completion --shell fish --path ~/.config/fish/completions/
mapache completion --shell powershell --path ~/Documents/PowerShell/

Generates shell completion scripts for the given shell. Supported shells include: bash, zsh, fish, powershell, elvish, and nushell.

19. Environment Variables

If MAPACHE_REPOSITORY is set, the -r flag can be omitted for all commands.

If MAPACHE_USERNAME and MAPACHE_PASSWORD are set, interactive authentication prompts are skipped.

20. Global Options Reference

These options are available on most commands (exceptions: bundle, cache, completion do not use repository connection).

Compression Levels

Bandwidth Limit Format

1000        -> 1000 bytes/s
1K / 1KB   -> 1000 bytes/s  (or 1024 for K)
1M / 1MB   -> 1,000,000 bytes/s  (or 1,048,576 for M)
1G / 1GB   -> 1,000,000,000 bytes/s  (or 1,073,741,824 for G)
1.5M       -> 1,572,864 bytes/s
10MiB/s    -> 10,485,760 bytes/s

Duration Format

30s     -> 30 seconds
5m      -> 5 minutes
5m30s   -> 5 minutes 30 seconds
1h      -> 1 hour
1d      -> 1 day
2w      -> 2 weeks
3m      -> 3 months (30-day months)
4y      -> 4 years (365-day years)

21. Full Command Reference

mapache init

Initialize a new backup repository.

mapache init -r <URL>

mapache snapshot

Create a new backup snapshot.

mapache snapshot [PATHS...] -r <URL>
  --as-root               Use a single directory as the snapshot root
  --exclude <PATH,...>    Paths to exclude (repeatable, or comma-separated)
  --exclude-file <PATH>   File with exclude paths (one per line)
  --tags <TAGS>           Comma-separated tags
  --description <TEXT>    Snapshot description
  --no-parent             Force complete analysis (no parent diff)
  --no-scan               Skip the initial file system scan
  --skip-if-unchanged     Skip saving if no changes detected
  --parent <ID|latest>    Use specific parent snapshot
  --readers <N>           Parallel file readers (default: 4)
  --packers <N>           Writer threads (default: 4)
  --dry-run               Simulate without making changes
  --stdin                 Read data from stdin as file /stdin
  --with-atime            Store file access times
  --pre-hook <CMD>        Shell command to run before snapshot (overrides config)
  --post-hook <CMD>       Shell command to run after snapshot (overrides config)

mapache restore

Restore a snapshot to a target path.

mapache restore [SNAPSHOT] --target <PATH> -r <URL>
  --target <PATH>         Restore destination (required)
  --include <PATH,...>    Paths to include (repeatable, or comma-separated)
  --include-file <PATH>   File with include paths (one per line)
  --exclude <PATH,...>    Paths to exclude (repeatable, or comma-separated)
  --exclude-file <PATH>   File with exclude paths (one per line)
  --strip-prefix          Strip longest common prefix from restored paths
  --strategy <MODE>       Conflict resolution: fail|skip|overwrite|newer
  --delete                Delete files in target not in snapshot
  --no-preserve-root      Allow --delete at root level
  --quit-on-error         Exit immediately on restore error
  --sparse                Create sparse files
  --verify                Verify content by hashing each blob (incremental restore)
  --batch-size <N>        Max files per batch (limits memory; default: no limit)
  --pre-hook <CMD>        Shell command to run before restore (overrides config)
  --post-hook <CMD>       Shell command to run after restore (overrides config)
  --dry-run               Simulate restoration

mapache forget

Remove snapshots and apply retention policies.

mapache forget [SNAPSHOT_IDS...] -r <URL>
  --force                 Immediately delete (don't just mark)
  --tags <TAGS>           Filter by tags
  --host <HOST>           Filter by host (repeatable)
  --keep-last <N|all>     Keep last N snapshots (or all)
  --keep-within <DUR>     Keep snapshots within duration
  --keep-hourly <N|all>   Keep N hourly snapshots
  --keep-daily <N|all>    Keep N daily snapshots
  --keep-weekly <N|all>   Keep N weekly snapshots
  --keep-monthly <N|all>  Keep N monthly snapshots
  --keep-yearly <N|all>   Keep N yearly snapshots
  --keep-tags <TAGS>      Always keep snapshots with these tags
  --keep-min <N|all>      Minimum snapshots to keep
  --dry-run               Show what would be removed
  --clean                 Run garbage collector after
  -t, --tolerance <%>     GC garbage tolerance (0–100)
  --pre-hook <CMD>        Shell command to run before forget (overrides config)
  --post-hook <CMD>       Shell command to run after forget (overrides config)

mapache log

Show snapshots in the repository.

mapache log [SNAPSHOT_ID] -r <URL>
  --dropped               Show only dropped snapshots
  --all                   Show active and dropped snapshots
  -c, --compact           Compact table listing
  --tags <TAGS>           Filter by tags

mapache ls

List nodes/files in a snapshot.

mapache ls [SNAPSHOT] -r <URL>
  --path <PATH>           Path within snapshot
  -l, --long              Long listing format
  -H, --human-readable    Human-readable sizes
  -R, --recursive         Recursive listing

mapache find

Find files and directories in the repository.

mapache find <TARGET> -r <URL>
  --snapshot <ID|latest>  Search in specific snapshot (all if omitted)
  --json                  Output results in JSON format

Exits with a non-zero code on error.

mapache diff

Show differences between snapshots.

mapache diff <SOURCE_ID> <TARGET_ID> -r <URL>

mapache dump

Print the contents of a file from a snapshot to stdout.

mapache dump [SNAPSHOT] -r <URL>
  --path <PATH>           Path to the file inside the snapshot (required)

mapache stats

Show repository statistics.

mapache stats -r <URL>
  --full                  Parse pack footers for physical stats (expensive)

mapache clean

Run garbage collection.

mapache clean -r <URL>
  -t, --tolerance <%>     Garbage tolerance (0–100, default: 0)
  --no-repack             Skip repacking
  --dry-run               Simulate without making changes
  --pre-hook <CMD>        Shell command to run before clean (overrides config)
  --post-hook <CMD>       Shell command to run after clean (overrides config)

mapache verify

Verify repository integrity.

mapache verify -r <URL>
  --read-packs            Read, decrypt, hash all data (slow)
  -p, --parallel <N>      Parallel pack reading (default: 4)
  --with-cache            Use local cache (disabled by default)
  --fail-early            Stop on first error
  --sample <%>            Verify random sample of packs (e.g., 10.5%)
  --pre-hook <CMD>        Shell command to run before verify (overrides config)
  --post-hook <CMD>       Shell command to run after verify (overrides config)

mapache key

Manage repository key files.

mapache key list -r <URL>
mapache key add -r <URL> [--path <PATH>]
mapache key delete <KEY_ID> -r <URL>
mapache key change-password -r <URL>
mapache key export <KEY_ID> -r <URL> [-o <PATH>]

mapache bundle

Create, extract, or mount .mapache bundle files.

mapache bundle -a <INPUT...> -o <OUTPUT.mapache> [OPTIONS]
mapache bundle -x <INPUT.mapache> [-o <DIR>] [OPTIONS]
mapache bundle -m <INPUT.mapache> <MOUNTPOINT> [OPTIONS]

  -a, --bundle            Create mode
  -x, --extract           Extract mode
  -m, --mount             Mount mode (FUSE, Unix only)
  -o, --output <PATH>     Bundle file (-a) or destination (-x)
  -e, --exclude <GLOB>    Exclude globs (bundle mode)
  --compression <LEVEL>   Compression level (bundle mode)
  --readers <N>           Parallel readers (default: 4)
  -c, --create            Create mountpoint (mount mode)
  --allow-other           Allow other users (mount mode)
  --metadata-only         Don't load file contents (mount mode)
  --cache-size-mib <MIB>  Data cache size (mount mode, default: 256)

mapache mount

Mount repository or bundle as FUSE filesystem (Unix only).

mapache mount <MOUNTPOINT> -r <URL>
  -b, --bundle            Force bundle mount
  --allow-other           Allow other users
  -c, --create-mountpoint Create mountpoint if missing
  --metadata-only         Don't load file contents
  --cache-size-mib <MIB>  Data cache size (default: 64)

Exits with a non-zero code on error.

mapache cat

Print repository objects.

mapache cat manifest -r <URL>
mapache cat <type>:<ID> -r <URL>
  Types: manifest, pack, blob, tree, index, key, snapshot, lock

mapache amend

Modify an existing snapshot.

mapache amend [SNAPSHOT] -r <URL>
  -a, --all               Apply to all snapshots
  --keep-old              Keep the old snapshot file
  --tags <TAGS>           Replace tags
  --clear-tags            Remove all tags
  --description <TEXT>    Replace description
  --clear-description     Remove description
  --exclude <PATH,...>    Exclude paths (rebuilds tree)
  --exclude-file <PATH>   File with exclude paths

mapache recall

Recover a dropped (forgotten) snapshot.

mapache recall <SNAPSHOT_ID> -r <URL>

mapache rebuild-index

Rebuild the index by scanning all packs.

mapache rebuild-index -r <URL>
  --dry-run               Simulate without making changes

mapache rechunk

Rechunk all snapshots with current parameters.

mapache rechunk -r <URL>

mapache copy

Transfer snapshots between repositories.

mapache copy --from <URL> -r <URL>
  --from <URL>                Source repository (required)
  --from-ssh-privatekey <PATH>  SSH key for source repository
  --from-ssh-known-hosts <PATH>  SSH known hosts file for source
  --snapshot <PREFIX>         Copy only snapshots with this ID prefix (repeatable)
  --host <HOST>               Copy only snapshots from this host (repeatable)
  --tags <TAGS>               Copy only snapshots with this tag (repeatable)
  --dry-run                   Show what would be copied without transferring

mapache sync

Synchronize a repository to another backend.

mapache sync --target <URL> -r <URL>
  --delete                Delete files in destination not in source
  --dst-ssh-privatekey <PATH>  SSH key for destination
  --dst-ssh-known-hosts <PATH>  SSH known hosts file for destination
  --dry-run               Simulate without making changes

mapache unlock

Remove stale locks.

mapache unlock -r <URL>
  -f, --force             Remove all locks (including non-expired)

mapache cache

Manage local cache directories.

mapache cache
  --delete <PREFIX>...    Delete cache folders by ID prefix (repeatable)
  --clear                 Delete all cache folders

mapache completion

Generate shell completion scripts.

mapache completion --shell <SHELL> --path <DIR>
  --shell <SHELL>         Shell type (bash, zsh, fish, powershell, elvish, nushell)
  --path <DIR>            Output directory

mapache config

Manage repository configuration.

mapache config template [--output <PATH>]
  --output <PATH>         Write template to file instead of stdout

mapache tui

Launch terminal user interface.

mapache tui -r <URL>

22. Troubleshooting & FAQ

Lock Issues

Q: I get “repository is locked” errors. What should I do?

A: If a previous command crashed, run mapache unlock -r <URL> to remove stale locks. For persistent issues, use --force. To avoid this in scripts, use --retry-lock to automatically wait for locks to be released. For read-only operations (e.g. log, ls, find), use --no-lock to skip lock acquisition entirely.

Performance

Q: Backups are slow on large file trees.

A: Adjust parallelism with --readers and --packers. Monitor disk I/O and CPU usage. Consider using a faster compression level (fastest) for initial backups, or --exclude to skip unnecessary paths.

Repository Corruption

Q: I suspect my repository is corrupted.

A: Run mapache verify -r <URL> to check logical consistency or mapache verify --read-packs -r <URL> for full physical verification. If the index is corrupted but packs are intact, use mapache rebuild-index -r <URL>.

Storage Space

Q: The repository is using more space than expected.

A: Run mapache stats --full -r <URL> to see compression ratios and dangling blobs. Then mapache clean -r <URL> to run garbage collection. If you have deleted many snapshots, consider mapache clean --tolerance 0 -r <URL> for aggressive cleanup.

Authentication

Q: I forgot my password. Can I recover it?

A: No. Mapache uses the password to derive the key that decrypts the master key. Without the password, the data is unrecoverable. Always export key files as a backup: mapache key export <KEY_ID> -o ~/safe/keyfile -r <URL>.

Cross-Platform

Q: Can I use the same repository from Windows, Linux, and macOS?

A: Yes, mapache repositories are platform-independent. However, FUSE mount is only available on Unix systems. Bundle files are fully portable.

Data Safety

Q: Is it safe to delete old snapshots?

A: Yes. Deleting a snapshot only removes its metadata file. The underlying data blobs are reference-counted by the index. Garbage collection (clean) removes only blobs that are no longer referenced by any snapshot.

Q: Can I interrupt a snapshot in progress?

A: Yes. Press Ctrl+C. All data written so far is persisted. Unfinished packs not yet indexed are harmless — they will be ignored and eventually cleaned up by GC. The snapshot will not be saved.

Design

Introduction

The mapache repository

Mapache implements a repository as its central location for data storage. The mapache repository is a collection of content-addressable files and binary objects distributed in a centralized layout in the repository directory. Content-addressable means that, with rare exceptions, all files and objects are indexed not by a name or an arbitrary identifier, but the hash of its binary contents. When mapache writes a snapshot file to the repository, it will first calculate the hash of its content and that hash will be the name of the file.

This way of indexing data offers two interesting properties, provided that a strong enough hash is used to avoid collisions:

1. Two identical objects will have the same ID, therefore if an object O with content C and hash H exists, all binary streams with hash H are guaranteed to have content C. This allows us to skip writing a second copy of the identical data and grants us the certainty that content C exists already in the repository.

2. If the contents of an object changed due to error or external manipulation, its hash would not correspond to its contents. If the hash were also manipulated to match the content, this new ID would no longer represent the original data. This allows us to detect errors or external manipulation of the data.

In the context of mapache, an object is a stream of binary data represented by its hash, or ID, regardless of how it is stored. An object can be stored as a single file, with its filename being the ID of the object, or bundled with other objects in a file with any known arbitrary format. In any case, the filename will identify its contents as a whole by their hash.

The mapache repository is a tree directory that contains files with different purposes:

repo
├── index
│   └── 771f7c9d05241c55a87e004b148310953c1565ef10f55daf02a68ba0ea4e0166
├── keys
│   └── cea3212843e488b168f6bb21b757d6cb4882a559e10ec4b33f305e982c4c3c8e
├── locks
├── manifest
├── objects
│   ├── 00
│   ├── 01
│   ├── ..
│   ├── ff
│   │   └── ff890e7aeb656e7f57004e05009fa66576448a150c6471d38e11e4e6d0f3227b
└── snapshots
    └── 1d0e210a91f77744f87420387e52a108b87fd900a1d8462f3b9def4b99a47b9e
    └── 84afca27f7c134131cf8d33a2ae70114889c3c31dc281b7fffad998e7ddb569a.dropped

Storage

Mapache is a de-duplicating, incremental backup program. It implements a content-defined chunking algorithm to split files into small chunks. Every chunk is identified by the hash of its raw data, or ID. This algorithm is able to detect content boundaries, so when you modify the file, the algorithm can detect the new data added as a series of new chunks. All these file chunks, or data blobs are stored in the objects directory, which works as a blob library. Since we can identify every blob by its ID, a single blob is stored only once. If two files share a blob, this blob is saved only once in the repository. When a file is saved along multiple snapshots, mapache will chunk the file and only store the new blobs that do not exist already in that blob library.

A snapshot is akin to a still photograph of the file system at a point of time. The content of files will be deduplicated and encoded into blobs in the object storage. However, if we only stored data blobs, we would not be able to reconstruct the original file system tree, the metadata, permissions, file times and attributes. Mapache stores the tree metadata as blobs in the object storage. File nodes contain the list of data blobs that constitute its contents, symlinks contain the target path, directory nodes have a reference to its own subtree, etc. These metadata blobs are also deduplicated so that if two trees are exactly identical, there only exists one copy in the object storage. This deduplication reduces the amount of data necessary to store the snapshot metadata, since most of the time a snapshot only introduces a few modified or new trees. The snapshot file will reference the root tree, and contain metadata like a timestamp, host, tags, description, etc.

Blobs in the object storage are not stored as individual files. We could do that, but it would be highly inefficient. A generic repository could potentially contain millions of blobs, exhausting the available file system inodes and requiring too many I/O operations to store all the blobs. Instead, blobs are packed in pack files. A pack file contains multiple blobs and some metadata describing them. Pack files can be configured to have any size up to 4 GiB. In order to find blobs efficiently within the pack files, the mapache repository maintains an index. This index, composed of one or more index files, describes the packs and their contents and allows mapache to find a blob by mapping the blob ID to the pack ID, its offset, length and type.

The index is the central piece of how mapache works. If a blob ID is not referenced by the index, the blob does not exist for mapache and will be subject to elimination by the garbage collector, even if there is a pack file that contains it. This allows mapache to append new data atomically. If a backup is interrupted, all blobs and packs not referenced by a persisted index will be left dangling, as if they were never added. Restarting the backup will not resume the process from where it was interrupted, but all indexed blobs will not be rewritten. To avoid losing all progress, mapache may periodically persist all finalized indices to file.

Chunking

Mapache uses a custom implementation of FastCDC. FastCDC is a Content-Defined Chunking algorithm proposal that offers a speedup of 3x - 12x with respect to Rabin-based CDC with comparable deduplication ratio.

Compression and encryption

A good chunker can do wonders to reduce the amount of data needed to store all the files of a snapshot, but we can go one step further. Mapache uses zstd to compress almost all the data stored in the repository, including data and tree blobs. The compression level can be configured, affecting the compression ratio and the total time needed to take a snapshot.

Security is a non-negotiable aspect of mapache by design. Everything except for a handful of bytes is encrypted. Encryption cannot be disabled or opted-out. Mapache implements AES-GCM-SIV. This is a modern cipher which implements encryption and authentication of data with a key. AES-GCM-SIV not only encrypts the data, it adds an authentication layer that allows mapache to detect when an object has been altered or manipulated.

Master key and key files

All the encrypted content in the mapache repository is encrypted with a unique 256-bit master key. This key is randomly generated when the repository is created. Users do not access the repository with the master key directly. Instead, they are asked for a username and a password. The password is used to derive a 256-bit personal key using Argon2. The personal key is solely used to encrypt personal copies of the master key in a KeyFile. Every user has a KeyFile that they use to open the repository. These KeyFiles can be stored in the keys directory or provided externally.

Mapache does not implement master key rotation at the moment, but it may do so in the future.

Hashing

All content IDs (hashes) used by mapache to identify objects in the repository are generated with a high-performance cryptographic hashing algorithm. The current implementation uses BLAKE3, which produces 256-bit hashes.

Threat model

Mapache assumes that the host system is secure. However, the repository itself may reside on an untrusted medium where an attacker has the ability to read, modify, or delete the repository files.

Security Guarantees

Mapache provides strong guarantees for the security of the data, but it cannot prevent file system operations by an external attacker:

• Confidentiality: An attacker should not be able to decipher the stored data.
• Integrity (Verifiability): Mapache can detect if the data has been modified or corrupted.

Protection Mechanisms

Mapache uses a dual-layer approach to ensure data integrity and confidentiality:

• Authenticated Encryption with Associated Data (AEAD): All data is protected using an AEAD cipher (AES-GCM-SIV). This means the information is both encrypted (ensuring confidentiality) and authenticated. The authentication layer allows Mapache to detect if a file has been altered or manipulated externally.
• Content-Addressability: Objects are identified by the cryptographic hash (ID) of their content. This provides a secondary and independent verification mechanism to ensure the content has not been corrupted.

Attacker Limitations

Due to the mandatory encryption and authentication:

• An attacker who reads the data will only obtain unintelligible, encrypted information.
• An attacker who modifies or renames files will cause the corresponding integrity check (both AEAD and hash verification) to fail, rendering the data unusable by Mapache.

Therefore, the most effective action an attacker on the untrusted medium can take is the destruction (deletion) of the repository files. Mapache is unable to prevent files in the repository from being modified, renamed, or deleted by an external entity with access to the storage medium.

Repository format

The mapache repository stores metadata in JSON format.

Manifest

The manifest file describes the repository: its ID, creation time and version number. The ID is a 256-bit identifier generated randomly at the time of creation.

The manifest file is compressed and encrypted with the master key.

{
  "version": 1,
  "id": "b7468f6331331302b06c63b98a14e50107f9cc26683afd064c0af84eec53b3e7",
  "created_time": "2025-11-20T13:06:52.266751300+01:00"
}

Snapshots

The snapshot file contains metadata about a snapshot: timestamp, user name, host name, paths, root tree, summary, etc. The file name (ID) is the hash of the encoded content.

{
  "timestamp": "2025-11-20T19:00:00.375479500+01:00",
  "tree": "b266241241162f84e91165d96d03de0719b385c5b2918cec02b5324869e6145c",
  "root": "I:\\mapache",
  "paths": [
    "I:\\mapache\\core"
  ],
  "hostname": "cocoon",
  "username": "mapache",
  "summary": {
    "processed_items_count": 94,
    "processed_bytes": 744155,
    "raw_bytes": 744155,
    "encoded_bytes": 197650,
    "meta_raw_bytes": 42694,
    "meta_encoded_bytes": 18639,
    "total_raw_bytes": 786849,
    "total_encoded_bytes": 216289,
    "new_files": 79,
    "deleted_files": 0,
    "changed_files": 0,
    "new_dirs": 15,
    "deleted_dirs": 0,
    "changed_dirs": 0,
    "unchanged_files": 0,
    "unchanged_dirs": 0
  }
}

Packs and blobs

Mapache stores backup data in the objects directory. A fanout of 1 byte is used to reduce the number of files per directory: 256 directories with names 00 to ff contain the pack files. A packfile with ID dd2856e... will be stored in objects/dd/dd2856e9f7ac9b495e7ebce1370f68a2e60824cee9546b817f7cbbb4d571bd4b.

All blobs are stored in pack files, which group multiple blobs together to optimize storage and I/O operations. The pack file name (ID) is the hash of its content. Since a pack file contains already pre-encoded blobs and an encoded footer, no further encoding is applied.

The layout of a pack file is the following.

Encoded pack 1 Encoded pack 2 … Encoded pack N PACK FOOTER

The blobs are encoded and appended to the pack. Their ID is the hash of their raw content. The pack footer describes the blobs contained in the pack footer. This footer includes a 4-byte field which is the length of the total footer. The pack footer is encoded, except for the length field.

Trees

Mapache stores file contents (data blobs) as well as tree metadata (tree blobs). Tree and data blobs are stored in separate packs. The data blobs contain chunks of the file contents. The tree blob contain metadata about the file system tree nodes in JSON format. Data and tree blobs are stored in pack files in the same way. Each tree is stored as a single blob. A tree node can be a file, directory, symlink, etc. Directories have a tree field which points to its subtree. File nodes have a blobs field which contains a list of its data blobs in order. Symlinks have a field symlink_info which contains metadata about the target path and node type of the target.

{
  "name": "tests",
  "type": "directory",
  "metadata": {
    "size": 0,
    "created_time": {
      "secs_since_epoch": 1763560510,
      "nanos_since_epoch": 935069400
    },
    "modified_time": {
      "secs_since_epoch": 1763560510,
      "nanos_since_epoch": 937075900
    }
  },
  "tree": "093ca4db9e88163d1e9e815e01d9b3a6e7e7980d21a674e433eb75fdb2f133cd"
}

{
  "nodes": [
    {
      "name": "integration_tests",
      "type": "directory",
      "metadata": {
        "size": 4096,
        "created_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 935069400
        },
        "modified_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        }
      },
      "tree": "9a426139fa0315872bb3da4ee6ed6f0955f7425846efd0c22c3d040d7f9dc9d7"
    },
    {
      "name": "tests.rs",
      "type": "file",
      "metadata": {
        "size": 834,
        "created_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        },
        "modified_time": {
          "secs_since_epoch": 1763560510,
          "nanos_since_epoch": 937075900
        }
      },
      "blobs": [
        "6772c0355a4f84a095343951e371677fc88ba92820bc4e12ab78c9de61ae3ea2"
      ]
    }
  ]
}

Pack footer format

The pack footer consists of a variable-length list of metadata blob entries, each 41 bytes long, followed by a fixed-size trailer. Each entry contains an ID (32 bytes), the blob type (u8), and both the encoded length and raw length (u32) of the associated data blob. The trailer is a single u32 field which stores the total length of the entire pack footer, allowing a parser to efficiently skip directly to the file’s data section. All data except the footer length field is encrypted.

┌────────┬────────┬─────┬────────┬─────────────────────┐
│ Blob 1 │ Blob 2 │ ... │ Blob N │ Footer length (u32) │
└────────┴────────┴─────┴────────┴─────────────────────┘

┌──────────────────────────────────────────┐
│     Pack footer blob entry (41 bytes)    │
│────────────────────────┬────────┬────────│
│ Field                  │  Size  │ Offset │
│────────────────────────┼────────┼────────│
│ ID (256-bit raw hash)  │   32   │   0    │ Hash of the raw blob data
│────────────────────────┼────────┼────────│
│ Blob type (u8)         │    1   │   32   │
│────────────────────────┼────────┼────────│
│ Encoded length (u32)   │    4   │   33   │ Length of the encoded blob
│────────────────────────┼────────┼────────│
│ Raw length (u32)       │    4   │   37   │
└────────────────────────┴────────┴────────┘
  ^ (41 bytes)

Index

The index files, located in the repo/index directory, are the repository’s central lookup mechanism and source of truth for all stored data.

These files contain metadata about the repository’s pack files. Crucially, they act as a map, linking the ID (cryptographic hash) of every content-addressable tree and data blob to its specific physical location, including the Pack ID, offset, and length within that pack.

The index determines what data exists in the repository; any pack or blob data not referenced by an index is considered non-existent and subject to garbage collection, which is key to mapache’s atomic append mechanism. The filename of the index file itself is the hash of its encoded content.

{
  "packs": [
    {
      "id": "c2334c5e29563850cb254b63fba125bb1b60e860b1886fae7bd4f5e5a637f4c0",
      "blobs": [
        {
          "id": "22bb6896db82b2fcc2d8057d6e16298e8318a04d1e0dd5b6c84fadc46a234f4c",
          "type": "Data",
          "offset": 25763,
          "length": 3265,
          "raw_length": 12108
        },
        {
          "id": "99ce77389066f2982e8350eefdbf21ccef026d94fc4827dbc24cd52d77e25e7f",
          "type": "Data",
          "offset": 73520,
          "length": 2462,
          "raw_length": 7515
        },
      ]
    },
    {
      "id": "9a2ad285d1dd3d9fd7ca7b8fc8bcbcbd62ef9871bbf1f74757ad392a58e4d71e",
      "blobs": [
        {
          "id": "ffcdcfc7ac3832168c952f4e396588b7cb523e009c0abf776c46806468146985",
          "type": "Tree",
          "offset": 3188,
          "length": 897,
          "raw_length": 3514
        },
        {
          "id": "23f87643e00c9f7c498c057a58ed2f38af22402eaab9e7b58a8bfa505b11756b",
          "type": "Tree",
          "offset": 5206,
          "length": 863,
          "raw_length": 3475
        },
      ]
    }
  ]
}

Locks

Lock files are the locking and synchronization mechanism at the repository level. When mapache opens a repository, it may need to acquire an exclusive or non-exclusive lock. Non-exclusive locks allow concurrent access with other non-exclusive locks. They are used for constructive operations which do not affect other users. Exclusive locks are used to get single, exclusive access to the repository for destructive operations like forget or clean.

A lock is considered expired if its timestamp is older than a predetermined threshold. Expired locks are not considered when trying to acquire a new lock and get deleted by the garbage collection mechanism. During normal operation, locks get periodically refreshed. If the process that holds a lock terminates abruptly, it might not release its lock. The expiration time serves as a way to remove those locks which did not get deleted.

The lock name is a unique 256-bit ID generated randomly at the time of creation and has nothing to do with its content. A lock that is refreshed will conserve its ID, even though the timestamp will change.

{
  "id": "3468acbec193d65560c8ff16fea30e2c566d48ddf1234aefa9b1bac09c1cf71f",
  "timestamp": "2025-11-20:00:00.390485800+01:00",
  "exclusive": false,
  "hostname": "cocoon",
  "username": "mapache",
  "pid": 4216
}

Keys

KeyFiles serve as keys to the repository. They contain metadata about the user, creation time, parameters for the password derivation function and the encrypted master key, using the password and a salt. KeyFiles can be stored in the keys directory or outside of the repository. KeyFiles are compressed but not encrypted. The KeyFile name (ID) is the hash of the encoded (compressed) content.

{
  "created": "2025-11-20T19:00:00.663953700+01:00",
  "username": "mapache",
  "m": 19456,
  "t": 2,
  "p": 1,
  "salt": "VRKrdGuf5T8iW3b5VYgJ+bMTMrYKRphbxkq2qW6C7VQ=",
  "encrypted_key": "bkwxbjuLbV094UiRFoO5t2YlV3ZhsCQdn3z2A+YyPSNewGFUo+AwL+L7Jae406yW4OHcHzFwN3wv7Vlj"
}

Local cache

Mapache maintains a local cache to speedup some commands and minimize the number of remote I/O operations. Certain metadata like tree blobs, snapshots and indices are cached locally to avoid having to download them from a remote backend. The cache is a mirror copy of the mapache repository layour, but it only contains those files that are eligible for caching. These cache directories are stored in the usual cache directory of the OS.

Mapache provides a cache command to list and manage cache folders, including a cleanup option.