T.A.R.D.I.S.

• Time-saving
• Automated
• Repository
• Data
• Integration &
• Synchronization

it's bigger on the inside - your backups, snapshots, and syncs all in one.

description

a cli utility combining btrfs, mergerfs, rclone, and borg into a solution. for managing backups, snapshots, syncing, and storage through one interface.

think of it as a borgmatic but for more than just borg. overcomplicated? maybe. lots of storage, backups, and snapshots? yes please :3

WARNING this is very much alpha level software with not a whole lot of testing.

P.E.M.S.A.R.

Please Excuse My Shitty Ass Rust. I'm still new to the language so please half merthcy :3 feel free to open a PR or make sugesstions.

installation + usage

dependencies

• mergerfs
• rclone
• borg
• btrfs filesystem
• cargo
• rustup
• just (optional)

setup

if you want to use the .justfile

just install

if you'd rather just use cargo

cargo build --release
sudo install -Dm755 target/release/tardis /usr/local/bin/tardis
sudo install -Dm644 service/systemd.service /etc/systemd/system/tardis.service

you then have to adjust the example config to your system

sudo mkdir /etc/tardis
sudo cp example.config.toml /etc/tardis/config.toml
sudo nvim /etc/tardis/config.toml

usage

you can review all the different commands using the --help flag.

sudo tardis --help

the main concept is it's an orcestration tool. you could have any number of timers, drives, subvolumes, snapshots, mounts, backups, and mergerfs pools.

you filter based on the toml GROUPS/LABELS and various cli flags. based on your criteria it'll execute the desired action for the specified resources.

an example with btrfs in reference to the example.config.toml:

# create a btrfs snapshot of the @data subvolume on buffer drive
sudo tardis snapshot create buffer @data

# list snapshots newer than a day using humantime
sudo tardis snapshot list --newer-than 1day buffer @data

# prune snapshots newer than a day using humantime
sudo tardis snapshot prune --newer-than 1day buffer @data

when you run tardis without any commands or flags, I'll run in daemon mode. daemon mode is controlled by the concept of timers. you specify in CRON like syntax when you'd like the timer to go off and it's criteria.

[timers.nightly_snapshots_2am]
schedule = "0    0    2    *    *    *"

it's pretty much CRON syntax with the exception of also having seconds at the front.

you then specify on resources that support timers:

[[btrfs.cache.snapshots]]
source = "@data"
destination = "@snapshots"
keep = 3
timers = ["nightly_snapshots_2am"]

you can also specify subsequent timers as a way to define what comes after:

[timers.nightly_syncs_2am]
schedule = "0    0    2    *    *    *"
after = ["nightly_snapshots_2am"]

which would then based on the action you assigned it to, will execute after:

[sync.cache_to_buffer]
preallocate = true
remove_source = false
source = "/mnt/tardis/cache/@data"
destination = "/mnt/tardis/buffer/@data"
timers = ["daily_syncs_2am"]

road_map

I have a few goals for the project, ordering in priority:

• mange/configure all the things through toml.
• intuitive cli for making one off snapshots and backups.
• be able to have some sort of systemd service.
• have some easy/interactive way to create each of the resources.
• polish the hell out of the tool and add some testing
• expand to more than just btrfs?
• support plist files and timemachine(maybe?) for macos backups?
• some sort of GUI for the normies?
• windows support?

support

this is a passion project of mine and I want it to be free for everyone forever. if you like my work or would love to see more of it, please support my efforts and buying me a coffee over at my ko-fi page.

you can also follow the development over at my twitch or youtube channel.

flowcharts

this is really just for my own sanity but your're free to explore.

example of my usecase this solves/simplifies:

flowchart TD
    archive_bucket["Archive Bucket  (Wasabi vfs rclone mount)"]
    snapraid["5TB SnapRAID Drive (external)"]
    subgraph mergerfs_pool["MergerFS Pool"]
        subgraph cache["1TB Cache Drive (internal)"]
            cache_data["@data"]
            cache_rclone["@rclone"]
            cache_restic["@restic"]
            cache_snap["@snapshots"]
            cache_data --> |daily snapshot before sync| cache_snap
        end
        subgraph buffer["5TB Buffer Drive (internal)"]
            buffer_data["@data"]
            buffer_snap["@snapshots"]
            buffer_data --> |daily snapshot before sync| buffer_snap
        end
        
        hot_bucket["Hot Bucket (Wasabi vfs rclone mount)"]
        cache_data --> |epmfs fallback| buffer_data --> |epmfs fallback| hot_bucket
    end

    subgraph cache_to_buffer["daily sync"]
        usage1{usage > 0.5TB}
        crit1["sort by size + mtime/atime >= 7 days"]
        usage1 --> |true| crit1 --> sync1[sync] --> checksum1{checksum} 
        checksum1 --> |success| delete1[delete]
        checksum1 --> |"fail (with X retires)"| sync1

        usage1 --> |false| sync2[sync] --> checksum2{checksum}
        checksum2 --> |"fail (with X retries)"| sync2
    end
    subgraph buffer_to_hot["daily sync"]
        usage2{usage > 5TB}
        crit2["sort by size + mtime/atime >= 15 days"]
        usage2 --> |true| crit2 --> sync3[sync] --> checksum3{checksum}
        checksum3 --> |success| delete3[delete]
        checksum3 --> |"fail (with X retries)"| sync3

        usage2 --> |false| sync4[sync] --> checksum4{checksum}
        checksum4 --> |fail| sync4
    end
    subgraph hot_to_archive["daily sync/restic snapshot"]
        usage3{"TTL(mtime) > 30 days"}
        usage3 --> |true| restic1[restic snapshot] --> |"(on exit 0)"| delete4[delete]
        usage3 --> |false| restic2[restic snapshot]
    end

    cache_data --> |evict| cache_to_buffer --> |evict| buffer_data
    buffer --> |parity| snapraid
    buffer_data --> |promote| buffer_to_hot --> |promote| hot_bucket
    hot_bucket --> |snapshot| hot_to_archive --> |snapshot| archive_bucket

    hot_bucket --> |rclone cache| cache_rclone
    archive_bucket --> |restic cache| cache_restic