How to Borg Backup

BorgBackup Github

BorgBackup Documentation

 

Introduction

What is BorgBackup?

BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.

The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

 

Main Features

Space efficient storage

Deduplication based on content-defined chunking is used to reduce the number of bytes stored: each file is split into a number of variable length chunks and only chunks that have never been seen before are added to the repository.

Speed
  • performance critical code (chunking, compression, encryption) is implemented in C/Cython
  • local caching of files/chunks index data
  • quick detection of unmodified files
Data encryption

All data can be protected using 256-bit AES encryption, data integrity and authenticity is verified using HMAC-SHA256. Data is encrypted clientside.

Compression

All data can be optionally compressed:

  • lz4 (super fast, low compression)
  • zstd (wide range from high speed and low compression to high compression and lower speed)
  • zlib (medium speed and compression)
  • lzma (low speed, high compression)
Off-site backups

Borg can store data on any remote host accessible over SSH. If Borg is installed on the remote host, big performance gains can be achieved compared to using a network filesystem (sshfs, nfs, …). Backups mountable as filesystems

Free and Open Source Software
  • security and functionality can be audited independently
  • licensed under the BSD (3-clause) license, see License for the complete license

The catch with BorgBackup is that it takes some time to understand the concept and use it appropriately. It definitely takes some time getting used to, especially for people not familiar with advanced backup solutions. But once you get the gist of it, it will definitely be a pleasure and I am sure a majority would stick to this solution for most backup targets.

 

Here's how I personally would explain how it works and how I use it

The program has 2 main functions. The first is creating a repository, the second is creating an archive within a repository.

 

Repository

The repository is basically the world that stores all the content as archives for you. The special thing about this world is that every single thing exists only once. Everyone can have a simulation of all things, e.g. everyone can have an orange and use it in this world, but actually there is literally only a single orange in the whole world.

 

Archive

An archive is a certain state of the world, defined by the time the state (snapshot, basically) was captured. Let's say, yesterday your brother had an orange, but your sister and you had none. Today, your brother gives you an orange, so you have one now, but your sister and brother have none. If you create a snapshot, i.e. archive, every day in the evening, then there will be 2 archives. 1 from yesterday where your brother had an orange but your sister and you didn't, the second one is from today where you have an orange but your brother and sister have none. That means that you have 2 entire archives that are basically standalone (you can delete one of them and the other one will remain as it should) while the space used for both archives equals to the space only 1 archive uses because all that changed in the world is that the orange changed its owner, so no additional data was added, which means that the size of the new archive seems to increase by 0 because the other archive already contains the needed data for the new archive. Now if your sister gets an orange tomorrow, so that you and your sister have one each now, then the archive from tomorrow will only increase the size of the respository by a couple of bytes (if the size of owning an orange would be a couple of bytes, that is; NOTE: the orange itself does not get duplicated, the only thing that gets saved additionally, is that your sister has the orange now, but the orange exists only once in the whole world, as explained above).

Now comes the even more interesting part. Let's say, every day there are major changes in the whole world but the only thing you care about is the orange situation at home, for now. Your very first archive already contains the whole world ( i.e. e.g. root directory / ). Now further backups only make a snapshot of the orange situation ( i.e. e.g. /home/*/oranges-directory ). This directory is part of the root directory so all the data is already in the initial backup and doesn't need to be additionally stored. The only thing that is stored in the newest archive, are the changes in the oranges-directory, effectively ignoring all other changes in other places.

 

Real world example

I had a repository containing an initial archive of my root directory /. Yesterday, I created an additional archive of my Downloads folder, because I downloaded some .deb files; i.e. /home/user/Downloads. Today, I updated my Debian archive mirror, so I only backed up the /var/debian folder. Tomorrow, I will update the
whole root directory / once again.

How much space will all this use? I have 2 separate backups from 2 separate days from the whole root directory / and yet all the space that will be used is pretty much the space that the whole root directory / + the couple of .deb files I downloaded, need. Nothing else. My Debian archive mirror only updated the packages, didn't add any new ones. My system overall didn't change much, except I have a couple more .deb files in my Downloads folder. So you can pretty much have 100s of different archives, each saving the state of when the snapshot was taken and at what location, but the size won't increase, at all, except you actually add entirely new data. Therefore it already takes almost no space to backup everything you need to backup, and yet you can optionally compress everything, too, so the space needed is EVEN SMALLER.

 

Real world example from my Raspberry Pi system:

The root directory / of my Raspberry Pi 3B takes about 12-14GB of space on my SD card. The actual initial Borg archive of the whole SD card takes up about 4GB in space, after low compression (so you can compress the data even higher if you have a more compute ready machine). Now, do you have several Raspberry Pis but don't want to use ~4GB for each Raspberry Pi? No problem, just make archives of all the different Pis in the same repository and if the data on all the Raspberry Pis is more or less the same datawise, then the repository will be maybe ~4.5-5GB in size, despite backing up 4 Raspberry Pis (real world example from my own setup).


 

I hope I could explain the system well enough to you, since I had to try out BorgBackup several times to finally get the gist of how to use it at best.

P.S.: You can also safely encrypt all your backup data. I personally don't need that option, but it definitely pumps up the value of this backup solution by a whole lot, as well.


 

Borg 1.1.9

Installation

Debian

Install Dependencies
sudo apt install python3 python3-dev python3-setuptools python3-wheel python3-pip python-virtualenv libssl-dev openssl libacl1-dev libacl1 build-essential libfuse-dev fuse pkg-config

 

If on Ubuntu, you should also run:

sudo usermod -aG fuse #appendyourusernamehere

 

Install BorgBackup through Python pip

Note that this part can take some time, especially on very old machines or SBCs, for example.

sudo pip3 install borgbackup[fuse]

 

User Guide

Create backup of your development files to another hard drive

  • ~/src is the folder with your development files
  • /dev/sda i.e. / is where your system is installed and /dev/sdb i.e. /mnt/usb-hdd0 is an attached
    external hard drive
# creates directory for backed up files
mkdir -p /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/
# changes to this directory
cd /mnt/usb-hdd0/backups/borg/repos/
# initializes a borg repository
borg --verbose init --encryption=none $HOSTNAME
# verify the creation was executed successfully
cat README
# creates the initial archive for your dev files in the above created repository
borg --verbose --progress create --stats --comment "Initial backup of my dev files." /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/::firstbackup /home/$USER/src

 

Now backing up your dev files the next day, again...

borg --verbose --progress create --stats --comment "Backup of my dev files after several commits to my main project." /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/::secondbackup /home/$USER/src
# creates the second archive for your dev files in the above created repository

 

Wait, when did I back up my dev files the last time and how much space do they take, exactly?

borg --verbose info --last 1 /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/
# shows all information about the newest backup in this repository
# e.g. time/date of creation, comment, size, etc.

 

Wait, what files exactly did I back up in my last backup?

borg --verbose list --short --last 1 /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/
# prints all file and directory names contained
# within the last archive of this repository

 

Create a complete backup of your entire OS

In case you didn't create a repository as in the above example, yet:

# creates directory for backed up files
mkdir -p /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/
# changes to this directory
cd /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/    
# initializes a borg repository      
borg --verbose init --encryption=none 
# verify the creation was executed successfully                   
cat README

 

Now the actual backup of the whole system:

sudo borg --progress --verbose create --comment "Use as root. My first system backup." -e /dev -e /run -e /tmp -e /sys -e /proc -e /mnt -e /media -s /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/::firstsystembackup /
# This creates a backup of the whole system, excluding locations
# that are re-generated at every boot up, as well as, the `/mnt` location
# as we don't want to back up our backups, especially when you
# have a backup drive containing several hundred GB of data.

Note that it is important to not mix up owners when backing up and later restoring backups. If you are unsure that you completely own a location, you should execute the backup as root. When doing a whole system backup, of course you have no choice but to execute the backup as root.

 

To make it even easier, I created a shell snippet containing a function that easily backs up your whole system with a very simple command:

borg_full(){
borg_comment=$1;
repo=full-system;
sudo borg --progress --verbose \
create --comment "Use as root. $borg_comment" \
-e /dev -e /run -e /tmp -e /sys -e /proc -e /mnt -e /media -e /var/log -s \
/mnt/bg/$repo::\{hostname\}+\{user\}:\{now:%Y-%m-%dT%H:%M\} /;
}

You may put this function into /etc/bash.bahrc or your ~/.bashrc.


Then, after executing source /etc/bash.bashrc ; source ~/.bashrcuse it as follows:

borg_full "This is my comment."

 

That will back up your whole OS (except /mnt) to /mnt/bg/full-system with the following archive name:

yourhostname+root:2019-04-01T04:52

 

You can of course change parts of the function, to customize it. E.g. replacing the repo variable in line 3 with full-OS will save your backup under the folder /mnt/bg/full-OS instead of /mnt/bg/full-system.

 

Create regular automated backups

Now taking the knowledge from the previous paragraphs, you can fill out the variables in this script to fit your personal settings.
Then you need to save this to a file on your disk. For example as /etc/auto-borg.sh.

#!/bin/bash
#########################################################################
# Copyright (C) 2020 Akito <the@akito.ooo>                              #
#                                                                       #
# This program is free software: you can redistribute it and/or modify  #
# it under the terms of the GNU General Public License as published by  #
# the Free Software Foundation, either version 3 of the License, or     #
# (at your option) any later version.                                   #
#                                                                       #
# This program is distributed in the hope that it will be useful,       #
# but WITHOUT ANY WARRANTY; without even the implied warranty of        #
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the          #
# GNU General Public License for more details.                          #
#                                                                       #
# You should have received a copy of the GNU General Public License     #
# along with this program.  If not, see <http://www.gnu.org/licenses/>. #
#########################################################################
# https://borgbackup.readthedocs.io/en/stable/deployment/automated-local.html
# 0.2.0
PURPOSE="SYSTEM"
TARGET_HOST="ssh://borg@borgserver:22"
LOCATION="/path/to/borg/repos"
BORG_REPO="borg-repo"
TARGET="${TARGET_HOST}${LOCATION}${BORG_REPO}"
ARC_NAME="${HOSTNAME}+${USER}:$(date +"%Y%m%dT%H%M%S")"
BORG_COMMENT="Automated \"${PURPOSE}\" backup to repository \"${BORG_REPO}\" at \"${TARGET_HOST}${LOCATION}\" issued $(date +"%Y-%m-%dT%H:%M:%S")."
BORG_OPTS="--compression auto,lzma,9 --checkpoint-interval 900 --warning"
LOGFILE_LOCATION="/tmp"
LOGFILE="borg-${PURPOSE}_$(date +"%Y%m%dT%H%M%S").log"
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
borg create ${BORG_OPTS} \
  --comment "${BORG_COMMENT}" \
  --exclude /media   \
  --exclude /dev     \
  --exclude /run     \
  --exclude /tmp     \
  --exclude /sys     \
  --exclude /proc    \
  --exclude /mnt     \
  --exclude /var/log \
  "${TARGET}::${ARC_NAME}" \
  / \
  2>> "${LOGFILE_LOCATION}/${LOGFILE}"

sync

After saving this file and making it executable, i.e. chmod +x /etc/auto-borg.sh, you can run it regularly and automatically by using cronjobs.

Restore a broken Raspberry Pi OS

At some point after creating all these backups, you maybe will face a problem with your system, leading you to use one of the backups made.

In this example, I will show how to restore an unbootable Raspberry Pi related OS on an SD card.

1. Find the right backup to restore

On the Borg server (where all backups are saved) go to the location of your borg repositories.
Then find the latest backup for the Raspberry Pi you have to restore. (It is expected that you used the recommended archive name schema from before, that contains the hostname as a prefix for the archive name.)

borg info borg-repo/ --prefix hostname-of-broken-raspberrypi --last 1

This will show you the last backup archive made by the broken Raspberry Pi.

2. Retrieve the SD card from the to be restored Raspberry Pi

  1. Unplug/turn off the Raspberry Pi that needs to be restored.
  2. Remove the SD card.
  3. Slide the SD card into an SD card reader.
  4. Plug the reader into the Borg server.
  5. Mount the SD card onto a folder.

Now the broken OS is attached to the Borg server, so we can start the restoration process.

3. Restore the broken OS to a working state

Change to the directory containing the mounted SD card's content. Then execute the following:

borg extract --verbose --progress /path/to/borg/repos/borg-repo::archivename

Once this process is done, the restoration is finished.

4. Start the restored Raspberry Pi.

  1. Reverse the actions from point 2.
  2. Start the Raspberry Pi again.

Now this Raspberry Pi should work again, as it did before an incident requiring you to restore the system.

 

Sources