Use Btrfs Send and Receive to Create a Secure Remote Backup Facility

Introduction

Having a backup is essential to avoid losing data. Using Btrfs and its send and receive features, we can configure secure sender and receiver systems to securely and efficiently remote backup a subvolume over SSH. Then, set up Systemd timers to perform regular, timely backups.

Note that only the changes made on the sender system are copied to the receiver system, reducing the backup facility's overhead. Since the systems use Btrfs snapshots, the disk usage for incremental backups is restricted only to the size of the changes.

Objectives

In this tutorial, you will learn how to:

Configure SSH to facilitate secure network-based backup
Create a Btrfs receiver script on the receiver system
Create a Btrfs snapshot on the sender system
Send the Btrfs snapshot to the receiver system
Create an incremental backup snapshot
Create an incremental backup script
Configure a Systemd service and timer unit for regular incremental backups

Prerequisites

Minimum of two Oracle Linux systems running the UEK kernel
Each system should have Oracle Linux installed and configured with:
- A non-root user account with sudo access
- A block device attached to each system and formatted with Btrfs
- Key-based SSH, also known as password-less SSH, between the hosts

Deploy Oracle Linux

Note: If running in your own tenancy, read the linux-virt-labs GitHub project README.md and complete the prerequisites before deploying the lab environment.

Open a terminal on the Luna Desktop.

Clone the linux-virt-labs GitHub project.

git clone https://github.com/oracle-devrel/linux-virt-labs.git

Change into the working directory.
```
cd linux-virt-labs/ol
```

Install the required collections.

ansible-galaxy collection install -r requirements.yml

Update the Oracle Linux instance configuration.

cat << EOF | tee instances.yml > /dev/null
compute_instances:
  1:
    instance_name: "ol-sender"
    type: "server"
  2:
    instance_name: "ol-receiver"
    type: "server"
add_block_storage: true
block_count: 1
passwordless_ssh: true
EOF

Deploy the lab environment.
```
ansible-playbook create_instance.yml -e localhost_python_interpreter="/usr/bin/python3.6" -e "@instances.yml" 
```
The free lab environment requires the extra variable local_python_interpreter, which sets ansible_python_interpreter for plays running on localhost. This variable is needed because the environment installs the RPM package for the Oracle Cloud Infrastructure SDK for Python, located under the python3.6 modules.
The default deployment shape uses the AMD CPU and Oracle Linux 8. To use an Intel CPU or Oracle Linux 9, add -e instance_shape="VM.Standard3.Flex" or -e os_version="9" to the deployment command.
Important: Wait for the playbook to run successfully and reach the pause task. At this stage of the playbook, the installation of Oracle Linux is complete, and the instances are ready. Take note of the previous play, which prints the public and private IP addresses of the nodes it deploys and any other deployment information needed while running the lab.

Prepare the Block Volumes

Open a terminal and connect via SSH to the ol-sender instance.
```
ssh oracle@<ip_address_of_instance>
```

Format the block volume with a Btrfs file system and then mount it.

sudo mkfs.btrfs /dev/sdb
sudo mkdir /source
sudo mount /dev/sdb /source

Repeat on the ol-receiver instance.

ssh ol-receiver \
  "sudo mkfs.btrfs /dev/sdb; \
  sudo mkdir /backup; \
  sudo mount /dev/sdb /backup"

Set Up a Dedicated User on the Receiver System

By creating a dedicated user to handle the btrfs receive requests, you can limit the likelihood that the backup process can be used as an attack vector to compromise systems.

Create a new user.
```
ssh ol-receiver "sudo adduser btrfsrcv"
```

Create a Btrfs Receiver Script on the Receiver System

Create the btrfs_receive.sh backup script on the receiver system in the btrfsrcv user's home directory.

ssh ol-receiver sudo tee /home/btrfsrcv/btrfs_receive.sh > /dev/null << EOF
#!/bin/bash
sudo /sbin/btrfs receive /backup
EOF

Set the ownership and mode of the script so that the btrfsrcv user can run it.

ssh ol-receiver \
  "sudo chown btrfsrcv /home/btrfsrcv/btrfs_receive.sh; \
  sudo chmod +x /home/btrfsrcv/btrfs_receive.sh"

Create a sudoer configuration drop-in so the btrfsrcv user can run the /sbin/btrfs receive command on the /backup directory.

ssh ol-receiver sudo tee /etc/sudoers.d/200-btrfsrcv > /dev/null << EOF
btrfsrcv ALL = (root) NOPASSWD: /sbin/btrfs
EOF

Configure SSH to Facilitate Secure Network-Based Backup

Create an SSH key pair on the sender system that you can restrict for backup purposes.
```
sudo ssh-keygen -t rsa -b 4096 -f /root/.ssh/btrfs_backup -N ""
```
The -N "" skips the request for a passphrase for this key. Restrictions on using this key are configured later in this tutorial.
Review the public key and assign its contents to a variable.
We'll use the variable to copy the public key to the configuration for the btrfsrcv user on the receiver system.
```
PUBLIC_KEY=$(sudo cat /root/.ssh/btrfs_backup.pub)
```
Verify the variable contains the public key.
```
echo $PUBLIC_KEY
```
Add the public key to the authorized_keys file for the btrfsrcv user on the receiver system.
Ensure you also create the .ssh directory if it doesn't already exist.
```
ssh ol-receiver \
  "sudo mkdir /home/btrfsrcv/.ssh; \
  sudo chown btrfsrcv:btrfsrcv /home/btrfsrcv/.ssh; \
  sudo chmod 700 /home/btrfsrcv/.ssh"
```
Add restrictions for the key to limit its usage.
Edit the authorized_keys file on the receiver system to restrict the backup SSH key to a specific command.
```
ssh ol-receiver sudo tee -a /home/btrfsrcv/.ssh/authorized_keys > /dev/null << EOF
command="/home/btrfsrcv/btrfs_receive.sh",from="$(hostname -i)" $PUBLIC_KEY
EOF
```
The format consists of:
command="<script>",from="<sender_host>" <public_key>
Substitute the following values:
- <script>: the full path and name of the script to which you want to grant access
- <sender_host>: the IP address or hostname of the sender system where we created the private key
- <public_key>: the value of the public key that you created on the sender system
For this tutorial, edit /home/btrfsrcv/.ssh/authorized_keys and add the entry.

Set the correct ownership and permissions on the authorized_keys file.

ssh ol-receiver \
  "sudo chown btrfsrcv:btrfsrcv /home/btrfsrcv/.ssh/authorized_keys; \
  sudo chmod 600 /home/btrfsrcv/.ssh/authorized_keys"

Test Sending a Btrfs Snapshot to the Receiver System

Touch a file in the Btrfs file system on the sender system.
```
sudo touch /source/testfile
```
Make a directory to store the local snapshots.
```
sudo mkdir /source/.snapshots
```

Create a Btrfs snapshot of the source file system.

sudo btrfs subvolume snapshot -r /source /source/.snapshots/test1

Send the snapshot to the receiver system.
```
sudo btrfs send /source/.snapshots/test1 | sudo ssh -T -i /root/.ssh/btrfs_backup btrfsrcv@ol-receiver
```
The -T in the SSH command avoids the Pseudo-terminal will not be allocated warning message.
Validate that the snapshot gets copied to the receiver system and that the testfile is present.
```
ssh ol-receiver sudo ls /backup/test1
```

Create an Incremental Backup Snapshot

Create an incremental backup snapshot so that you can see the replicated changes.

Add some content to the Btrfs source directory on the sender system.

sudo touch /source/testfile2
echo "updated text in testfile"|sudo tee -a /source/testfile

Create a new backup snapshot to capture the changes.

sudo btrfs subvolume snapshot -r /source /source/.snapshots/test2

Send the updated snapshot to the receiver system.

sudo btrfs send /source/.snapshots/test2 | sudo ssh -T -i /root/.ssh/btrfs_backup btrfsrcv@ol-receiver

Validate that the snapshot gets copied.
Confirm that testfile2 is present and that the new content exists within the testfile.
```
ssh ol-receiver \
  "sudo ls /backup/test2/testfile2
  sudo cat /backup/test2/testfile"
```

Create an Incremental Backup Script

Create a script directory.
```
sudo mkdir /scripts
```

Create a backup script.

cat << 'EOF' | sudo tee /scripts/btrfs-backup.sh > /dev/null
#!/bin/bash

# Variables
SOURCE="/source"
SNAPSHOT_DIR="/source/.snapshots"
LATEST_SNAPSHOT=$(ls -1 ${SNAPSHOT_DIR} | tail -n 1)
NEW_SNAPSHOT="${SNAPSHOT_DIR}/$(date +'%Y%m%d%H%M%S')"
RECEIVER="btrfsrcv@ol-receiver"

# Create a new snapshot
sudo btrfs subvolume snapshot -r ${SOURCE} ${NEW_SNAPSHOT}

# Send the new snapshot
if [ -z "${LATEST_SNAPSHOT}" ]; then
  sudo btrfs send ${NEW_SNAPSHOT} | sudo ssh -T -i /root/.ssh/btrfs_backup ${RECEIVER}
else
  sudo btrfs send -p ${SNAPSHOT_DIR}/${LATEST_SNAPSHOT} ${NEW_SNAPSHOT} | sudo ssh -T -i /root/.ssh/btrfs_backup ${RECEIVER}
fi
EOF

Note: If running outside of the free lab environment, set the RECEIVER variable in the script to btrfsrcv@ and the IP address or hostname of the receiver system.

Change the mode on the backup script so that it can run.
```
sudo chmod +x /scripts/btrfs-backup.sh
```
Test the script by creating additional content in the Btrfs source directory.
```
sudo touch testfile3
```
Run the backup script.
```
sudo /scripts/btrfs-backup.sh
```
Check the backup directory on the receiver system to see if it contains the new snapshot.
```
ssh ol-receiver sudo ls /backup
```
Note that the backup script uses the date and time to name the snapshot directory.

Configure a Systemd Service and Timer Unit

You can configure a systemd service and timer unit on the sender system to perform regular incremental backups. This example creates a systemd unit to run the backup script daily.

Create a systemd service unit file for the backup script.

cat <<EOF | sudo tee /etc/systemd/system/btrfs_backup.service > /dev/null
[Unit]
Description=Btrfs Backup Service

[Service]
Type=oneshot
ExecStart=/bin/bash /scripts/btrfs-backup.sh
EOF

Create a systemd timer unit file to schedule the backup service.

cat <<EOF | sudo tee /etc/systemd/system/btrfs_backup.timer > /dev/null
[Unit]
Description=Run Btrfs Backup Service Daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable and start the systemd timer to schedule the backups.
```
sudo systemctl enable --now btrfs_backup.timer
```
Verify that the systemd.timer unit is running correctly.
```
sudo systemctl status btrfs_backup.timer
```

Test the systemd service.

sudo systemctl start btrfs_backup.service

Check the backup directory on the receiver system to see if it contains the latest snapshot.
```
ssh ol-receiver sudo ls /backup
```

Next Steps

This tutorial taught you how to securely back up your Btrfs file system over the network using SSH. You can do many things with Btrfs and SSH, so check out the Related Links section for more details and training for Oracle Linux.