Rescaling Incus ZFS Storage on Hetzner Cloud

In a previous article about setting up Incus on Hetzner Cloud, we showed how to configure a CAX11 server with a separate ZFS partition for optimal container management. This tutorial builds upon that setup and demonstrates how to safely upgrade the server from a CAX11 to a CAX21 instance, focusing on expanding the ZFS storage partition from 40GB to 80GB. While CPU and RAM upgrades (from 2 to 4 cores and 4GB to 8GB respectively) are relatively straightforward, storage expansion requires careful attention to avoid data loss.

Prerequisites

• A Hetzner Cloud server running Incus with ZFS storage
• Root access to the server
• Basic understanding of disk partitioning and ZFS concepts
• Backup of all important data

⚠️ Important Safety Warnings

1. Always backup your data before attempting any partition modifications
2. Double-check all commands before execution
3. Verify disk and partition numbers to avoid data loss
4. Have a recovery plan in case something goes wrong

Step-by-Step Guide

1. Prepare ZFS Pool for Expansion

First, enable auto-expansion for your ZFS pool:

# Enable auto-expand
zpool set autoexpand=on incus-zfs

# Verify the setting
zpool get autoexpand

2. Upgrade the Server in Hetzner Cloud Console

1. Stop the server
2. Create a snapshot (for recovery if needed)
3. Navigate to the rescale tab
4. Important: Uncheck “CPU and RAM only” to include disk space increase
5. Apply the upgrade (40GB to 80GB)
6. Wait for the server to restart automatically

3. Resize the ZFS Partition

After the server restarts, use parted to adjust the partition:

parted

You’ll see a welcome message:

GNU Parted 3.5
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.

When you enter print free, you’ll encounter this important warning:

Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 80003072
blocks) or continue with the current setting?
Fix/Ignore? 

Important: You should type F to fix the GPT table. This is a crucial step that ensures the partition table correctly recognizes all available space on the expanded disk. Without this fix, you might not be able to use the full expanded capacity.

After fixing the GPT table, you’ll see the current partition layout:

Number  Start   End     Size    File system  Name  Flags
14      1049kB  2097kB  1049kB                     bios_grub
15      2097kB  258MB   256MB   fat16              boot, esp
 1      258MB   10.0GB  9742MB  ext4
 2      10.0GB  41.0GB  31.0GB  zfs
        41.0GB  81.9GB  41.0GB  Free Space

Now proceed with resizing partition 2 (ZFS partition) to use all available space:

(parted) resizepart 2 100%
(parted) print
(parted) q

4. Expand the ZFS Pool

First, verify your ZFS pool status:

zpool status

Identify your drive path:

ls -al /dev/disk/by-id/

Check current pool size:

zpool list

Expand the pool:

zpool online -e incus-zfs scsi-0QEMU_QEMU_HARDDISK_53309279-part2

Verify the new size:

zpool list
df -h

5. Final Steps

Disable auto-expand (recommended for security):

zpool set autoexpand=off incus-zfs

# Verify the setting
zpool get autoexpand

Verification

After completing all steps, your ZFS pool should show the increased size (about 66.5GB usable from the 80GB raw disk space). You can verify this with:

df -h

Look for the Incus containers mount point to confirm the new size:

incus-zfs/containers/ct1   65G  204M   65G   1% /var/lib/incus/storage-pools/default/containers/ct1

Troubleshooting

• If the partition resize fails, restore from the snapshot created in step 2
• If ZFS pool doesn’t show increased size, verify autoexpand setting
• If you see the GPT warning and selected ‘Ignore’ instead of ‘Fix’, you may need to start over with the partitioning process
• If errors occur during any step, stop and seek assistance before proceeding

Additional Notes

• The procedure can be adapted for different server types and sizes
• Always monitor system logs during the process
• Consider performing this during maintenance windows
• Test the procedure on a non-production system first if possible

Remember that while this guide provides a structured approach, each system might have unique considerations. Always understand the commands before executing them and have backups ready.