Ramblings of Narc

For the longest time, I’ve wanted to have a backup of Eris, complete duplicate of all the partitions and their data on /dev/hda (the primary master hard-drive; also the one the OS runs on).

This isn’t as straightforward as one may think, though. The *NIX program, dd, can make exact duplicates of an entire hard-drive (or other device, including a simple file, passed in to the if= parameter) by sequentially reading all its sectors and writing them to the output file (which, in turn, can be another hard-drive, or a file, or even a tape drive). This is absolutely wonderful, and the perfect copy that I’d want… except it doesn’t really work.

See, the first thought I had was to dd from the hard-drive partition to a file on an SMB-mounted partition on Hermes. So I tried it. First partition was the boot partition, which is approximately 64 MB in size. That went through like a charm. So I tried the main partition after that, and… well, it didn’t quite work. Apparently, SMB has a problem with files greater than 2 GB, and the main partition is 3.something GB in size, and so… it didn’t quite work. I toyed with the idea of splitting it into 2 GB files, but I didn’t know how to do that with dd and didn’t want to learn the innards of this particular black box.

The next thought was to try NFS, but setting up an NFS share on Windows was another black box I didn’t want to mess with. I also had no reason to believe the 2 GB file size limit wouldn’t still be there, so I dropped the whole idea of backing up altogether. After all, it wouldn’t matter that much if I had to take a long time setting up Eris again, since Eris didn’t do all that much that was important.

Since then, however, I’ve been giving Eris more and more responsibilities, and so it was time to re-examine the backup idea.

The current plan I’m trying out right now also uses dd, but also ssh and… well, let me go over it in detail.

What I started with:

• one non-privileged (sort of) user account with an SSH public/private key combo in its home dir. This makes it possible to ssh into any of the SSH-enabled computers in the house without a password, including (especially) Hermes.
• root. You need root if you want to read from /dev/whatever, unless you had root change the permissions on the whatever to let you access it.

The process

• First, I tested ssh (the client)’s ability to open a shell to another computer and run a random command on it. So I ran: ssh [my user]@hermes "ls -l". This worked as expected.
• Then, I tried transferring a file from one computer to the other using ssh. The idea, in this case, was to have a process on Eris that would read the file to standard output, and a process on Hermes that would read its standard input and write it to a file. In-between would lie the ssh connection, taking its input from the reader’s output, and passing that along as the input to the writer on Hermes. The command: cat file_on_eris.txt | ssh [my user]@hermes "cat - > file_on_hermes.txt". To cat, – is shorthand for standard input, and I’m using the shell’s redirects to chain things together. cat on hermes is just a pass-through for standard in, which in this case is the output from cat on eris, which is reading the file. This worked. So far, so good.
• But I also had the root/non-root barrier to break, which is easier from the root side than the non-privileged user side. Still, I could’ve done it any way I wanted, using sudo, for example, to execute the dd that would read the hard-drive. But I didn’t like that idea. Instead, I decided on a named pipe, or FIFO, if you prefer. So, as the non-privileged user, I executed mkfifo temp-pipe in my home directory. This created a FIFO that I could attach a non-privileged reader to, and that could take the privileged writer’s input.
• The next step: putting things together. Part one, the FIFO reader. I did cat temp-pipe | ssh [my user]@hermes "cat - > disk-image.img". SSH launched, connected, then waited for data. If my SSH had required a password, this is where I would’ve entered it.
• Part two: as root, I went to the non-privileged user’s home directory, for brevity, and then did dd if=/dev/hda3 > temp-pipe, which hooked up the writer end of the pipe, and data started flowing.

So, to summarize, as the non-privileged user:

cd ~
mkfifo temp-pipe
cat temp-pipe | ssh user@remote "cat - > disk-image.img"

And, as root:

cd ~non-privileged-user
dd if="/dev/hda3 > temp-pipe

This is using my already-existing infrastructure, defined under the “What I started with” header, so it’s the best solution for me. It might not be the best for you. But I like it.

One further improvement: Using gzip to compress the disk image. The last non-privileged user command then becomes cat temp-pipe | ssh user@remote "gzip -c > disk-image.img" if compression is going to be done on the remote side, or cat temp-pipe | gzip -c | ssh user@remote "cat - > disk-image.img for compression on the local side.

As you can see, it’s relatively easy to improve on the backup. The next part will probably be to automate everything, and have it run weekly. If I feel like it.

Oh, I’m doing compression on the Hermes side, because Eris is kinda weak on the horsepower front. However, if I were aiming to use minimal bandwidth, the compression would be happening before the pipe to ssh. This might be a better idea for future backups.