Ramblings of Narc

For the longest time, I’ve wanted to have a backup of Eris, complete duplicate of all the partitions and their data on /dev/hda (the primary master hard-drive; also the one the OS runs on).

This isn’t as straightforward as one may think, though. The *NIX program, dd, can make exact duplicates of an entire hard-drive (or other device, including a simple file, passed in to the if= parameter) by sequentially reading all its sectors and writing them to the output file (which, in turn, can be another hard-drive, or a file, or even a tape drive). This is absolutely wonderful, and the perfect copy that I’d want… except it doesn’t really work.

See, the first thought I had was to dd from the hard-drive partition to a file on an SMB-mounted partition on Hermes. So I tried it. First partition was the boot partition, which is approximately 64 MB in size. That went through like a charm. So I tried the main partition after that, and… well, it didn’t quite work. Apparently, SMB has a problem with files greater than 2 GB, and the main partition is 3.something GB in size, and so… it didn’t quite work. I toyed with the idea of splitting it into 2 GB files, but I didn’t know how to do that with dd and didn’t want to learn the innards of this particular black box.

The next thought was to try NFS, but setting up an NFS share on Windows was another black box I didn’t want to mess with. I also had no reason to believe the 2 GB file size limit wouldn’t still be there, so I dropped the whole idea of backing up altogether. After all, it wouldn’t matter that much if I had to take a long time setting up Eris again, since Eris didn’t do all that much that was important.

Since then, however, I’ve been giving Eris more and more responsibilities, and so it was time to re-examine the backup idea.

The current plan I’m trying out right now also uses dd, but also ssh and… well, let me go over it in detail.

What I started with:

• one non-privileged (sort of) user account with an SSH public/private key combo in its home dir. This makes it possible to ssh into any of the SSH-enabled computers in the house without a password, including (especially) Hermes.
• root. You need root if you want to read from /dev/whatever, unless you had root change the permissions on the whatever to let you access it.

The process

• First, I tested ssh (the client)’s ability to open a shell to another computer and run a random command on it. So I ran: ssh [my user]@hermes "ls -l". This worked as expected.
• Then, I tried transferring a file from one computer to the other using ssh. The idea, in this case, was to have a process on Eris that would read the file to standard output, and a process on Hermes that would read its standard input and write it to a file. In-between would lie the ssh connection, taking its input from the reader’s output, and passing that along as the input to the writer on Hermes. The command: cat file_on_eris.txt | ssh [my user]@hermes "cat - > file_on_hermes.txt". To cat, – is shorthand for standard input, and I’m using the shell’s redirects to chain things together. cat on hermes is just a pass-through for standard in, which in this case is the output from cat on eris, which is reading the file. This worked. So far, so good.
• But I also had the root/non-root barrier to break, which is easier from the root side than the non-privileged user side. Still, I could’ve done it any way I wanted, using sudo, for example, to execute the dd that would read the hard-drive. But I didn’t like that idea. Instead, I decided on a named pipe, or FIFO, if you prefer. So, as the non-privileged user, I executed mkfifo temp-pipe in my home directory. This created a FIFO that I could attach a non-privileged reader to, and that could take the privileged writer’s input.
• The next step: putting things together. Part one, the FIFO reader. I did cat temp-pipe | ssh [my user]@hermes "cat - > disk-image.img". SSH launched, connected, then waited for data. If my SSH had required a password, this is where I would’ve entered it.
• Part two: as root, I went to the non-privileged user’s home directory, for brevity, and then did dd if=/dev/hda3 > temp-pipe, which hooked up the writer end of the pipe, and data started flowing.

So, to summarize, as the non-privileged user:

cd ~
mkfifo temp-pipe
cat temp-pipe | ssh user@remote "cat - > disk-image.img"

And, as root:

cd ~non-privileged-user
dd if="/dev/hda3 > temp-pipe

This is using my already-existing infrastructure, defined under the “What I started with” header, so it’s the best solution for me. It might not be the best for you. But I like it.

One further improvement: Using gzip to compress the disk image. The last non-privileged user command then becomes cat temp-pipe | ssh user@remote "gzip -c > disk-image.img" if compression is going to be done on the remote side, or cat temp-pipe | gzip -c | ssh user@remote "cat - > disk-image.img for compression on the local side.

As you can see, it’s relatively easy to improve on the backup. The next part will probably be to automate everything, and have it run weekly. If I feel like it.

Oh, I’m doing compression on the Hermes side, because Eris is kinda weak on the horsepower front. However, if I were aiming to use minimal bandwidth, the compression would be happening before the pipe to ssh. This might be a better idea for future backups.

Very much worth the read:

It is a cliche in our business that the first 90 percent of the work is easy, the second 90 percent wears you down, and the last 90 percent – the attention to detail – makes a good product.

— The Graphing Calculator Story

So for a random post at a random time, I figured I should probably do a little write-up of the set-up I’ve got here. Perhaps I should’ve called this “Everything you never wanted to know about Narc, but were afraid he’d go ahead and say it anyway”.

The first and foremost (Internet-wise) computer in this house is Starfleet, a decent little computer (2GHz P4, 512MB RAM) running Windows 2003 SP1 with all the latest updates (I think), and port forwarding some stuff to the various other computers involved. It’s also running DNS and NATting for the computers behind it. It’s also sometimes used as a workstation by my mum — nothing more complex than Firefox and Thunderbird, but that’s really enough for her immediate needs. Starfleet was named a long time ago and I’m loathe to re-rename it yet again (it used to be called Oracle before that). One peculiarity of Starfleet’s is that it doesn’t have a C: drive — a random repair install of a fuxored version of a previous install changed all the drive letters, and you can’t change drive letters on the fly (not even in safe mode). So we had to do a registry search & replace, and now everything *seems* to be working, but yeah, no C: drive. I find that amusing, to be perfectly honest. For a future project, I should try running a Windows install off the Z: drive, just to see if it works.

Second, we have my Dad’s computer, which is more or less an unknown value as far as I’m concerned. It’s Intel on the inside (HyperThreading P4 — it’s from before the dual core craze — I think), and that’s about all I know. It’s the only computer in the house with a TV capture video card (Radeon All-In-Wonder of some kind), so we’ve used it in the past to make backup copies of various broadcast TV shows (the ultimate in time-shifting; who needs TiVo?).

Thirdly, we have my computers… and I’m not sure which one to start with. I know, I’ll work backwards from the youngest (factory-wise), that’ll be good.

- Sisiphus is my number one workstation, and the only one of four computers with a working soundcard and things attached to it. It’s an Athlon 64 X2 (Dual Core) 3800+, and runs at a real clock speed of 2000 MHz. No, 2000.1. No, 1999.9. Make up your mind, you! It’s also got 2 GB of RAM, and a nice shiny SATA 2 160 GB hard-drive, giving it the fastest throughput in the house (more or less). If it weren’t illegal, I could run Mac OS X on it. As it is, it runs all my games, and multi-tasks like a pro (I don’t have to kill Firefox to play a game anymore, OMG!), so for all intents and purposes, this is the best computer I’ve ever had.

- Hermes is my previous number one workstation + server, now relegated to, uh… number two workstation and server. It doesn’t have sound anymore (boo-hoo…) because its on-board sound card fried (and its floppy controller is fried, too — that’s a semi-fried motherboard, what?), but still runs my Apache + PHP + MySQL + Cygwin + Mail server + all the other crap I run on it, and it’s hooked up to the other one using Synergy (if you have more than one computer side by side and don’t want to invest in a KVM switch (and hate VNC), I can’t recommend Synergy enough), so I can still feel like I did when I had two monitors on the same computer, *and* actually have one computer per each monitor. All in all, a good investment that keeps paying for itself over and over and over again.

- Bast (fka Sirimi), an old Dell Optiplex which is really probably not that anymore internally, running a shiny copy of LFS, but temporarily out of comission for the dual reason that I don’t have enough power outlets in this room, and I don’t have an Ethernet cable for it (and can’t make one because I have no crimping tool. And haven’t bought a crimping tool because I’m a lazy git). When I bring Bast up (and tell it what its new name is), I’ll migrate the Web server there so I can stick the whole dang thing in a corner and forget about it. Similarly, I’ll see if I can migrate my current mail server to a Postfix running on same, because that Optiplex is so much quieter than Hermes with its million and one fans.

- And, finally, I have Eris, an amazing little beast that’ll give any modern computer a run for its money (as long as you’re patient). Eris is a 100 MHz 486 (I’ve seen Pentiums at less than that!) with 32 MB of RAM, and for most of its lifetime, I’ve relegated it to very secondary status. It runs a cross-compiled version of LFS (because 32 MB of RAM isn’t enough to compile some of the larger dependencies… like glibc or gcc, I don’t remember which one failed), with an Apache server (no PHP, just CGI), and now SVN via SVNServe (internal only!). Eris also serves as my SSH gateway, letting me reach home from anywhere on the Internet as long as I have my public key with me to authenticate with (quick lesson in public/private key encryption: public key encoded messages can only be decrypted with the private key, and vice-versa; server doesn’t even see a password, it just sees you’re capable of authoring messages that can be decoded via private key). It also automatically starts screen, possibly the most useful piece of Linux text-mode software aside from the shell.

All in all, a heck of a stable. I’m quite proud of it, myself.

Once upon a time, I was surfing the Web more or less randomly, when I came across the concept of a “quine” (probably on this page). I had a look and was suitably impressed — the idea of a program that can print its own source code seems very simple until you try to do it for real, at which point, you quickly abandon all hope of ever really doing it without cheating.

What this misses out on is that most programs like this involve some type of cheating — exactly how, though, is up to the author. The most obvious cheat is to read in the file containing your own source code and print that — but that’s too easy. Not to mention, hey — maybe there is no such file. Then what do you do?

In any case, I abandoned the thought and went back to my normal world of programs that don’t really know themselves… until today. Today I came across this self-recursive story (via archive.org — original website no longer exists) (and I encourage you to read it if you have a strange sense of humor like mine). Following links around that site, I eventually found this reference to “selfish code”, which links back to that quines page I mentioned first.

This time, armed with a bit more patience and a crapload more experience, I was able to decypher the code so much easier, and I could see the little “cheats”, and why they worked. But among all the programs on that page, there was one I didn’t particularly like. It was this one:

$b='$b=%c%s%c;printf$b,39,$b,39;';printf$b,39,$b,39;

It looks nice, and does print itself, but… it doesn’t have the #!/usr/bin/perl at the start, so if you want to execute it, you need to do perl -c source-file or perl -e "[code]". So I figured I could extend it to include the hash-bang, couldn't I?

...

Well, after a number of iterations, I eventually came up with this:

#!/usr/bin/perl

$a="#!/usr/bin/perl";
$b='%c%c$a="%s";%c$b=%c%s%c;%cprintf$a;printf$b,10,10,$a,10,39,$b,39,10,10,10;%c%c';
printf$a;printf$b,10,10,$a,10,39,$b,39,10,10,10;

It successfully reproduces itself, and uses quite a little bit of magic to achieve that.

What's the cheat? We're printing the \n characters using sprintf('%c',10), and the single quotes in a similar way, but with the character code 39. This is going to work in any simple string encoding, up to and including UTF-8 (but without a BOM). But don't try to format it for UTF-16 or anything exotic (like EBCDIC, or some such nuttery). It probably won't work there. The assumption is that character 10 (0x0A, if you prefer) is a \n character, and that 39 (or 0x27) is a single quote character.

And speaking of quote characters -- when you copy/paste from here, watch for any character encoding issues: this page is served as UTF-8, and auto-replaces quotes with "smart quotes", which you'll have to replace with simple quotes if you want to use the code.

Update: If you have a bash shell handy, this is also a quine:

bash: bash:: command not found

Proof:

[narc@eris ~]$ bash: bash:: command not found
bash: bash:: command not found
[narc@eris ~]$

Update 2007-12-10: This doesn't really belong in Projects...

So it seems Mihai, one of my co-workers, has a new blog. I beat you to it by three weeks, dude! *g*

In other news, very soon now I’ll move this blog to http://wp.narc.ro/, so be careful of borken trackbacks.

Update 2007-11-21 18:01:00: It’s up, it’s up!

SVN is cute. But you should include the projects touched by each commit when you make it.

In fact, the template should look like this:

(CommaSeparated,ListOf,Projects) Summary: This is a short summary of
what this revision changes (for the better, one would hope).

- This is one item of the detailed changes.
- This is another item. The change detailed here includes an API
change, so we mark it. [CA]
- This is just another sample item. If it included an API change, we'd
use CA to mark it, too.

Deps: rev 77 SomeProject, rev 63 SomeOtherProject

Note that Deps can be omitted altogether, in which case it’s assumed the revision doesn’t depend on any specific revision of another project.

I might remove the whole Deps thing altogether, it seems a bit useless, IMO.

Update: The text “Summary:” can also be omitted, as it is expected the first line will be a comma-separated list of projects in parens, and a summary line.

…I’m compiling Subversion. On a 486 (eris).

Then I’ll need some hard-drive space dedicated to the SVN repository and its soon-to-be-myriad of versions, micro-versions, revisions, and such-like, that make the whole shebang work properly. Of course, some parts don’t really belong in the repository, either because they’re too big, or because they’re not really part of the stuff I wrote. And then there’s other parts that aren’t publicly accessible, but need to go into version control anyway.

In total, all the stuff that makes up narc.ro and subdomains is just shy of 2.3 GB. In there, we have duplicates (remaining from various moves like http://www.narc.ro/man/ -> http://man.narc.ro/ ), and we have just short of 350 MB of images (some of them enormously high-quality), and some 1 GB or so of mp3s (sorry, that part is private), not to mention another 225 MB of “temporary” files, some of which are temporary scripts built to play with some random feature or another. The actually important, personally-written, valuable parts probably clock in just over 20 MB or so. Talk about dense packaging — that’s probably three or four years of development, all fitting in so little space it seems almost pitiful.

Nevertheless, that doesn’t include the millions of previous versions of all the files involved, and with that, the repository would probably reach well over 200 MB, and that’s not counting several small, but important backups of the various databases I’ve always cared about.

Think of it like this — about 90% of the code I’ve written has been lost, because there was no reason, at the time, to save it. This code isn’t that valuable in and of itself — after all, most of the various old versions would’ve just been bug-ridden crap that got superseded by revisions of the same files that didn’t have those bugs. Or else we could be talking about versions of a library that didn’t include some particular function, and when a new page was added suddenly that function was required, and added. So it’s not that big a loss.

However, you get to a point where you look at what you have, and you say “You know, this stuff is barely enough to fill 13 floppy disks, have I really done so little? The old versions, the bug fixes, the little “HACK HACK HACK” comments that got replaced by real code, or the functions that were perfectly good but had to be rewritten to provide a more uniform “look”, or to use a particular new helper function that made the work of debugging a lot easier — all that stuff is impossible to see. So, for all the work I did creating these 20 MB, all I have to show for it is the latest version, with no real sense of how much has gone before.

Which is why I’m changing things now. With subversion now installed (make install finished just as I was typing this), it’s time to give Eris an extra hard-drive, and put all the stuff I care about into SVN. And then run commits every once in a while.

*whistles* I’m on my way… to structured programming…

Found in the CSS3 spec in the section about HSL color, an algorithm for converting HSL to RGB.

See DelayedIndexing on the C2 wiki. This would be very good to implement in CMWiki (or NeoWiki), together with an email to me whenever anyone other than myself (or another trusted user) edits it.

Edit: Also see linked pages of same.

I need to put Sirimi up (and rename it to Bast, or maybe Bastet), update all the important software on it (Apache, PHP, MySQL), dump all the databases I have here, and copy everything over, then change the port forwards for the HTTP and HTTPS stuff to point that way. That way, Hermes can have fewer things to do, I get a more native Perl, and once I figure out how to make Postfix do everything my existing mail server is doing now, I can move the mail server there, and keep Hermes as nothing more than the secondary workstation.

The point of this whole thing is to have a quieter sleeping experience: Hermes really isn’t the quietest computer to sleep around. I’m still considering getting pissed off and installing a Windows on the-computer-soon-to-be-called-Bast, so that my mail server will continue to work the same on it. But I’d honestly prefer Postfix, since shareware mail server is not the happiest thing in the world. Almost all my applications right now are free — the fewer non-free ones I have to deal with, the better.

Is there a good, syntax-highlighting, CVS-enabled, Perl and PHP editor for Linux (not counting Eclipse)? I might switch Hermes to Linux full-time, were such a thing to exist. Actually, considering I’ve always been a vim user, perhaps I’ll use that, together with a decent VPN client (it needs to support extssh, or ssh) might work better.

Hm… so let’s recap, to switch Hermes to Linux I’d need:

• to move the HTTP, HTTPS, and SMTP/POP3 servers to Sirimi (soon to be called Bast)
• to pick and install a Linux distribution. Though I may love LFS, the thought of compiling and installing X.org on it is daunting. I wonder if one of the BSDs might not be a better choice, actually? I’ll need some opinions.
• to find equivalent software to provide the following:
  • two-pane file-system browsing (Midnight Commander would do this in text mode, but for graphics mode? I saw something that looked good a few days ago, but I’ve lost the link)
  • some torrent client (whatever. Do some market research. At worst, keep using uTorrent on Sisiphus)
  • Maybe a decent GUI editor, though vim is fine for most of the stuff I do, especially if it has syntax highlighting enabled.
• Also, I’ll need the following software installed at first:
  • Synergy (for Linux)
  • Firefox (IceWeasel?) for Linux
  • Apache, PHP, MySQL (for testing stuff before putting it up)

Another one for the notebook: a wiki. This will need to be the prototype for the more centralized CM-application stuff (too many of the NeoFW applications are using copies (!) of certain programming practices that would be better handled as a single unit (see especially index.php in the apps Blog3, NeoFW, jgh, pt), and parts that are very similar, but with different values (see inc/conf.inc in the same apps)).

The CMWiki should unify the approaches in such a way that the other applications can be relatively easily updated. This has been brought up before (see point 7 in the given link). I’m still not sure whether to use a class or group of functions, though I’m leaning more towards using classes, for better forward-compatibility.

Completely off-topic, I’m impressed by all the early PHP 4 (or maybe even PHP 3!) applications that still work, today, in PHP 5.2, with nothing more than a few configuration changes. That’s backwards-compatibility. Me, I got into this whole thing around PHP 4.3, so my applications don’t even require special configuration options — they just continue to work.

I had more to say, but it turned so much longer that I’m going to make it a separate post. Watch this space…

Update 2007-12-10: Updated link to point to the correct post.