ATTENTION ALL MURKNET USERS - READ THIS NOW!

[solarbird]

Our main mail/web server, lodestone, got rootkitted last night. Unless I've missed a security bulletin recently - which I could well have, given how things have been - it was probably a local exploit via a stolen user account. However, that's not guaranteed. (I have some suspicions about where to look; I saw a login yesterday that somewhat surprised me, but not enough for me to be alarmed. That was probably a mistake.) The rootkitting happened last night, mostly around midnight PDT.

The box has been fundamentally compromised; in addition to widespread webspace hacking, /bin and files in it have been changed. The box will need to be flattened and rebuilt. Unfortunately, this work cannot really begin until annathepiper gets back. We are also in the middle of trying to move, making things much more difficult.

Assume we will be down for AT LEAST four days, probably longer. During this time, our net connectivity will probably be intermittant. I may be able to get a skeleton box online to catch incoming mail before then, but no promises.

When lodestone comes back up, we will be requiring new passwords from ALL users. ALL executable files in userspace should be deleted, and either recompiled or reinstalled. These were not expert crackers; the work was sloppy; they left fingerprints everywhere, and some of their attempts to modify binaries resulted in nonexecutable files, rather than trojaned files, which is good. But that doesn't mean they were totally inept. And they were certainly trying to do as much damage and backdooring as possible, so we need to react as if they were better than they are.

If there are any former Murknet admins who think they can help/do something useful/provide advice, let me know.

Comments

[mamishka.livejournal.com]

Is it possible that we can rebuild from a backup and just change all the passwords, or am I just a big dummy that doesn't understand that that won't work?

[lyonesse.livejournal.com]

who/where was the unusual login?

my work site got hacked last week; if it was my account then i probably have more cruft to clean up at work :/

[lyonesse.livejournal.com]

there are two reasons one usually rebuilds in cases like this:

first, people don't usually back up the "system" files -- like the programs you run and stuff, which are what live in /bin -- just people's own data. but since dar knows that stuff like that has been changed, it would need to be replaced, and may not have been backed up. since she's asking people to reinstall/recompile, my guess is that some system libraries have been changed as well. it's safer to just thoroughly clean out if you are uncertain of (a) the extent of the damage and (b) the exact time of the compromise, and know you have a clean backup from before it.

second, if you don't know how the crackers got in, there may be a security hole in your system that you don't know about. in which case updating the entire system is the right thing to do.

[smeehrrr.livejournal.com]

If you want, I can set up secondary MX for you here. You won't be able to access your mail until your server is back up, but I can queue it up for you so it won't bounce.

Although it looks like you don't actually have any working DNS either, so that probably won't help.

[solarbird]

Yeah, we're pretty fucked. 209.20.199.4 is our backup DNS, and it is up, but isn't a registered DNS server for internic purposes. I will correct that as soon as I've validated that door was inadequately interesting to damage.

[solarbird]

Not yours.

[mamishka.livejournal.com]

Okay, so there is the rebuild, but can we still get data from the backups and input it, or is it potentially risky? I wouldn't think so, but then again I'm not a computer person. I hate feeling this stupid and useless. I want to help and I can't. :(

[lyonesse.livejournal.com]

anything like text or html is absolutely fine and can be restored from backups. though you'll want to check and see if they still say what you wanted them to say :) it's only executables -- runnable programs -- that you would not want to restore. for such things you would want to reread the code, see that it has not been tampered with, and rebuild it into a computer executable.

don't fret that you don't happen to be a sysadmin. i work with BRAIN SCIENTISTS!! -- and very very very few have the least idea how to administer computers :)

[cow.livejournal.com]

I'm not involved in murknet, but I'm a systems administrator and have experience running both my own and other peoples' boxes. If I can be of any assistance, don't hesitate to ask. :)

[solarbird]

Do you know how to take an ISO image and make a CD-ROM out of it under MacOS X?

[cow.livejournal.com]

I believe it's Applications -> Utilities -> Disk Utility. You should be able to drag the ISO into the main program, and then burn it to a CD-ROM.

I've never tried this, however, as my OS X box doesn't have a burner. But it should do it.

(this is Panther -- 10.3. Your mileage may otherwise vary, but I'm pretty sure it's been that same program back to 10.0.)

[annathepiper]

I have semi-recent backups on tape--this will help us most likely rebuild most of the most active web pages, since a lot of the web page data we have on the system doesn't change very quickly. So at least in that respect we should be okay. I will be touching bases with Dara as well to evaluate the status of the rest of the user account files on the system so that we can determine exactly what got fucked up and what didn't.

The more worrisome part of this is the operating system itself. :( As Vicka has posted, it will be most secure for us to flatten the box, start from scratch, and make damn sure that we have all the latest security updates before we go live again.

And people bitch at Windows boxes for these problems. :P GAH!

(My own personal worry re: user data is going to be the huge mysql databases that power both my web pages--the personal one and the Two Moons one. If I can't get that back online and have to retype all that in, I will be EXTREMELY pissed. :P )

[solarbird]

Yay! I'll let you know how it works once these images finish downloading. Or as soon as one of 'em does, at least.

[cow.livejournal.com]

*bows* Glad to be of service. :)

[backrubbear.livejournal.com]

A paranoid sysadmin friend recommends the following:

http://www.grsecurity.net/

[solarbird]

turned out it's Disk Copy, which makes sense. but it's one of the places where the MacOS user experience falls down and goes HA HA FUCK YOU. i'm still trying to get the coaster i made to eject without having to do something stupid like reboot. i'm hoping whatever process I confused will time out soon.

[solarbird]

There is no joy. Since I have a multi-hour download going that I don't want to interrupt, do you know how to force-eject an unburned CD that isn't showing up in Finder (Finder thinks it's ejected already), won't respond to umount, and isn't responding to the dedicated eject key?

[(Anonymous)]

Mrr, Disk Utility - NOT disk copy - in OS X back at least to 10.1 should have an eject button. In OS X (panther) it's one of the blue ones at the top. This assumes you can select the drive with the coaster in it. You don't have to select the coaster - trying to eject the drive will have the desired effect. (meaning, it will not attempt to spit the drive across your living room...)

Failing that, there are paper clips. Does your CD tray have a tiny hole anywhere? It may be hidden under the faceplate. If the OS doesn't know it has a CD, it won't hurt it to remove the coaster without it ever knowing otherwise. :)