HostingGeek

Housekeeping: Well… It’s been a while since I have posted. I had been meaning to post some things but Google turned off their FTP service in Blogger making their service really useless to me and so this blog languished while I finally got around to sorting things out and moving it to a proper blog software. I hope to be providing more updates now, but I won’t fool myself too much.

On with the show….

I recently came across a binary application on a Linux server that had been coded to prevent people from snooping into what the application actually did. If you ran the application using strace or gdb the application would detect it and stop running. It would throw an error similar to:

“Debugging detected… goodbye!”

and the application would simply exit. Now I understand a programmer’s desire to protect their code, however if you are going to be running an application on my server I should at least know what it is you are actually doing. Apparently this is known as “anti debugging” and is designed to prevent reverse engineering of an application. Not being one to turn down a challenge…. I accepted. Below I will outline some very simple processes that can be used to circumvent some of the more basic checks.

There are apparently a few different methodologies involved in anti debugging, and it seems that our application used a couple of checks. I will show you some of what the application was doing and the very simple workarounds to get around them.

When you attach a debugging tool such as strace or gdb to the command it would throw off the process id of the parent application. Apparently the application was checking that by using a ptrace call to attach to the parent process id (getppid). Here is how I worked around this issue.

We override the getppid system call with our own. Create a very simple C application that will define our new getppid() calls.

#include <stdio.h>

int getppid() { return getsid( getpid() ); }

What this does is return the session id for current process’s id anytime getppid() is called. So now we need to compile this into a shared library that we can load before we execute the application we want to debug.

gcc -shared -fPIC -o fakegetppid fakegetpppid.c

This compiles the C application into a shared library that we can then load. Here is how we tell something like strace to use this instead of the normal getppid system call.

strace -fxi -E LD_PRELOAD=./fakegetppid /path/to/the/main/application

If there are no other anti-debugging tricks in place, the application will execute as normal.

What if you have an anti-debugging trick that isn’t so simple to thwart? What if the application makes multiple calls to a system call that you need to catch a specific instance of? Well there is a way to accomplish that as well.

#define _GNU_SOURCE 1

#include <stdio.h>
#include <dlfcn.h>

static int(*next_ptrace)(int a, int b, int c, int d ) = NULL;

long ptrace( int a, int b, int c, int d )
{
    if( next_ptrace == NULL ) {
        next_ptrace = dlsym( RTLD_NEXT, "ptrace" );
    }

    if( a == 16 ) { /* PTRACE_ATTACH */
        fprintf( stderr, "PTRACE_ATTACH called with pid %i\n", b);
    }

    return next_ptrace( a, b, c, d );

}

This will allow you to specifically capture PTRACE_ATTACH calls and do whatever you want with the calls, sending all other calls to the original system ptrace call. That’s pretty darn powerful and works exactly the same way as the code above.

I hope this information is useful to some people out there. I know it is a little complex, but it just goes to show that it is possible to debug an application, even if it doesn’t appear to be at first.

Good luck and happy hunting.

There isn’t a whole lot of information about how to setup CentOS as a DomU under a Debian 4.0 based Dom0 and still maintain the use of pygrub to boot the CentOS kernels. This howto will give you a general overview on what steps to take without having to use an incomplete CentOS image. This is not going to be a copy and paste sort of howto, but rather a more high level detail, and a couple of fixes to make it all work correctly.

A couple of assumptions I am making here:

1. You have a working Xen install already under Debian
2. You can edit files using vi or a comparable editor.
3. You understand how Xen and LVM can work together at least at some basic level
4. You are confident to compile your own applications using make, etc…

Here is what you need to do to get started:

Step 1:

Download the kernel image and ram disk for CentOS and put them some place you can access them on the Dom0.

In my case, I put them in /usr/local/src/xen/ (vmlinuz and initrd.gz respectively). I downloaded these files from a CentOS mirror. The files you are after are located in the centos/5.1/os/i386/images/xen/ directory as these contain the Xen code compiled into the kernel so that you can boot the DomU in paravirtualization mode.

Step 2:

Create a Xen DomU configuration file that points to these files for the boot kernel.

I edited the two lines:

kernel = “/usr/local/src/xen/vmlinuz”
ramdisk = “/usr/local/src/xen/initrd.img”

This tells Xen to use these kernels on boot up.

Step 3:

Modify your DomU config to point to your disks:

disk = [ ‘phy:/dev/xen01/centos5-disk,xvda,w’, ‘phy:/dev/xen01/centos5-swap,sda1,w’]

It is important to note that you must export the drives from the Dom0 as xvda, otherwise the CentOS installer will not be able to detect them properly and you will have no target drive to install to.

We will also want to modify the default restart behavior as you will see later, this is important:

on_reboot = ‘destroy’

Step 4:

Go ahead and boot up the Xen DomU using xm create -c

Install CentOS as a normal network installation (point it at an FTP or HTTP mirror and let it install normally).

Step 5:

Once the CentOS installation is completed, the DomU will attempt to reboot itself. This is why we set the on_restart to destroy instead of the default of restart. We need to edit the configuration to boot up via pygrub instead:

bootloader = “/usr/lib/xen-3.0.3-1/bin/pygrub”

Step 6:

Here is where things get a little tricky. The pygrub application is missing a library that it needs in order to boot up CentOS based kernels. We must build this ourselves.

Download the xen-3.0.3 source (the new sources do not build this file, so I used this version specifically, I don’t know if others will work). I know for a fact that xen-3.2.0 does not work.

wget http://bits.xensource.com/oss-xen/release/3.0.3-0/src.tgz/xen-3.0.3_0-src.tgz

Untar the file and cd into the directory xen-3.0.3_0-src

Then:

cd tools/pygrub

Then you need to run make. Pay attention to the errors, you might need to install additional libraries if you don’t have them on your Dom0. (e2fslibs-dev comes to mind).

Step 7:

Once your build has successfully completed, you will need to copy the files to your local xen installation.

cd build/lib.linux-i686-2.4/grub/fsys/ext2
mkdir /usr/lib/xen-3.0.3-1/lib/python/grub/fsys/ext2
cp * /usr/lib/xen-3.0.3-1/lib/python/grub/fsys/ext2/

Step 8:

Boot your DomU using:

xm create -c

Finished:

You should now have a working Xen DomU under Dom0 without having to resort to broken CentOS images.

What is my DNS server responding to?

We have been in the process of moving from an old server to a newer server. The process is straight forward, we move the sites over to the new server and then update their zone records to point at the new server (the zone has a low TTL – Time To Live to make this transition smoother). Overall everything has gone smoothly with little interuption in the service of each site.

Finally once everything was moved over, we updated the nameserver records to point at the new server so now everything should be running off the new server’s DNS. We are ready to turn off the old server, but noticed that named (bind) was still handing out DNS responses (based on its activity in top). We thought we had everything updated so that this server shouldn’t be used at all.

So we had to find out what DNS requests were still hitting the old server and why we missed those. Here is what we did to find out.

Edit your named.conf (ours was in /etc).

Add the following section if you do not already have a section called logging {}.

logging {
channel query_logging {
syslog daemon;
severity debug 9;
};
category queries {
query_logging;
};
};

What this does it record any DNS query named serves up in the default syslog for named (generally /var/log/messages). This will help you see what domains are being requested from your server.

We determined what DNS queries were coming in, and based on the whois information found out that there were some very old nameserver records pointing at the server’s IP. Without the logging change above, we could have lost 3 or 4 long time customer’s DNS information when the old server was turned off. As it is now we have updated those nameserver records to point at the new nameservers, and will need to keep the old server up and running for at least another 48 hours (the amount of time a root nameserver record is cached). Saved us a black eye for sure.

What else is my DNS server handing out?

Additionally, you might want to look at the log information and determine if anybody is using your server for recursive lookups too.

What is DNS recursion?

Well, recursion itself isn’t bad, and actually a vital part of DNS. Recursion means that if you request a DNS lookup against a DNS server, and that server isn’t authoritative for that domain (it doesn’t have a zone for that domain), it must pass the DNS request to another server.

Why is it bad to allow recursion?

Until recently DNS recursion wasn’t really a bad thing, but hackers have determined that it is possible to “amplify” or magnify their DDoS (Distributed Denial of Service) attacks using spoofed UDP based DNS requests. (UDP is extremely easy to spoof the originating IP address of the request.) The hackers send a spoofed UDP request for a given domain with a large number of records to a DNS server that allows recursive lookups. Since the initial UDP request is realtively small, and the response (because it has so many records in it) is very large, hackers can amplify the amount of data they can send at a target using recursive third party DNS servers.

How do I turn off recursion in named/bind?

To turn off recursive lookups from unauthorized sources you can add the follownig ACL to your named.conf:

acl recursion { 127.0.0.1; 1.2.3.4/24; };

And then in your options do:

options {
allow-recursion { “recursion”; };
};

The first line creates an ACL (Access Control List) to let named (bind) know who is allowed to do recursive lookups against the server. The IP’s should be listed in CIDR notation, and be followed by a semicolon. Include any IP address that uses this server for legitimate DNS lookup purposes.

The second section should already exist in your named.conf, and you just want to add the allow-recursion line to that section. This will apply the ACL to your server. Then you just need to restart named, and you are good to go.

So that is why you should know exactly what your DNS server is doing.