« Archives in September, 2006

Find Out What Your DNS Server is Doing

September 25, 2006 / posted in SysAdmin / Comments Off

What is my DNS server responding to?

We have been in the process of moving from an old server to a newer server. The process is straight forward, we move the sites over to the new server and then update their zone records to point at the new server (the zone has a low TTL – Time To Live to make this transition smoother). Overall everything has gone smoothly with little interuption in the service of each site.

Finally once everything was moved over, we updated the nameserver records to point at the new server so now everything should be running off the new server’s DNS. We are ready to turn off the old server, but noticed that named (bind) was still handing out DNS responses (based on its activity in top). We thought we had everything updated so that this server shouldn’t be used at all.

So we had to find out what DNS requests were still hitting the old server and why we missed those. Here is what we did to find out.

Edit your named.conf (ours was in /etc).

Add the following section if you do not already have a section called logging {}.

logging {
channel query_logging {
syslog daemon;
severity debug 9;
};
category queries {
query_logging;
};
};

What this does it record any DNS query named serves up in the default syslog for named (generally /var/log/messages). This will help you see what domains are being requested from your server.

We determined what DNS queries were coming in, and based on the whois information found out that there were some very old nameserver records pointing at the server’s IP. Without the logging change above, we could have lost 3 or 4 long time customer’s DNS information when the old server was turned off. As it is now we have updated those nameserver records to point at the new nameservers, and will need to keep the old server up and running for at least another 48 hours (the amount of time a root nameserver record is cached). Saved us a black eye for sure.

What else is my DNS server handing out?

Additionally, you might want to look at the log information and determine if anybody is using your server for recursive lookups too.

What is DNS recursion?

Well, recursion itself isn’t bad, and actually a vital part of DNS. Recursion means that if you request a DNS lookup against a DNS server, and that server isn’t authoritative for that domain (it doesn’t have a zone for that domain), it must pass the DNS request to another server.

Why is it bad to allow recursion?

Until recently DNS recursion wasn’t really a bad thing, but hackers have determined that it is possible to “amplify” or magnify their DDoS (Distributed Denial of Service) attacks using spoofed UDP based DNS requests. (UDP is extremely easy to spoof the originating IP address of the request.) The hackers send a spoofed UDP request for a given domain with a large number of records to a DNS server that allows recursive lookups. Since the initial UDP request is realtively small, and the response (because it has so many records in it) is very large, hackers can amplify the amount of data they can send at a target using recursive third party DNS servers.

How do I turn off recursion in named/bind?

To turn off recursive lookups from unauthorized sources you can add the follownig ACL to your named.conf:

acl recursion { 127.0.0.1; 1.2.3.4/24; };

And then in your options do:

options {
allow-recursion { “recursion”; };
};

The first line creates an ACL (Access Control List) to let named (bind) know who is allowed to do recursive lookups against the server. The IP’s should be listed in CIDR notation, and be followed by a semicolon. Include any IP address that uses this server for legitimate DNS lookup purposes.

The second section should already exist in your named.conf, and you just want to add the allow-recursion line to that section. This will apply the ACL to your server. Then you just need to restart named, and you are good to go.

So that is why you should know exactly what your DNS server is doing.

How To Install NetBSD as a DomU in Xen 3.0

September 14, 2006 / posted in Uncategorized / 2 Comments

Ever since the first time I heard about Xen and its ability to run any OS side by side on the same server I have had the urge to run a BSD based OS with a Linux OS. Today I have sucessfully achieved my goal, and this is how I did it.

First some background on the server itself. The server is a Dell PowerEdge 1750 with Dual Xeon processors and 3GB of RAM and 500GB of RAID storage. The server is running the Xen 3.0.2 hypervisor kernel (the main kernel that handles the paralization, or virtualization, of the hardware). The Dom0 system is running Debian 3.1 with some patches to the kernel to work with the LSI based RAID 5 card in the server. Each virtual OS installed on the server is given its own partition and is managed using LVM in Dom0.

The vast majority of information about NetBSD running under Xen as a DomU seems to be either Xen 2.0 specific, or assumes you are running NetBSD as Dom0. Unfortunetly, the Xen 2.0 information is no going to work on a Xen 3.0 machine, and more so our Dom0 is Debian, so we needed to come up with our own.

Here is how I did it, and what sort of problems I encountered.

The entire process is pretty easy, but finding the actual information can be tough, and finding the files you need can be even tougher. Here is kind of a rough over view of the process…

1) Set up your partition that will hold NetBSD. We are using a LVM partition named vg00-netbsd.
2) Set up the xen domU config file.
3) Boot the netbsd install kernel for Xen 3.0.
4) Follow the sysinstall steps like you normally do to install NetBSD. I had to use an FTP based installation, because I could never get the CDROM to work correctly.
5) Complete the install and shutdown NetBSD.
6) Edit the domU config file and change the kernel from the install kernel to the normal NetBSD kernel.
7) Boot NetBSD DomU and enjoy.

So here are the specifics.

Step 1: You need to download the NetBSD Xen 3.0 kernels (install and normal) and put them some place on your Dom0. I put mine in the /boot of the server, because it sort of made sense to me, but they can be almost anywhere. You can download the DomU kernels from NetBSD’s FTP servers from the daily build areas. The kernels for Xen 3.0 are not in the release versions of NetBSD so you have to find them. I would post links to them, but most likely the would go stale over time. Go to ftp://ftp.netbsd.org/pub/NetBSD-daily/ and navigate through to either the NetBSD 3.1 tree or the NetBSD 4.0 tree. You are looking for the directory i386/binary/kernel/ in that directory you will find the two kernels you need. The install kernel is called netbsd-INSTALL_XEN3_DOMU.gz and the normal kernel is named netbsd-XEN3_DOMU.gz. Download both of those kernels as you will need them later.

Step 2: Once you have downloaded your kernels you will need to create a xen config file for your NetBSD DomU. Here is an example of the one I used:

kernel = “/boot/netbsd-INSTALL_XEN3_DOMU.gz”
memory = 128
name = “netbsd”
vif = [ ” ]
disk = [ ‘phy:/dev/mapper/vg00-netbsd,0x01,w’ ]
root = “/dev/wd0d”

You will need to change the disk = line to match where you are installing NetBSD to on your server. After you have created that file in your xen config directory (our was /etc/xen/).

Step 3: We are ready to boot NetBSD for the first time. To boot NetBSD we run the command:

xm create -c netbsd

“netbsd” is the name of the DomU config file we created in step 2, so change that to match what you used in that step.

A couple of times we noticed that Xen didn’t attach us to the console of the booting NetBSD DomU, so you may need to connect to it manually. To do so do the following:

xm list

Which will print out a list of running Xen instances like this:

Name ID Mem(MiB) VCPUs State Time(s)
debian 0 1374 4 r—– 2354.5
plesk 10 1024 1 -b—- 161.5
netbsd 50 128 1 -b—- 1.2
qmail 8 128 1 -b—- 953.4

We will need to know the ID of the instance we want to attach to. In the example above this is 50. Then we attach to the console of that DomU by typing:

xm console 50

To break out of the console at any time simple press CTRL+] at the same time.

Step 4: Once you are in the console you should see the sysinstall application. You can follwo the prompts and install NetBSD like you would normally do. One problem I did encounter was that for what ever reason the server would stop talking to the FTP server due to some sort of DNS lookup failure. It did this no matter which kernel I tried. I eventually resorted to using the IP address instead, and the installation worked perfectly.

Step 5: Once the install is completed, break out of the server and shut it down via the command:

xm shutdown 50

Again replace 50 with the ID of the DomU of your NetBSD install.

Step 6: Edit the DomU file and change the kernel line to point to your normal NetBSD kernel. So your DomU config file should look somethnig like this now:

kernel = “/boot/netbsd-XEN3_DOMU.gz”
memory = 128
name = “netbsd”
vif = [ ” ]
disk = [ ‘phy:/dev/mapper/vg00-netbsd,0x01,w’ ]
root = “/dev/wd0d”

Step 7: Reboot your NetBSD DomU via the command:

xm create -c netbsd

Enjoy your NetBSD running under a Debian/Linux Dom0 in Xen 3.0!

Gotchas:

Having used the NetBSD system only breifly I have noticed that there is something “funky” with the networking and the way it behaves. I noticed over a sustained ping that the network interface starts to drop packets, every other packet it seems. Modifying the vif = line in the DomU config to read:

vif = [ ‘bridge=xenbr0’ ]

seems to have cleared up the issue. This line bridges the ethernet interface inside the DomU to the xenbr0 interface in the Dom0. It seems to have cleared up the issue to date.

And there
you have it! NetBSD running under a Linux Dom0 on top of Xen 3.0. The world just got a whole lot smaller.