HostingGeek

I previously wrote a “How-To” article detailing how to setup VLAN Tagging (802.1q) using FreeBSD and a Cisco 2924-XL-EN. You can read the article for more information, but basically VLAN tagging is a way isolate traffic to particular ports on a switch. In addition that How-To also covers how to use the FreeBSD machine to provide rate limiting and firewall protection to those VLANs.

Anyways… In that article I touch on the fact that we must use device polling to avoid context switching of the kernel back and forth between userland applications and kernel processes. To truly understand the benefits of device polling you should read the device polling web page which explains the reasons behind device polling better than I ever could hope to.

So I never got a chance to go back and write the information on how to set up device polling like I wanted to… Until now.

First some quick and pretty graphs of a P4 server pushing about ~50Mbps (core router for a data center) without device polling enabled:

   1 users    Load  0.00  0.00  0.00                  Aug 29 22:54

Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER
       Tot   Share      Tot    Share    Free         in  out     in  out
Act   16496    4784    24640     5304  645424 count
All  375240    5984  2742020     7080         pages
                                                                Interrupts
Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow   16713 total
    1        6        10    1   6016714    3    3 103724 wire        stray irq7
                                                   16412 act         em1 irq5
1.9%Sys  21.5%Intr  0.0%User  0.0%Nice 76.6%Idl   255080 inact       em2 irq12
|    |    |    |    |    |    |    |    |    |         24 cache  7133 em3 irq10
=+++++++++++                                       645400 free      1 ata0 irq14
                                                         daefr  1355 mux irq11
Namei         Name-cache    Dir-cache                     prcfr       fdc0 irq6
   Calls     hits    %     hits    %                     react  1000 clk irq0
                                                         pdwak   128 rtc irq8
                                         zfod            pdpgs     3 mux irq5
Disks   ad0                               ofod            intrn  7093 mux irq12
KB/t   2.00                               %slo-z   113712 buf
tps       1                               tfree         3 dirtybuf
MB/s   0.00                                         69954 desiredvnodes
% busy    0                                         59766 numvnodes
                                                      26 freevnodes

Note the long line of “+” signs. Those are IRQ interrupts that the CPU has to handle. At the time this screen scrape was taken, over 20% of the CPU was spent handling IRQ requests from device em3 (right column 7153 IRQ requests).

Now the same exact server with the same exact data flow:

   1 users    Load  0.00  0.00  0.00                  Aug 29 22:55

Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER
       Tot   Share      Tot    Share    Free         in  out     in  out
Act   17504    4784    26020     5304  645424 count
All  375240    5984  2742020     7080         pages
                                                                Interrupts
Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    1124 total
             8         9    1   50 1124    4    3 103724 wire        stray irq7
                                                   16408 act         em1 irq5
3.1%Sys   0.8%Intr  0.0%User  0.0%Nice 96.2%Idl   255084 inact       em2 irq12
|    |    |    |    |    |    |    |    |    |         24 cache       em3 irq10
==                                                 645400 free        ata0 irq14
                                                         daefr       mux irq11
Namei         Name-cache    Dir-cache                     prcfr       fdc0 irq6
   Calls     hits    %     hits    %                     react   996 clk irq0
                                                         pdwak   128 rtc irq8
                                         zfod            pdpgs       mux irq5
Disks   ad0                               ofod            intrn       mux irq12
KB/t   0.00                               %slo-z   113712 buf
tps       0                               tfree         4 dirtybuf
MB/s   0.00                                         69954 desiredvnodes
% busy    0                                         59766 numvnodes
                                                      26 freevnodes

Now note the difference. The CPU is now free to handle other things instead of IRQ requests. That is the power of device polling.

Here is how you set it up.

First you have to make sure that the network interface you are using supports device polling. The author’s web site linked above lists support for “fxp” (Intel 10/100 cards), “sis” (SiS based network cards – not sure which), and “dc” (Some sort of DEC card). One card which is not listed on that page, but I know supports device polling is the “em” card (Intel Gigabit cards). If there are other cards that support device polling please let me know. I have only used fxp and em cards so I know for a fact that they work.

So now that we know we have a network interface that is going to support polling (it is really the device driver that supports polling) we can get started on adding the support to the kernel. This is really simple and is comprised of two directive in the kernel config file:

options DEVICE_POLLING

options HZ=1000

Once you have those in your kernel config, you can recompile your kernel and boot it up. If you need help doing that part consult the FreeBSD How-To for creating a custom kernel (and you are running a custom kernel with FreeBSD, right?).

Now you have a computer that has a network card that supports device polling, a kernel that is device polling enabled as well, a real time clock that is more responsive (the HZ=1000 line), and the kernel is booted and we are ready to rock and roll. Now we just need to issue:

sysctl kern.polling.enable=1

Now we have it enabled and you are all set to go forth into that dark night and let your devices be polled.

I hope that sheds some light on a subject that can be a of great use for high network traffic servers (routers, firewalls, proxies, etc).

Let me know if you found it useful.

Many distros have easy ways to keep the OS up to date with security fixes and patches. Probably the easest to use of this group is apt-get. Apt-get originally started off with the Debian distribution where it is responsible for not only updating software, but also install and removing software easily.

Additionally… Some one has ported apt to the Fedora/Redhat distribution where it (in my opinion) blows away any other implementation (yum, up2date, etc).

I install apt on all my Fedora machines and use its features to find, install and keep up to date the server’s installed software.

Probably the biggest problem with Fedora these days is that Fedora’s OS is phased out very quickly. What do you do if you have Fedora Core 1 or Core 2 installed on your server, and the Fedora project has (as of this posting) moved on to other releases. Enter apt, and the Fedora Legacy group. Using Fedora Legacy you can keep your OS up to date with recent patches using their apt repository. You can read more about their repositories here.

You can setup apt under Fedora and point apt to use Fedora Legacy’s apt repository and keep your server up to date quickly and easily.

Here are some handy commands to run with apt:

apt-get update

This command is to update apt and its sources list to ensure that it is looking for the correct version of files to download and install. As a general rule you should run this at least once before using apt.

apt-get install {name of package}

This command will go out to the respoitory and fetch the latest version of the package you specify. eg: apt-get install httpd will install the latest version of apache if it is not installed yet.

apt-get remove {name of package}

This command will uninstall the software package you specify. eg: apt-get remove httpd would uninstall the httpd package if it was installed.

apt-cache search {name of package}

This command is extremely handy if you can’t find the name of the package you are looking for. Sometimes, especially with lib packages, the names can very greatly. apt-cache search php would show you a listing of all the package names that contain php or contain php in the description.

apt-get upgrade

This command is the best command of all. It will look at your server and the list of packages installed, and then it will look at the repository to determine which packages you have installed have been updated. It will then download and install those packages. Very handy. I run this once a week for a server.

Some packages are excluded from being downloaded. Of those is the kernel. On some servers you don’t want to upgrade the kernel if you don’t have to, but sometimes you want to do that because of local user kernel exploits. To get a list of available kernels to install you can run:

apt-get install kernel

This will output a list of kernels that are available and you can select a name from the list to force it to be installed.

I should note that installing a kernel can be risky business, especially when you are not local to the server. Having said that, I have been very lucky with installing kernels from apt-get.

I hope this information shows you that maintaining a server and its software isn’t as hard as you think it is. It just takes a little setup and the right tools. Using apt-get and Fedora Legacy’s apt repositories, you can keep your server safe and secure.

Recently I did an entry on SCP (Secure CoPy) which uses SSH to copy a single file over a secure ssh tunnel to a remote server or to copy a remote file to a local directory. This works great for a single file, but what if you want to do an entire directory?

Well one way is to tar up the directory, then copy the file to the remote server (using scp perhaps?) and then login to the remote server via SSH (you aren’t using telnet any more right?) and then untar the file on the remote side. Pretty simple, but since we are geeks, we try to do things as efficiently as possible (even if there are better solutions).

Enter tar over SSH.

Tar has the great ability to send data to stdout/stdin using the “-” (dash) as a filename in the command line. So using that we can string together pipes to send the data to a remote server. Let’s explore how:

tar -zcf – ./ | ssh remoteuser@remotehost tar -C /path/to/remote/dir -zxf –

What this does is pretty simple: it creates a compressed tar file of the current directory (./) and sends it to stdout (-). We catch stdout with the pipe (|) and then call ssh to connect to a remote server where we execute the remote command tar. Using the remote tar we change directory to /path/to/remote/dir and then run a decompress extraction of stdin (-).

About the only caveat of this method is that tar must be in the “remoteuser’s” path, otherwise you would have to specify the fully qualified path to the binary for tar.

Great way to transfer a bunch of files securely, as well maintain ownership and permissions.

Try and come up with some other uses, and post them in the comments.

Today’s blog post is going to be a little bit off topic for me… Still something that I deal with… but not 100% related to hosting in general…

Web development tools.

I do development in PHP. It helps pay the bills, and keeps me busy, plus in some weird sort of way it gives me a creative outlet.

Recently I had to build a rather large intranet/internet application for a gymnastics company (you know the kind where you take your kids there and the teach them how to tumble and not break their necks while doing so). Anyways… One of the design criteria that was imposed was to make sure that the site was standards compliant, with the client specifically outlining XHTML compliant.

So I setup the site with templates (since I am a PHP programmer and I know it well, I used Smarty). Making a site that is standards compliant as well as CSS enabled is not easy, as I quickly found out. CSS coding is an art form, because not only do you have to apply the CSS to the site and make sure it works as you intended, but you also have to check it in other browsers to make sure it renders properly there as well (a particular area that IE fails miserably at).

Enter one of the most useful extensions for the FireFox browser to date, Web Developer. To say that Web Developer, or webdev, is good would demonstrate a textbook definition of understatement. Flat out… IT ROCKS.

Probably the feature I use the most is the “View Style Information” feature that allows you to select any CSS element on a rendered page in your browser (regular web page) and see the CSS method behind it. This is such a great feature that it is the primary reason I started using webdev in the first place. About 8 months ago I had a page that I couldn’t figure out how to get to render how I wanted. I must have played with the style sheet for 3 hours before a friend recommended I try this FireFox extension. Within 10 minutes I had figured out the stylesheet quirk and fixed it. Amazing.

But it has other features too… It can outline block level elements (divs, p, etc..), tables, and table cells. It has built in menu items for CSS and HTML validation (using the W3 validator websites). It can display passwords hidden in input/password fields (you can view the source sure, but this is so much easier). You can automatically populate form fields for testing. You can even clear HTTP based authentication, session cookies… It is like the swiss arm knife for web development (complete with the little toothpick).

Another great extension that I use for web development is ColorZilla. It is another extension that sits in the status bar of FireFox and has an eye dropper that you can use to sample any color on the page. It doesn’t matter if it is a picture, a CSS style element, text, whatever… You can get the hex code for the color as well as RGB code for it. I think I use it almost as much as I use the webdev extension.

So there you go… Two of the most useful FireFox extensions for web developers. Load them into FireFox today and start basking in the glow.

One of the greatest features of SSH is the ability to use key based authentication. Key based authentication uses the public/private key method to allow logins to SSH without the use of passwords.

This method is better than password based authentication as a password can be brute forced. While it is theoretically possible to brute for a public/private shared key combination, the amount of computing power is pretty stagering, as well as the amount of time invloved to accomplish this using standard computers today. (Quantum computing is said to be able to accomplish this same feat in seconds… But as of yet quatum computers don’t exist outside of the lab that are large enough to accomplish this… so we are safe… for now.)

It works pretty simply. You create a private and public key pair for your login. You can do this by running:

ssh-keygen -t dsa

This will generate a public and private key using the DSA method (you can also use RSA).

You will be prompted on where to store the key pairs, as well as for a passphrase. The passphrase will protect the private key from unauthorized usage, but also negate any advantage of using keys to authenticate automattically without passwords (there is a way to do this but it is outside the scope of this current discussion and will be another blog entry). I would recommend skipping the passphrase for the time being.

Once you have created your public and private key pairs (in your home directory in the .ssh directory generally). You will need to copy the public key to each server you wish to authenticate with, and not have to type a password to do so. You can use the scp method (covered in a previous post) to copy the files to the remote server under the username you login in the .ssh directory in your home folder (so for user xyz it would be /home/xyz/.ssh). After you have copied that file to the server, you will need to copy that file to the file authorized_keys. You can do that by running:

cat /home/xyz/.ssh/dsa.pub >> /home/xyz/.ssh/authorized_keys

Now to test it, type in ssh xyz@remoteserver and you should be logged in right away with no password prompt. If you didn’t do it right, then you will be prompted for a password.

Now I have to mention… Because we aren’t usnig a passphrase for your private key… if somebody were to get control of your private key, they can now login to all the servers that use that private/public key combination. So keep that in mind.

I use key based authentication for most of my servers, because I have gotten so tired of password based brute force attacks on my SSH daemon. I keep my private keys on a keyfob on my keychain. So if I have my car keys… I have access to my servers. I also have the private keys in another safe place… and if I told you where that was… I would have to kill you… So lets leave it at that.

Recently while working on a server, I noticed that there was some unusual files in the /tmp directory. This always sends up red flags in my head so I investigated more closely and determined that somebody had placed a file on the server via a PHPBB exploit and was using it as a means of building a zombie network.

Typically this type of activity is closely linked to “script kiddies” and not legitmate hackers. Script kiddies are people with no more hacking ability than anybody else, they simple know how to read about holes in certain software and use other legitmate hackers work to exploit those holes, to some unknown end.

This case was no different. Here are some fun things you can do to mess with them. Many times the script kiddies will leave the software on the server, and that itself is a goldmine of information.

Case in point, I found a binary on the server that was connected to an IRC server. I did a quick review of the process list (ps -auxwww) and determined the process id of the application running. Then I ran:

strace -s 16000 -p {process id}

This command is called stack trace and will attach to an already running application to see what it is doing. This is particularly useful if the binary in question is no longer on the hard drive.

So I attached to the process and was able to determine that it was connected to an IP address on port 6667. Port 6667 is one of the default ports for IRC. So I fired up my trusty IRC client and connected to it. Sure enough I was connected to a server with 106 clients and 3 operators.

So now I had to figure out what channel to join. Lucky for me the binary was still on the hard drive. Here is where the second part comes in. Because script kiddies only use the software and they don’t know how it actually works, we can often get more information from the binary. So to glean information from the binary I ran:

strings {filename}

This printed out a long list of text based strings in the binary. The best part is, the IRC channel that the zombie was supposed to join on the IRC server, and the password for the IRC channel, were right there in front of me. So… I joined the channel.

I must have scared the living hell out of the operator of the zombie network, because as soon as I started talking to him, I was firewalled off of the server completely. All new connections coming into the server were also blocked. Luckily I was able to collect a list of IP addresses of the zombies in the channel before I was disconnected, and I have been systematically notifying the operators of those machines that they have been comprimised.

I am sure somewhere in the world… a script kiddie had to go clean out his pants after that episode. I hope I put a dent in his “zombie network”.

Armed with some of this information I hope that you will be able to torment script kiddies when you encounter them. Not all of them are this easy… But when they make it easy… I say take the time to mess with them… If nothing else it may scare them just enough to stop for a little bit.

I was looking for a way to generate some random password in a shell script I was writting and I stumbled across a handy little program: pwgen

According to the pwgen man page:

pwgen generates passwords which are designed to be easily memorized by humans, while being as secure as possible.

The pwgen program is designed to be used both interactively, and in shell scripts. Hence, its default behaviour is(sp) differs depending on whether the standard output is a tty device or a pipe to another program. Used interactively, pwgen will display a screenful of passwords, allowing the user to pick a single password, and then quickly erase the screen. This prevents someone from being able to “shoulder-surf” the user’s chosen password.

So there you have it, a nice quick way to create a password. And just a sample of how I used it in my shell script:

CLEARPASS=$(pwgen -N 1)

This returns one somewhat random password to the shell variable CLEARPASS.

Tomorrow I hope to have a nice write up on how to do password-less authentication via SSH shared keys.

Background: We host a very large organization that has chapters in all 50 states (of the United States that is). We choose to host these sites under Plesk because it offloads some of the more mundane support requests to the users, and trust me that is a good thing.

All the sites are in the format .somedomain.com. Now, we also manage the codebase for the content management system for these sites. It is my job to set these up. It is boring and I might throw myself in front of a large bus if I had to do this all by hand.

Enter one well crafted shell script:

for i in `find /var/www/vhosts -type d -name “*.somedomain.com”`; do FOO=$(cat /etc/passwd |grep $i| sed ‘s/:.*//’); cd $i; rm -rf $i/httpdocs $i/sql $i/.svn; svn checkout https://svn.somedomain.com/svn/someproject/trunk ./; chown $FOO:psaserv $i/httpdocs; chown -R $FOO:psacln $i/httpdocs/*; done

Here is what this one does: Since we already have all the domains on the server (and this is a Plesk server) we are just going through the list of sites in the /var/www/vhosts directory and working with each of them. Now Plesk (particularly the latest version 7.5.x) can be very picky about ownership and permissions (especially on the httpdocs folder), so we need to correct them as we go along.

Also it should be noted that all of the sites were previously under SVN control, however that SVN was removed due to the wrong codebase being imported into the SVN originally. So…

This loop goes through and takes the full path for the domain and places it in the variable $i. We then parse the passwd file to find out which user owns that directory (important for FTP reasons) and place that information into a bash variable. Then we remove the previous httpdocs, .svn, and sql directories since SVN will not export if a directory of the same name already exists. Then we do an SVN checkout of the code to the current directory. Then using the previous bash variable we created by parsing the passwd file ($FOO) we correct the ownership of the httpdocs folder and the files contained within.

Simple huh?

I hope to write up a simple howto in the near future on how to manage your website using SVN. It is extremely handy when dealing with website revisions as well as revisions to many sites that use the same code.

OpenSSH has some great programs that come bundled with it. One specifically I use on a regular basis is scp which stands for Secure CoPy. And it does exactly what its name says it does. It copies a file securely via an SSH tunnel between two machines (that both have SSH running).

You can use scp to copy a file from the a local directory to a remote directory, or a remote directory to a local directory, or even from a remote directory to another remote directory, which is pretty handy for the lazy.

Here is an example of how to use scp to copy a local file to a remote server:

scp ./somelocal.tar.gz remoteuser@some.remote.host:/path/to/where/you/want/the/file

Pretty handy… Now here is how you do a remote file to a local directory using scp:

scp remoteuser@some.remote.host:/path/to/file/filename.tar.gz ./

Simple. Now lets go a step further… Remote to remote file copy with scp:

scp someuser@some.remote.host:/path/to/file/filename.tar.gz anotheruser@another.remote.host:/path/to/where/you/want/the/file/

SSH is more than just a replacement for Telnet, it is a mutlipurpose suite of tools that will make your life easier once you become familiar with them.

I have a customer that I am supposed to be migrating from one server to another server due to their current server being hacked (PHPBB hack… it really is sad how easy it is to do this… and even more so how easy it is to prevent it).

Anyways, I attempted to migrate the server earlier this week, but the install of the latest version of Ensim on the new server went horrifically wrong (in case you are wonder Ensim is the WORST control panel software for webhosting ever created. It is slow, poorly documented, poorly supported, and the company uses all of these facts as a profit center to charge you $150 per hour for anything that goes wrong in their shit hole software — but I digress.)

Long story short, I spent about 8 hours on trying to get Ensim to install after the previous install went down in a giant flaming fireball of crap and python exception tracebacks. I threw in the towel and told one of the other techs to reimage the drive and give me another server…

First there was a problem with the drive images, so they had to be corrected. Then they took the servers to the data center and put them online… only nobody bothered to check to see if they were actually online or not… and of course… they weren’t. So…

They finally got the servers online today… Cool! I am getting ready to get started on my Ensim install, but I am going to take every precaution known to man to prevent the massive shit storm that happened last time. So I set the hostname of the server, setup the hosts file (/etc/hosts) to point to the local IP address for the server. I even threw a reboot at the server, just to make sure everything was all set. So I reboot the server and wait. And wait. And wait… The damn thing never came back online. I am furious.

I tried everything I could think of, accessing it via the router, neighboring servers, changing the vlan of the ports to see if it was a routing issue. NOTHING.

I would hard reboot the server is somebody had bothered to tell me which power switch port the server was on, but I don’t even have that. So here I sit… 3:00am in the morning. With my proverbial thumb up my ass.

Okay, well we will give this another try tomorrow, but for now bed is calling.