HostingGeek

I am going to vent a little here.

I don’t understand why web hosting companies (and dedicated server providers for that matter) continue to provide Fedora Core based servers to customers. It just doesn’t make sense. Fedora Core was never intended to be used in a production enviroment and why people keep using in those enviroments baffles me.

We had a server that was running Fedora Core 4 and cPanel and has been in production for almost a year. It was always updated and properly maintained… that is until RedHat dropped updates for it (which they do every 6 months as that is the development cycle for Fedora). Dropping updates isn’t that big of a deal as there is the Fedora Legacy Project that continues to provide updates after RedHat has dropped updates.

Here is what happened with this server, Fedora Core 4 was dropped from RedHat support, and was being taken over by Fedora Legacy. No big deal, this has happened a bunch of times in the past with previous Fedora Core versions. The problem enters when you throw in a local kernel exploit that is discovered for the /proc fs. We attempted to upgrade this kernel to prevent this problem, but because FC4 was in a state of transition between the two providers, the updates weren’t available yet. As such this server was put into the “holding for update” queue.

In that time the server was hacked and needed to be wiped and reinstalled with another OS that won’t cause this particular problem (CentOS 4.3).

What we recommend to customers instead of Fedora Core?

CentOS: RedHat Enterprise Linux based OS. They grab the source RPM files from RedHat and remove the images that reference RedHat and replace them with CentOS based images, and other minor modifications, but generally this is the same code that is used to run RedHat Enterprise Linux. Because it is based on RHEL it has a good development cycle as well as excellent support for new drivers. Does it have it’s problems? Yes, any problem that exists in RHEL will also exist in CentOS (various spin_lock kernel panic issues have been documented).

We recommend the use of CentOS for any application that does not support our number one OS of choice.

Debian (stable): I admit it. I am a big fan of Debian. It works great and I am very familair with it. It handles in place version upgrades (moving from woody to sarge was done with minimal effort using apt-get dist-upgrade). It has an easy to use package manager, with a large selection of applications, as well as an active developer community. My biggest problem with Debian is the that the kernel gets out of date very quickly. Unlike CentOS which actually backports drivers from a more current kernel revision (for instance 2.6.16) into the current standardized kernel for the OS (2.6.9 for CentOS 4.3), Debian does not do that. So it is often a chore to get Debian to run (or even install) on newer chipsets.

Overall, Debian is the winner for us. We install it anytime we have the option (even for Virtual Private Servers) as it just works. It has its faults and is by no means perfect, but we are willing to over look those as they are minor in comparison to the benefits it can provide in ease of management and administration.

So to recap:

• Don’t use Fedora Core for a production server, it will only cause you pain and suffering in the long run.
  
• Anyone that suggests you use Fedora Core in a production enviroment has not analyised the potential ramifications of that decision completely.
  
• There are many viable alternatives that are actively developed and maintained as well as providing up to date and current device driver support and security updates.

In conclusion:

Don’t use Fedora Core on a production machine.

Recently I have had to deal with some insecure PHP scripts on a couple of servers that have caused us some serious problems, as well as time, to recover from.

I am going to list some of the important steps you can take to protect your server and your site from insecure PHP applications. This will by no means a complete list of items to stop them, but this should help prevent most of the basic attacks (“script kiddie” attacks – where the attacker doesn’t have a great deal of skill and is depending on a preset scenario).

Most of these solutions will assume you have some level of control (root) over the server.

• Turn off allow_url_fopen: This modification will prevent the use of http://www.domain.com/somescript.php and ftp://www.domain.com/somescript.php from being used in include(), include_once(), require(), require_once(), as well as fopen(). This will prevent a hacker from including malicious code remotely. This modification will prevent the vast majority of hack attempts in PHP from working.

  The downside of this modification is that some legitmate applications that use fopen() to open a remote web page might be broken. You can encourage users to use the PHP curl functions instead as they accomplish the same results.

  Edit your php.ini and change allow_url_fopen = On to allow_url_fopen = Off and restart your web server.

• Use open_basedir: open_basedir allows you to dictate which paths PHP is allowed to access for a given site. Generally we set this to the path of the website (/var/www/html), /tmp (for writting session data as well as for file uploads), and the path to the PEAR repository.

  The good thing about open_basedir is that it is default deny, meaning if you don’t specify the path in the open_basedir it is blocked.

  The bad part about open_basedir is that it come sometimes block access to legitmate applications needed by your PHP applications (Gallery is one that comes to mind, as it makes use of some external applications in /usr/bin, so you must open access to that location).

  Generally, you should use this directive for every website you host as it allows you to control which directories PHP can access and you can make sure that those directories have the correct permissions to prevent potential exploitation.

• Mount /tmp noexec, nosuid and nodev: This one is particularly useful as it allows you to tell the operating system not to run any applications in a given area of the hard drive. I can tell you personally that this one as saved me numerous times. When used in combintation with the open_basedir directive you can effectively limit what the hackers have access to, and what they are able to run.

  If you have a seperate /tmp partition on your hard drive you can edit the /etc/fstab file and change the options for the /tmp line.

  /dev/hda3 /tmp ext3 noexec,nosuid,nodev 0 0

  If you don’t have a seperate /tmp partition you can create one as a loopback filesystem and mount that as noexec. I will make a seperate post on how to do this later.

• Mount your web space noexec: This might not be possible, but is a great option just like mounting /tmp in noexec. The reason for this is to prevent binary applications from being uploaded and executed in your webspace. Like I said this might not be possible, but is an excellent prevention method to prevent local root exploit binaries (where a binary exploits a problem in the kernel to gain escallated priviledges on the server, the /proc race in the 2.6 kernel is a recent example of this type of exploit).

• Make sure your permissions are correct: You should not allow world write permissions on any files, other than those required (configuration files that are written by the web server, etc). This is very important as it will save you a great deal of trouble in the event that a hacker does gain access to your site, as they won’t be able to overwrite your files. Additionally, you should only allow write access to specific folders where needed.

• Prevent PHP from parsing scripts in world writtable directories: This will prevent hackers from uploading malicious scripts and running them from your site. This is a great way to tighten security, and is pretty easy to manage, just set “php_admin_flag engine off” for any directory that can be written to by the web server user. This one will save you more hassle than you can ever imagine.

• Install mod_security: Mod Security is basically a web server firewall, allowing you to block specific types of requests, as well as inspect data as it is sent to the server. My only problem with mod_security is that it isn’t default deny, which means you must identify what is blocked, rather than identifying what is allowed and rejecting everything else. Still it has its uses and you should have it installed in order to react to problems that do not have patches available yet.

We hope that this list of items will help protect your server from hackers and allow you to have some peace of mind that your sites are as secure as you can make them.

What fun things did I learn today? Let me see…

• Intel based ICH7 chipsets are not fully supported in the linux kernel until kernel version 2.6.11. With additional features added in 2.6.15.
• The Debian Installer is based on the 2.6.8 kernel, and therefore can not install on to hard drives attached to the ICH7 chipset (at least in SATA2 mode, which is the whole point).
• The northbridge chipset on the MSI 945GM2 motherboard is OMG HOT! (I have the burn marks to prove it.)
• Motherbards are moving to a new auxillary power connector (in addition to the ATXv2.0 spec). This new connector is called an EPS12V connector and consists of an 8 pin connector for 12V power to the CPUs.
• The EPS12V power connector seems to be standard on most “Cedar Mill” (Intel’s new 65nm fab CPUs) “Ready” motherboards. Where as most power supplies do not yet have this connector.
• The case I buy do not have the EPS12V connector on them – which made me sad.
• The standard 4 pin 12V connector that is standard now on most power supplies DOES work in the EPS12V plug – which make me happy.
• SATA2 (SATA 3.0Gbps) can be used in “comptability mode” in the BIOS, but it makes them SLOW.
• The default kernel installed with Debian doesn’t support over 1GB of RAM. You must install another kernel if you have more than 1GB of RAM.

I think that about covers what I ran into today. Just some random facts that might help somebody some day.

Let’s face it, moving from one server to another server is nasty and something any of us would try to avoid as often as we can. Unfortunetly, there are instances where we can not avoid this and we must bite the bullet and do it.

Here are some tips that I have gathered up that will help you maintain your sanity as much as possible. Honestly even with all the planning and preperation, there will be issues so just prepare yourself mentally for that. It will happen.

So here are some things you can do to help the transition:

• DNS Caching: DNS has to be one of the most difficult things to deal with, because you only have a certain level of control over what occurs. Caching of DNS results will often cause many problems. The trick here is to plan ahead and remove the caching from the picture until the move is completed.

  Setting your DNS to a low TTL (I use 5 seconds versus the recommended 86400 seconds/ 24 hours). The TTL tells other DNS servers and clients how long they should hold the information they just received, in their cache. Since we are going to be moving the site to a new IP soon, we don’t want them caching the old IP once the site is on the new IP. The low TTL will help prevent this.

• Pathing: If you are moving to a server with different paths, you should at least try to setup symbolic links to help any hard coded paths in the site’s content.

• PHP register_globals: This one has bitten me a couple of times. Most ofthen when you move to a new server the server will have the default settings for PHP which turns off register_globals. This will cause all sorts of problems that you won’t be able to track down easily. Check it early on and make sure that both servers match settings wise. (A a general rule register_globals should be set to off, but there are some legacy PHP apps that need it.)

• SSH Tunnels: Most of the time you should adjust your TTL for your DNS long before you do the actual move, but sometimes that isn’t an option. In a pinch you can use an SSL tunnel to redirect traffic from your old server to the new server. This will help users that have the old IP cached, still access the new site. It should be noted that SSH tunnels are pretty unstable and shouldn’t be really be depended on, but like I said, they are good in a pinch. Here is an example of one, you run this on the old server as root with the httpd process stopped:

  ssh -g -C -N -f -L 80:newserverip:80 root@newserverip

  This will prompt you for a password (if you don’t have key based access already setup). You will get a warning ‘bind: Address already in use’, but the process is working. This message is just letting you know that the remote endof the tunnel has something already on port 80, which is actually a good thing.

  You can read more about this process in my blog post Redirecting TCP Ports with SSH

• DNS: Once you have everything moved over to the new server, update the current authoritative DNS servers with the new IP as soon as possible. Once you have done that change the domains name server records also. The name server change takes longer to complete (will normally be done in 8 hours at the root servers).

• How To Deal With a Highly Dynamic Website: For sites that have alot of content, or for sites that have users contributing data often (mainly forums) you need to draw a line in the sand and take the site offline during the move. This ensures that you have all the content and that no one loses any content that was contributed during the move.

• Rsync: Also in the same vein as dynamic sites (user images etc) rsync is your best friend. Run rsync, and run it often. Run it even before you move. It keeps the two sites in sync with each other and can be run while the old site is still up and running. The more you run rsync leading up to the actual move, the faster the actual move will occur. Faster move == less downtime for your users.
  

There are many other issues to consider, but if you take these items into consideration you should not have any major surprises coming your way.

So today I came across this site for Apple’s Boot Camp.

Basically the concept behind Apple’s Boot Camp is to allow users to install Windows XP side by side with MacOSX, which many people have been trying to do since the first Mac’s shipped with Intel based processors.

So now it is possible to run Windows XP on a Mac with Apple’s blessings.

Here is where things get interesting: Throw Xen 3.0 into the mix. For those that don’t know what Xen is, it is a hardware virtualization technolgoy (based ont he linux kernel) that allows multiple OS’es to run on the same hardware at the same time. It allows concurrent access to the network, hard drives, memory, cpu etc. Prior to Xen 3.0 you needed to have an OS specifically compiled for Xen in order to run it. That has all changed with the new technology included in the latest processors from Intel (and soon AMD – June 2006 I think). The new processors have a technolgoy called Virtualization that allows hardware virtualization applications like Xen (and VMWare) to run OS’es that are not compiled natively for Xen itself (such as OSX and Windows XP).

This is where I think there is a huge opportunity for Apple (and for all I know they might actually be working on this already, I haven’t a clue). To allow Xen to boot OSX as a Xen Dom0 (the OS that is allowed access to the Xen kernel for control over the other OS’es on the virtualized machine). Then boot Windows XP as a DomU. Then a user would be able to boot their computer, access OSX and switch to Windows XP all in one machine.

An interesting thought… If you can get both OS’es to boot at turn on, and used OSX as the Dom0, then you would be able to access the Windows XP virtual machine via the Mac OSX RDP (Remote Desktop Protocol) client and never have to actually leave OSX or figure out how to switch between the two. Oh man… my head spins… I need to get a MacBook Pro and start working on that.

Today was a fun day. I have a buddy that has spent the last 3 or 4 months (maybe longer) working on a complete ground up redesign of a PHP/MySQL website. The current code base was horrid, and that is putting it lightly. The code was so poor that it was hampering the overall performance of the server. The pages were very slow to display, often taking 30-45 seconds to complete the rendering of one page. Looking at the top process list I could see that several httpd processes were dying off, and dying off frequently. I had assumed that this was due to the poor codebase of the existing application.

The launched the new codebase today, and things didn’t improve. So my buddy called on me to help him out. First a little bit of background on the server: Dual Athlon (yes it was weird to me too), 1GB of RAM, and 1 IDE HD (again, yes I know who ever speced out this server should have their head examined). Server is running Cpanel 10.8.1-C112 under Fedora Core 2. We had rebuilt apache already (using easyapache) and the httpd zombies kept occuring. They weren’t true zombies, only momentary zombies, because the child process was dying without permission from the parent process, so they only showed up in the zombie count for a couple of seconds, so they were being cleaned up properly.

Here is what we found: the main site for the server is a heavily trafficed PHP/MySQL based site with a lot of images. I started looking through the error logs and came across a bunch of messages like:

[notice] child pid 10009 exit signal File size limit exceeded (25)

After some quick googling we were able to determine that this message means that the child httpd process died, because it encountered a file that is too large for it to handle. The max file size is 2GB in this instance. There is some discussion on if this is an Apache 1.3.x limitation or a limitation a little further down (glibc or filesystem).

How did we fix it? Well short term was to shutdown apache and move the offending log file out of the way, and restart apache. Longer term fixes involve some settings inside Cpanel itself. You basically need to tell Cpanel to remove the log files after it has processed the log files.

Log in to WHM and click on “Tweak Settings”, then check mark “Delete each domain’s access logs after stats run”

That will delete the log files daily, and all should be well with the world.

Recently I did a Xen 3.0 install on a Dell 1750 server with RAID5. By default the Xen 3.0 installation (from source) doesn’t compile in the LSI drivers for the RAID card, so I had to enable the option in the Dom0 kernel in Xen.

So I went in to the xen0 directory under my Xen source (/usr/src/xen-3.0.0/linux2.6.12-xen0) and ran:

make ARCH=xen menuconfig

I then went to Device Drivers -> SCSI Device Support -> SCSI low-level drivers -> and enabled the “LSI Logic New Generation RAID Device Drivers”, because after all… Newer is better right?

I recompiled Xen and the Xen kernels (cd ../; make kernels; make install) and reboot the machine and went on with my day.

I got a call from the client later in the day that the server was reporting a problem with the RAID array. Long stroy short, the customer was running a NAGIOS plugin to monitor the RAID array via the /proc/megaraid entry. Little did I know that the new generation RAID device drivers in the 2.6 kernel no longer make use of the /proc/megaraid proc entry, so it was breaking the NAGIOS plugin.

To correct this problem use the “LSI Logic Legacy MegaRAID driver” instead of the “LSI Logic New Generation RAID Device Drivers” in the 2.6 kernel (this is not an option in the 2.4 kernel). That will give you back the /proc/megaraid entry and you should be all set.

Additionally, since you are already compiling a kernel, you should enable the kernel option “Use register arguments” under “X86 Processor Configuration”. This option will allow you to use the Dell OMSA device drivers for additional control over your Dell PowerEdge server. You can read more about it here: http://www.uta.fi/~pauli.borodulin/dellomsa/omsa44.html

I have many servers that I have to watch on a regular basis. We use a KVM to switch between the consoles of the servers and generally we only see the console for any given server for a couple of seconds.

The KVM we use has this great feature where if it detects any activity on the console, the KVM will swtich to that console and display the output for us.

This is how we use that functionality to quickly display any problems a server might be having. It is by no means a fool proof and accruate, but it does help to bring things to our attention on a regular basis, doesn’t require you to install anything special (all programs should come with a basic install of a linux OS), and its cool… What else do you really need?

What we do is one turn off the console’s screen saver. It is evil, and we don’t like it. Actually it serves a valid purpose, but not for our needs so lets turn it off:

/usr/bin/setterm -blank 0

Now you can put that into /etc/rc.local for RedHat based distros (Fedora, Redhat Enterprise Linux, CentOS, etc).

For Debian you can create a file in /etc/init.d/ and name it blank_screen and chmod 755 the file. Then just create a symbolic link to the startup dir:

ln -s /etc/init.d/screen_blank /etc/rcS.d/S61blank_screen

Now that we have the console set not to turn off, we can do the next part of our monitoring mojo. We will setup to monitor files that are important to us, such as /var/log/messages, and /var/log/syslog. These files might not be the files you want to monitor, so you can modify the line below to match the files of your choosing.

We use tail, because it displays the lines as they are added to the file we are monitoring, and it is simple and lightweight.

/usr/bin/tail -v –follow=name /var/log/messages –follow=name /var/log/syslog > /dev/tty2

What this does is runs tail to monitor the files /var/log/messages and /var/log/syslog. We monitor the filename, rather than the descriptor (which is the default behavior of tail). Monitoring the descriptor means that if the filename is changed — for example logrotate, then we will follow the file to the new name. Since once a file is rotated by logrotate it won’t be updated, we don’t want this behavior. We add the -v (verbose) so that tail will tell us the name of the file that corresponds with the output. Then we redirect the ouput to /dev/tty2 (virtual console 2).

This will output the information directly on the screen so that we can see it via the KVM.

Additionally you might want to turn off the login prompt on the virtual console 2. We can do this by editing the /etc/inittab file and commenting out the following line:

2:23:respawn:/sbin/getty 38400 tty2

You can just put a # in front of that line and then have init re-examine the file by running:

kill -HUP 1

So there you go… you now have the output of any file you want to the console of your server. Set your KVM to autoscan, and enjoy your easy to view logs.

Every once in a while I come across the need to access data that is stored in a database inside MySQL from a shell script. Here is a quick and dirty way to access that information without having to resort to PHP or Perl.

Run a SELECT:

By default this would return output like:

id
514

So we need to use sed to remove the name of the field.

SITEID=$(mysql -D database -u root –password=password –silent –exec=”SELECT id FROM domains WHERE name = ‘`echo $SITE`’;” |sed ‘s/id//’)

Since INSERTs, UPDATEs and DELETEs don’t return any information you won’t need to pipe the information returned from MySQL to sed.

This handy tip will work for basic MySQL queries, but for more complex queries, I would recommend something with a little more control over the information such as PHP or Perl.

I hope that helps somebody.

Let me introduce you to my new favorite piece of open source software: Xen.

Xen is a virtualization operating system that allows you to share hardware between multiple OS’es. The concept isn’t really new, as there are several commercial as well as open source implementations of this concept already.

In the past I have used Linux-Vserver which is a set of kernel patches that create security contexts inside the kernel that different versions of Linux can co-exist in. It works very well and I have been using it for over 2 years in production. The main drawback is that it only works with Linux based OS’es. The same with Jail in FreeBSD which I have also used for a while, but again is specific to the OS, in this case FreeBSD.

Now I have never used VMWare, but VMWare uses a “core” application that the guest OS’es run in. This works very well with all current OS’es from Windows, Linux to *BSD. I have heard that there is some performance degradation with this method, but I have never used it so I don’t really know for certain.

Now enter Xen. Xen works much the same way as VMWare, but is more lightweight. Xen loads up as a base kernel that in turn loads a Dom0 kernel. The Dom0 is the OS that runs with higher priviledges than any other guest OS on the system. The purpose of the Dom0 is to allow an interface between the guest OS’es and the Xen kernel (to start and stop guest OS’es, create new ones, etc).

The biggest limitation for Xen currently is that in order to run under Xen, an OS must be compiled to run under Xen. Currently there are hardware compile options for Linux (patchset), FreeBSD, and NetBSD (both support Xen as a native hardware architecture, like i386, or PowerPC). You will notice that Windows is missing from this list, and that is because while Xen will run Windows as a guest OS, there is currently no way the average Joe can get access to the code to Windows to make the changes necessary to get Windows to run under Xen. The Xen guys have gotten Windows to run under Xen by modifying the Windows source, and I am betting by signing NDAs and giving away their first born children, so we know that Windows will work under Xen with some modifications.

That is the basics of Xen. Now we turn an eye to the future and find out what is on the horizon.

Here is where things get interesting, both Intel and AMD are planning on offering a new technology into their CPUs that supports virtualization. Intel calls theirs VT (Virtualization Technology – catchy right?). Currently (as of the last processor table guide from Intel) two processors support the VT technology, the Pentium 4 662 and 672. I am assuming (using my secret decoder ring) that the XX2 designator will indicate the presence of VT for the chip. I have looked around, but I haven’t found a single place that sells these processors yet so who knows how much they will cost.

I don’t have any details on AMDs VT offering yet.

So what is so special about VT? Well Xen 3.0 (released December 2005) supports this new feature (VmWare also supports this feature). With support for VT technology it is now possible to run guest OS’es in Xen that are not compiled for the Xen architecture…. Hello Windows!

So lets stop and think about this for a minute, it will now be possible to purchase a fairly large server (lots of RAM and disk space) and run Linux, FreeBSD and yes, even Windows side by side by side.

It seems that this next innovation might just be a great way to improve the over all cost of operating multiple servers with multiple OS’es. Only time, and probably usability tools will tell.

Update: I want to thank Karsten Kruse for quoting me in an article done on installing XEN under NetBSD. If you want to know how to install XEN on NetBSD you can read Karsten’s post about how to do it here.