First of all I want to wish everyone a very Happy New Year. It has been a while since I made my last posting and I want to apologize for that.
Now let’s get down to business…
So recently we discovered that a Windows 2003 server had been exploited via an apparently well known 0day exploit in MailEnable’s SMTP service. This has since been corrected by the MailEnable developers (you can read about that here).
At first there was some doubt as to whether or not there was a hacker on the server. Our first clue was the abnormal amount of traffic the server was doing. Typically this server moved about 100-200KBps per day. We knew something was up when this server started moving 2MBps. Upon inspection of the server we couldn’t see anything out of the ordinary, however we did notice that taskman.exe (Task Manager) was running at 100% CPU utilization when ever we looked at it. This threw up all sorts of red flags to us, and we knew that we had a hacker on the server and we needed to find out what they were doing.
I have to say that Event Viewer is your friend. You must look at it every once in a while to make sure you know what is going on. Even with a hacker on the server and rootkit installed on the server to hide his activity, he still wasn’t able to hide some log entries in Event Viewer. After looking in Event Viewer we noticed several references to VMWare. After asking around, we determined that none of the legitimate administrators had installed VMWare and we knew that this must be the hacker.
Here is how we found him, and how we removed him.
We could see that there were some hidden directories on the server that we couldn’t access through the normal Explorer interface, so we knew we were dealing with an on boot rootkit. (You can see file accesses using the file system and disk tools from Windows SysInternals Tools.)
We installed HiJack Free from a-squared. This piece of software is pretty powerful and does some deep inspections of the registry to find services and applications that are not normally visible in the Control Panel services listing. We sorted the services and looked for services that were set to start at boot up, and looked for anything that wasn’t signed. Hi-Jack Free displays the company that signed the driver/service, and any service that was set to run, but wasn’t signed was on our hit list to remove.
With a list of services to disable, we installed the Windows Recovery Console and rebooted the server into the recovery console. We disabled the services that we identified as problems and rebooted the system normally.
At this point we could see the directories that were hidden from us earlier. We discovered that the hacker had installed VMWare. Because we wanted to see what the hacker was actually doing with the VMWare installation, we used Virtual Disk Driver to mount the WMWare disk images to see what they were doing. Turns out they were downloading Pokemon episodes. Ugh… All that hassle and it wasn’t even anything good. 🙂
So we removed the rootkit from the server, removed the VMWare installation, and patched the MailEnable install, and the server has been cruising along ever since.
We hope that this description of what we did will help you find and remove Windows rootkits on your servers.