Is there a business case in planning for data breaches?

When I was learning to fly, one of the many pearls of wisdom imparted to me by my instructor was, as I transitioned from pre-flight planning and considering a myriad of “what-if” scenarios to prevent problems, to actually going aloft was to mentally move to continually considering what to do “when” an event, such as an failure, eventually takes place.  The primary objective remained constant: to ensure a safe outcome with minimal consequences (you may call it applied risk management).  This shift in attitude appears to be apt for custodians of information systems, moving from planning services and incident prevention to operational preparedness in order to best ensure a successful outcome in the event of an unplanned incident.    Sadly, even with sophisticated layers of defense, many organizations are facing similar thought processes of what to do “when” a data breach takes place rather than “if”.  Staples looks like it is the next addition to the list of notable incidents that includes Target, Home Depot, Chase, Goodwill, Michaels and P.F. Chang’s.

The recent Ponemon Institute benchmark research “2014 Cost of Data Breach Study : United States” identified a number of factors that could materially affect the impact and cost of managing a data breach. Apart from the headline average cost of an incident of $5.4 million with a per record number rising to $201 there were some interesting observations relating to the root causes.

The involvement of a third party was one of the biggest contributors to the cost of managing a data breach, at 12.5% above the mean cost.   There is ample indication that this is an extremely common situation that is developing rapidly with the adoption of computing and application services .  As well as the HVAC issue that was a vector for the Target breach, incidents at Lowe’s, Goodwill and AutoNation earlier this year were attributed to third-party vendors (E-DriverFile, C&K Systems and Trademotion respectively).  The need for third party diligence has been identified as necessary by financial and healthcare regulators. If we look at the potential for loss avoidance, effective vendor security management that includes incident management makes good sense as both preventative and response measures.

The maturity of breach response plan represented another interesting opportunity to either increase or reduce the cost of a breach.  Typically organizations that provided quick, less coordinated announcements and response activities that did not follow a clear protocol experienced management costs 7% above the mean.   On the other hand, those with a clear incident response plan reported average costs around 8.5% below the mean.  The difference in response approach represents over $830,000 in a $5.4 million event.  To return to the pilot analogy: preparedness training and the effective use of checklists have been proven to significantly improve the outcomes.

MPTCP Roams Free (By Default!) – OS X Yosemite

Further to the BlackHat USA Work by Patrick Thomas (@coffeetocode) and I (@secvalve).

MPTCP is enabled by default in Mac OS X Yosemite. So we can expect to see Multipath TCP on most networks, and on a total of tens to hundreds of millions of devices.

Embedded image permalink

Thanks to Ilias Marinos (@marinosi) who tripped my twitter search bot 

More to come…. We have stuff as yet unreleased that has suddenly become VERY relevant.

Shellshock bug exposes web servers, home routers

With Shellshock, the recently discovered vulnerability in Bash yet to hit full stride in exploitation, there are numerous systems and devices that are immediately at significant risk of exploitation across the Internet.  Neohapsis Labs has released advance Shellshock guidance to our clients, including immediate considerations, and short and medium term remediation steps to mitigate the impact of the vulnerability. What follows is a guest post by industry acclaimed journalist Byron Acohido on the immediate impact of this vulnerability:

Shellshock bug exposes web servers, home routers

By Byron Acohido

Yes, you should be very concerned about Shellshock, the latest software bug to arise with the potential to degrade the overall safety of the Internet by several notches.

Shellshock, also referred to as Bash, is a glaring weakness in an otherwise innocuous bit of coding that’s been around since 1987. Bash, shorthand for Bourne-Again Shell, is a program that allows you to type commands on computing devices that use the Unix, Linux, Apple Mac and Android operating systems. You’ve encountered  Bash if you’ve ever typed text commands on the black screen sitting behind the graphical interface of your computing device.

The existence of the Shellshock flaw was made public on Tuesday, riveting the attention of the global security community. It’s almost certain elite hacking groups have been aware of the vulnerability for some time prior, and have been taking advantage.

And now the rest of the cyber underground can make hay. The mad scramble is on. Much as they did earlier this year upon disclosure of the Heartbleed bug, companies of all sizes must identify and patch systems exposed to the Shellshock flaw.

“It’s at least equal to Heartbleed for sure,” says Garve Hays, software architect at NetIQ. “Heartbleed was ephemeral, you could gather personal data and move on. But with Shellshock, you can plant a backdoor on a server and stay there for years. It’s the gift that keeps on giving.”

Apache servers targeted

White hat researchers have begun probes to find vulnerable systems. The biggest, most obvious targets are Apache web servers. These Linux-based machines are used to run about one-third of the websites on the Internet.

“System administrators will be working long shifts to go through every single server, router and other piece of equipment that uses the bash shell,” says Jerome Segura, senior security researcher atMalwarebytes Labs.

Big enterprises have the resources and motivation to expedite patching. But hundreds of thousands of small and medium sized businesses will be slow to patch, or never patch at all. In the meantime, every hacker from script kiddies to spammers to account hijackers can now do simple reconnaissance to find and infect unpatched Apache web servers and related networking equipment.

“It’s a race against time,” Segura says. “The bad guys are hard at work trying to hack into affected systems.”

That’s just the first wave. Another obvious target that hacking gangs surely will triangulate are the routers used in home networks and small businesses. Up until about two years ago, Bash was widely used in Linksys, Belkin and many other brands of consumer-grade routers, says NetIQ’s Hays.

Home routers ripe for attack

So if you’re using an older router in a home or small business setting, you should assume the bad guys will soon turn their attention towards seeking out your unpatched router and taking control of it – because it’s simple and profitable to do so.

Bash also comes into play on certain Apple Macs, and certain older versions of Android handsets. Apple issued a statement saying the majority of Mac OSX users are safe from bash exploits.

Even so, it will take some time to determine the full extent of the ramifications of this flaw, says Dr. Mike Lloyd, CTO of RedSeal Networks.

“It’s relatively easy to tell whether the flaw is present, but it’s hard to tell if it’s reachable,” Lloyd says. “The maze of software and configuration interactions is too complicated for a human analyst to be able to say categorically ‘the cheese is exposed, or is not exposed, to the rat.’”

So what can individuals and companies do? Pay close heed to patches and get them installed. Tools are readily available to check whether your network is using a vulnerable version of Bash, says Jeff Schilling, CSO at FireHost.

Advised Schilling: “Step one is to figure out if you have any systems that are vulnerable. If so, how many? Step two is to figure out how to put a compensating control in place to buy time to wait for a patch. Step three, patch your systems in a methodical manner to ensure your most important servers are fixed first.”

More on emerging best practices

3 steps for figuring out if your business is secure

Encryption rules ease retailers’ burden

Tracking privileged accounts can thwart hackers

Impenetrable encryption locks down Internet of Things

Shellshock without the Shellac

A post by our exploit-herder in residence, Jason Royes

The Problem

Have you heard about Shellshock? If not, you may be living under a rock. To summarize:

If an application sets an environment variable name or value to a value that is derived from user input and subsequently executes bash (and possibly other shells), an attacker may be able to execute arbitrary code.


When I first read the post from Robert Graham, my first thought was: “when did we begin storing function definitions in environment variables?” I scanned through the section of the bash manual dedicated to environment variables and could not find anything on the topic.

I knew I was not alone after googling and finding this on Stack Overflow. Luckily, I had an old VM handy that I never update.

Here’s bash:

$ bash --version
GNU bash, version 4.2.24(1)-release (i686-pc-linux-gnu)

So, according to the stack overflow article, what’s actually going on is that bash stores exported functions in the environment.

$ f1
f1: command not found

Let us create a file that will define a function and export it:

$ cat
#! /bin/bash

f1() {
echo "in f1"

export -f f1

Now to include it:

$ source

Voila, f1 is now defined within the shell environment.

$ env|grep -A1 f1
f1=() {  echo "in f1"

If you’ve already read about the Shellshock attack, the value of f1 above should look familiar.

Bash 4.2 and Exported Functions

Bash 4.2 (vulnerable) processes environment variables in initialize_shell_variables (see variables.c). What happens when an environment variable has a value that begins with “() {“? A new buffer is allocated and the variable name is concatenated with the variable’s value. This basically creates a normal bash function declaration. The concatenated string is then evaluated with parse_and_execute:

temp_string = (char *)xmalloc (3 + string_length + char_index);

strcpy (temp_string, name);
temp_string[char_index] = ' ';
strcpy (temp_string + char_index + 1, string);

parse_and_execute (temp_string, name, SEVAL_NONINT|SEVAL_NOHIST);

Imagine an exported function named f1 that has a value resembling “() { ls -l; }”. The code above combines the name and value into temp_string, resulting in “f1() { ls -l; }”. This string is then evaluated and a function definition is burnt in memory.

The vulnerability arises because user input is being evaluated directly with the same function used to evaluate all other bash commands. If commands are appended to the end of the function definition, ex. “() { ls -l; }; ps”, they are executed. This is because they fall outside the bounds of the function declaration and so are treated just like they would be in a regular bash script. Note that anything inside the function declaration should not be executed unless the function is invoked.

The construction of temp_string also means an attacker can inject through the environment variable name. For example:

$ ./
total 6868
drwxrwxr-x 12 user1 user1    4096 Feb 13 17:28 bash-4.2
-rw-rw-r--  1 user1 user1 7009201 Feb 13  2011 bash-4.2.tar.gz
-rw-rw-r--  1 user1 user1      52 Feb 13 16:19
-rw-rw-r--  1 user1 user1      49 Feb 13 16:47
-rwxrwxr-x  1 user1 user1     101 Feb 13 17:30
-rwxrwxr-x  1 user1 user1      96 Feb 13 16:58
Segmentation fault

Whoops! Bonus segfault. Here’s

#! /usr/bin/python
import os

os.putenv('ls -l;a', '() { echo "in f2"; };')
os.system('bash -c f2')

Bash 4.3 and Exported Functions

The bash patch seems fairly concise. The patch now includes a check to make sure the variable name only contains legal characters (thwarting injection through name). There’s also a new flag called SEVAL_FUNCDEF. If parse_and_execute parses a command that is not a function definition and this flag is set, an error condition results.

This seems to correct the issue, however, relying on the function parsing code still feels dicey.

Perhaps there are other ways around these new defenses yet to be revealed.

Blackhat USA Multipath TCP Tool Release & Audience Challenge

We hope everyone found something interesting in our talk today on Multipath TCP.

We’ve posted the tools and documents mentioned in the talk at:

Update: We’ve now also added the slides from the talk.

At the end we invited participants to explore MPTCP in a little more depth via a PCAP challenge.

Without further ado, here’s the PCAP: neohapsis_mptcp_challenge.pcapng

It’s a simple scenario: one MPTCP-capable machine sending data to another. The challenge is “simply” to reassemble and recover the original data. The data itself is not complex so you should be able to tell if you’re on the right track, but getting it exactly right will require some understanding of how MPTCP works.

If you think you have it, tweet us and follow us (@secvalve and @coffeetocode) and we’ll PM you to check your solution. You can also ask for questions/clarifications on twitter; use #BHMPTCP so others can follow along. Winner snags a $100 Amazon gift card!

Hints #0:

  • The latest version of Wireshark supports decoding mptcp options (see “tcp.options.mptcp”).
  • The scapy version in the git repo is based on Nicolas Maitre’s and supports decoding mptcp options. It will help although you don’t strictly need it.
  • The is an mptcp option field to tell the receiver how a tcp packet fits into the overall logical mptcp data flow (what it is and how it works is an exercise for the user :) )
  • It’s possible to get close with techniques that don’t fully understand MPTCP (you’ll know you’re close). However the full solution should match exactly (we’ll use md5sum)

Depending on how people do and questions we get, we’ll update here with a few more hints tonight or tomorrow. Once we’ve got a winner, we’ll post the solution and code examples.

Update: Winners and Solution

We have some winners! Late last night @cozinuzo contacted us with a correct answer, and early this morning @darkfiberiru got it too.

The challenge was created using our fragmenter PoC tool, pushing to a netcat opened socket on an MPTCP-aware destination host:

python -n 9 --file=MPTCP.jpg --first_src_port 46548 -p 3000

The key to this exercise was to look at the mechanism that MPTCP uses to tell how a particular packet fits into the overall data flow. You can see that field in Wireshark as tcp.options.mptcp.dataseqno, or in mptcp-capable scapy as packet[TCPOption_MP].mptcp.dsn.


The mptcp-capable scapy in our mptcp-abuse git repo can easily do the reassembly across all the streams using this field.

Here’s the code (or as a Gist):

# Uses Nicolas Maitre's MPTCP-capable scapy impl, so that should be
# on the python path, or run this from a directory containing that "scapy" dir
from scapy.all import *

packets = rdpcap("pcaps/neohapsis_mptcp_challenge.pcap")
payload_packets = [p for p in packets if TCP in p
                   and p[IP].src in ("", "")
                   and TCPOption_MP in p
                   and p[TCPOption_MP].mptcp.subtype == 2
                   and Raw in p]

f = open("out.jpg", "w")
for p in sorted(payload_packets, key=lambda p: p[TCPOption_MP].mptcp.dsn):

These reassemble to create this image:


The md5sum for the image is 4aacab314ee1a7dc5d73a030067ae0f0, so you’ll know you’ve correctly put the stream back together if your file matches that.

Thanks to everyone who took a crack at it, discussed, and asked questions!

Rob Beck’s MS-SQL Rootkit Framework Presentation @ DefCon Skytalks 2014

SQL Gestalt: A MS-SQL Rootkit Framework will be presented by Rob “whitey” Beck (@damnit_whitey) at the DefCon Skytalks 2014 in Las Vegas, NV this year.  This talk will provide an overview of a basic framework for the creation, deployment, operation, and persistence of a MS-SQL rootkit for all versions of Microsoft SQL Server 2005 and above.


This talk illustrates the various facilities in the MS-SQL database environment for performing code execution.  Using these facilities, attendees are presented with the basis of the SQL Gestalt – A rootkit framework, utilizing various aspects of the SQL core facilities, working in conjunction to provide persistence in the database.


This talk benefits pen testers, forensic analysts, and database administrators by exposing methods and tactics that may not be commonly known or widely employed in traditional database compromises. Examples will be provided in a variety of languages including T-SQL, C#, C++, VBscript, and Powershell utilizing SQL facilities such as SQL Assemblies, the Extended Stored Procedure API, SQL Agent, and OLE Automation.  At the conclusion of this presentation a basic framework will be released with sample code to illustrate all of the functionality discussed in this talk.

Talk Agenda

The following topics will be discussed in the presentation:

  • Concept of the SQL Gestalt rootkit
  • Facilities for executable code in SQL
    • Overview
    • Advantages
    • Disadvantages
    • Examples
  • Module installation
    • Deployment
    • Execution considerations
  • Securing a native code execution point
  • Persistence in SQL
  • Advanced rootkit operations


Multipath TCP – BlackHat Briefings Teaser

Multipath TCP: Breaking Today’s networks with Tomorrow’s Protocols. is being presented at Blackhat USA this year by Me (Catherine Pearce @secvalve) as well as Patrick Thomas @coffeetocode. Here is a bit of a tease, it’s a couple of weeks out yet, but we’re really looking forward to it.

Come see us at Black Hat Briefings in South Seas AB, on Wednesday at 3:30pm.

(UPDATE 8/14: A followup post and the talk slides are now online.)

What is multipath TCP?

Multipath TCP is a backwards-compatible modification that allows a core networking protocol, TCP to talk over multiple paths at the same time. In short, Multipath TCP decouples TCP from a specific IP address, and it also allows you to add and remove network addresses on the fly.

Multipath TCP in brief

Multipath TCP splits connection data across N different TCP subflows



Why do I care?

MPTCP Changes things for security in a few key ways:

  • Breaks Traffic Inspection - If you’re inspecting traffic you need to be able to correlate and reassemble it. We haven’t found a single security technology which does so currently.
  • Changes network Trust models – Multipath TCP allows you to spread traffic around, and also remove the inherent trust you place in any single network provider. With MPTCP it becomes much harder for a single network provider to undetectably alter or sniff your traffic unless they collaborate with the other ones you are using for that connection.
  • Creates ambiguity about incoming and outgoing connections – The protocol allows a client to tell a server that it has another address which the server may connect back to. To a firewall that doesn’t understand MPTCP it looks like an outgoing connection.

MPTCP and Reverse connections

MPTCP can have outbound incoming connections!?



Backwards compatible

Did I mention that MPTCP is designed to be backwards compatible and runs on >= 85% of existing network infrastructure [How Hard Can It Be? Designing and Implementing a Deployable Multipath TCP ]

Like IPv6, this is a technology that will slowly appear in network devices and can cause serious security side effects if not understood and properly managed. MPTCP affects far more than addressing though, it also fundamentally changes how TCP traffic flows over networks.

MPTCP confuses your existing approaches and tools

If you don’t understand MPTCP, things get really confusing. Take this wireshark “follow TCP stream” where I follow an http connection. Why does the server reply to an invalid request this way?

MPTCP Fragmentation confuses wireshark

Why does the web server reply to this garbled message? – MPTCP Confuses even tools that support it


Network flows can also become a lot more complicated. Why talk over a single network path when you can talk through all possible paths?


That’s what your non MPTCP-aware flows look like.

But, if we are able to understand it then it makes a lot more sense:


What are the implications?

Technologies are changing, and multipath technologies look like a sure thing in a decade or two. But, security isn’t keeping up with the new challenges, let alone the new technologies.

  1. I can use MPTCP to break your IDS, DLP, and many application-layer security devices today.
  2. There are security implications in multipath communications that we cannot patch our existing tools to cope with, we need to change how we do things. Right now tools can correlate flows from different points on the network, but they are incapable of handling data when part of it flows down one path and part of it flows down another.

To illustrate point 2:

What if you saw this across two subflows… Can you work out what they should be?

  • Thquicown fox jps ov the az og
  • E k brumerlyd.

Highlight the text below to see what that reassembles to

[The quick brown fox jumps over the lazy dog.]

Follow up with our Black Hat session as we discuss MPTCP and the effect on security in yet more detail. We ma not be ready for the future, but it is fast approach, just ask Siri.

How does your security decide what to do with a random fragment of a communication?