Camouflage at encryption layer: domain fronting

In today’s post we are goint to talk about a somewhat old technique (although programs like Signal have recently started using it) that I have always found to be a really clever hack:
domain fronting.

For example, let’s take the IP address of the frontal that serves www.google.es:

$ host www.google.es
www.google.es has address 216.58.210.227

If we take a look at the Common Name (CN) field of the TLS certificate returned by the server:

$ echo | openssl s_client -CApath /usr/lib/ssl -connect 216.58.210.227:443 | openssl x509 -subject 
depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = google.com
verify return:1
DONE
subject= /C=US/ST=California/L=Mountain View/O=Google Inc/CN=google.com
-----BEGIN CERTIFICATE-----
MIIgWjCCH0KgAwIBAgIIHcanliJsJ+UwDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
[...]

We see that it’s not www.google.es. This certificate is authenticating a name that is not present in its CN.

The subcommand x509 (with which we can get more info about the cert) has the answer:

$ echo | openssl s_client -CApath /usr/lib/ssl -connect 216.58.210.227:443 | openssl x509 -text
[...]
        X509v3 extensions:
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:google.com, DNS:*.2mdn.net, DNS:*.android.com, DNS:*.appengine.google.com,
                DNS:*.au.doubleclick.net, DNS:*.cc-dt.com, DNS:*.cloud.google.com,
                DNS:*.de.doubleclick.net, DNS:*.doubleclick.com
[...]

The HTTPS server is using the Subject Alternative Name (SAN) extension, that makes it possible to serve more than one HTTPS website from the same IP address using just one X.509 certificate.

We can also use (HTTPS client side) another more recent TLS protocol extension called Server Name Indication (SNI) to request any of the virtual hosts available from the same IP address.
We can invoke the s_client subcommand with the -servername modifier to request any of the other vhosts available; for example, mail.google.com:

$ echo | openssl s_client -CApath /usr/lib/ssl \
  -connect 216.58.210.227:443 -servername mail.google.com | openssl x509 -subject 
depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = mail.google.com
verify return:1
DONE
subject= /C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
-----BEGIN CERTIFICATE-----
MIID1jCCAr6gAwIBAgIIGOTyaAwrvk4wDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
[...]

If we look at a capture of the previous interaction with Wireshark, we can clearly see the field of the SNI extension; the TLS client has told the server (in the ClientHello message of the TLS handshake) that it wants the virtual host of the GMail service:

In fact, in the case of Google (and this also happens with many other CDN networks and frontal servers, hence the name of the technique: domain fronting), it’s not even necessary to use the TLS SNI extension; it’s enough to point against the same IP address, but specifying the desired virtual host using the Host HTTP header:

$ wget -q -S --max-redirect=1 --header 'Host: mail.google.com' www.google.es
  HTTP/1.1 301 Moved Permanently
  Location: /mail/
  Content-Type: text/html; charset=UTF-8
[...]

Thus, as a result of the command above, someone who is monitoring the network just gets to see a request to www.google.es: the traffic of the interactions with mail.google.com stays inside, camouflaged in the TLS layer unless we break that layer with dynamic cert generation from a certification authority present in all workstations (even in this case, the MitM is going to have trouble with things like certificate pinning).

So, what’s the advantage? Yes, we all know that Google uses the same frontals for everything, so what? What we get by hiding GMail traffic inside generic Google searches? Well, if we base our command & control or exfiltration server on GMail, we have a direct advantage, but the thing goes beyond: Google App Engine (GAE) apps, hosted (among others) on *.appspot.com domains, are also served from the same frontals.

So in this particular case, it’s not only that we could deploy our C&C server on GMail web services (or Google Drive, or Google Docs…): it’s that we can hack it directly as a GAE app in myc2server.appspot.com, and network monitoring will not even see the DNS requests against that domain (the only registered traffic will be DNS requests for www.google.es and apparently legitimate, generic, boring Google search traffic).

And what is better: to dismantle our stuff, or Google detects somehow that we have an ilegitimate C&C server hosted on myc2server.appspot.com, or the network monitoring staff has to “break” HTTPS (or block all HTTPS traffic in general, or HTTPS access to all Google frontal servers in particular). This is not going to be good news for the company: no Google web search available? What!? :-) And this is the reason that this technique is so powerful: if you block the frontals, you block a lot of other useful services, and this has high collateral damage. It’s because of this that the technique is widely used for content filter and DPI bypass, and it’s used in Internet censorship circumvention tools such as Lantern, Psiphon, now Signal, and Tor pluggable transports such as Meek.

Without having to search too much :-) we can find numerous gadgets that people have deployed on The Internets…
For example, here we have a Twitter reflector:

$ wget -qO- https://3benben.appspot.com
    <html>
        <head>
        <title>GAE Twitter API Proxy</title>
        <link href='https://appengine.google.com/favicon.ico' rel='shortcut icon' type='image/x-icon' />
        <style>body { padding: 20px 40px; font-family: Verdana, Helvetica, Sans-Serif; font-size: medium; }</style>
        </head>
        <body><h2>GTAP v0.4.2 is running!</h2></p>
        <p><a href='/oauth/session'><img src='/static/sign-in-with-twitter.png' border='0'></a> <== Need Fuck GFW First!! 
        or <a href='/oauth/change'>change your key here</a></p>
        <p>This is a simple solution on Google App Engine which can proxy the HTTP request to twitter's official REST API url.</p>
        <p><font color='red'><b>Don't forget the "/" at the end of your api proxy address!!!.</b></font></p>
    </body></html>

With it we can visit the SecurityArtWork Twitter account :-) without even touching directly Twitter servers (obviously, the reflector gets all your requests, so be careful):

$ wget -O- --header='Host: 3benben.appspot.com' \
    https://www.google.es/securityartwork |
    fgrep description | head -1
[...]
  <meta name="description"
  content="The latest Tweets from Security Art Work (@Securityartwork).
  Information Security Blog 
#infosec #cybersecurity #hacking
  #intelligence
S2Grupo Company @s2grupo · s2grupo.es.
  España, Mexico, Colombia">
[...]

This will work from countries or networks in which Twitter is blocked, as long as Google HTTPS frontal servers are not blocked ;-)

There are a lot of CDN networks that can be (ab)used in this way; for example, Meek supports the usage of a reflector hosted in Amazon CloudFront. This reflector proxies to a Tor network bridge:

$ wget -qO- https://a0.awsstatic.com/ --header 'Host: d2zfqthxsdq309.cloudfront.net'
I’m just a happy little web server.

In this way, it’s possible to use the Tor network, and from the outside network monitoring will only get to see HTTPS accesses to the front domain a0.awsstatic.com, used for a bunch of stuff. The same happens with the Microsoft Azure CDN network:

$ wget -qO- https://ajax.aspnetcdn.com/ --header 'Host: az786092.vo.msecnd.net'
I’m just a happy little web server.

In the same way that we could host our C2/exfiltration server on Google App Engine, the above examples show how to make the same thing with a VPS hosted on CloudFront or Azure. This technique can be used to mask any type of network traffic; from outside it will seem like usual access to the corresponding frontal server.

So be careful with contacts with CDN networks and frontal server: they are not always what they seem! :-) From the ignorance of the one who writes (any info is welcome! Feel free to comment!), there are no published reports of APT groups that use this technique directly, but it would not be really surprising that it’s in active usage as a common practice to circumvent egress controls and monitoring (there is information about some cases in which services like Google Docs, Google Drive, Google Apps Script, and so on have been used for C&C or exfiltration, but as far as we know by now, they didn’t use domain fronting: they didn’t hide the access to domains like docs.google.com, which would be a step beyond).

Of course, this is a technique relatively easy to detect by using traffic analysis (because although the domain that we see is only the frontal server domain, in reality the traffic corresponds to that of the embedded service, and that’s going to produce a more or less distinguishable fingerprint, size and timing of network packets, and so on), or anomaly detection (for an easy one, if the HTTPS/TLS client fingerprint it’s not hidden properly, we’ll detect the malware), but that’s another story… ;-)

Best regards!
@pmarinram

Really cool stuff

See also in: