How To: Use mitmproxy to read and modify HTTPS traffic

Capturing HTTP and HTTPS traffic on your own machine is quite simple: Using tools like Wireshark or Firebug, sniffing the local connections is only a matter of seconds. Capturing and/or altering the HTTP/HTTPS traffic of other machines in your network (such as your smartphone or other laptops) on the other hand is not so easy. Especially sniffing into SSL-secured HTTPS-connections seems impossible at first. Using mitmproxy, however, makes this possible in a very easy and straight forward way.

This small tutorial shows how to use mitmproxy to transparently sniff into and alter (!) HTTPS connections of your phone or other devices in your network.

Contents

1. How it works

Mitmproxy is an open source proxy application that allows intercepting HTTP and HTTPS connections between any HTTP(S) client (such as a mobile or desktop browser) and a web server using a typical man-in-the-middle attack(MITM). Similar to other proxies (such as Squid), it accepts connections from clients and forwards them to the destination server. However, while other proxies typically focus on content filtering or speed optimization through caching, the goal of mitmproxy is to let an attacker monitor, capture and alter these connections in realtime.

1.1. Attacking HTTP connections

For unencrypted HTTP connections, this is quite simple: mitmproxy accepts a connection from the HTTP client, say a mobile browser, displays the request (and its request parameters) to the attacker on the screen, and forwards the request to the destination web server as soon as the attacker confirms — maybe after adjusting the request payload a bit. mitmproxy simply acts as a middle man: To the client, it looks like as if the mitmproxy server was simply relaying its connection (like your router or your ISP’s servers do). And to the server, it looks like the mitmproxy server is the client.

1.2. Attacking HTTPS connections

While attacking unencrypted HTTP traffic can be done without having to deal with X.509 certificates and certificate authorities (CA), SSL-encrypted HTTPS connections encrypt every request and response between client and server end-to-end. And because the transferred data is encrypted with a shared secret, a middle man (or a proxy) cannot decipher the exchanged data packets. When the client opens an SSL/TLS connection to the secure web server, it verifies the server’s identity by checking two conditions: First, it checks whether its certificate was signed by a CA known to the client. And second, it makes sure that the common name (CN, also: host name) of the server matches the one it connects to. If both conditions are true, the client assumes the connection is secure.

In order to be able to sniff into the connection, mitmproxy acts as a certificate authority, however, not a very trustworthy one: Instead of issuing certificates to actual persons or organizations, mitmproxy dynamically generates certificates to whatever hostname is needed for a connection. If, for instance, a client wants to connect to https://www.facebook.com, mitmproxy generates a certificate for “www.facebook.com” and signs it with its own CA. Provided that the client trusts this CA, both of the above mentioned conditions are true (trusted CA, same CN) — meaning that the client believes that the mitmproxy server is in fact “www.facebook.com”. The figure below shows the request/response flow for this scenario. This mechanism is called transparent HTTPS proxying.

For this attack to work, there are a few conditions that must be met:

• Mitmproxy as standard gateway (HTTP and HTTPS): For both HTTP and HTTPS proxying, the server running mitmproxy must of course be able to intercept the IP packets — meaning that it must be somewhere along the way of the packet path. The easiest way to achieve this is to change the default gateway in the client device to the mitmproxy server address.
• Trusted mitmproxy CA (HTTPS only): For the HTTPS proxying to work, the client must know (and trust!) the mitmproxy CA, i.e. the CA key file must be added to the trust store of the client.

1.3. More Details

In case that description is not detailed enough for you: There is a wonderful explanation of how mitmproxy works and its different modes of mitmproxy on the official website of the project.

2. Install & run mitmproxy

After this little theory session, lets get down to business.

2.1. Install mitmproxy

The installation of mitmproxy is very simple, because it’s been packaged using the Python package management system (pip). Other mitmproxy dependencies can be installed with apt-get:

2.2. Install mitmproxy CA certificate in the phone

Next, you need to install the mitmproxy-generated CA certificate in the device for which you want to capture/alter the HTTPS connection. This can be a desktop browser, or a mobile phone (Android, iOS, ..). The mitmproxy documentationhas a good section that tells you how to do this.

Here’s a short summary for Android:

1. Upload the certificate located at ~/.mitmproxy/mitmproxy-ca-cert.cer to /sdcard/Download/mitmproxy-ca-cert.cer of your Android device.
2. Go to Settings, Security and click “Install from device storage”
3. Enter “mitmproxy-ca-cert” (no suffix!) and click “OK”
4. Now click on “Trusted credentials” and select the “User” tab. The certificate should now appear in the list.

2.3. Enable IP forwarding and port redirection

The mitmproxy application internally runs on TCP port 8080, but externally has to listen on ports 80/HTTP and 443/HTTPS. Therefore, a IP forwarding in general (the system must act as a router) and a redirection from 8080 to 80 and 443 is necessary for all arriving IP packets. The “nat” table of iptables can be used to do that pretty easily. This is also described in the Linux section of the mitmproxy manual.

It’s not clear to me why the application does not simply bind to the ports 80 and 443 ports, but that’s how it is right now.

2.4. Start mitmproxy

As mentioned above, mitmproxy runs on port 8080, but also needs 80 and 443 to be bound, all three ports cannot be used by any other applications at the same time. So simply stop anything that might be blocking those ports (Apache web server, Tomcat, Glassfish, etc.). If you’re unsure, check out netstat -ntap.

Important side note: mitmproxy has the option “-p”, which allows you to choose a different port than 8080. Thisoption did not work for me when I tried it. While it did listen on the given port, no requests were intercepted. The display just stayed blank. It unfortunately cost me three hours to find that out.

To actually start mitmproxy, simply run this command:

You’ll see a blank black screen with a blue bar on the bottom — and nothing will happen because your phone (or browser) does not yet communicate via mitmproxy. Familiarize yourself with the UI, read the docs on the official site, and check out the help page by typing ?.

2.5. Change the standard gateway of the phone

To be able to sniff into your phone’s (or browser’s) HTTP/HTTPS connections, you need to change the standard gateway to the IP address of the mitmproxy server.

You can do this on Android like shown below:

1. Go to Settings, Wi-Fi and long-press on your connected network.
2. Choose “Modify network config.”
3. Check “Show advanced options”, set IP settings to “Static” and manually change the gateway address to the mitmproxy server address. Click “Save”.

2.6. Capture and alter HTTP/HTTPS requests and responses

After you’ve changed the settings, start interacting with the device (go to facebook.com, update your WhatsApp status, open your browser). If everything works, you should see lots of HTTP and HTTPS (!) requests on the mitmproxy screen.

You can navigate up and down using your arrow keys. Press the return key to look at a request in more detail if you like.

As you’ll notice, right now, you do not have the chance to alter any request. You can simply look at them after they happened. To actually intercept, alter and then send the request (or response), you have to set an interception filter. To do so, press i and enter a filter expression such as “facebook.com” (intercept all requests that contain “facebook.com” in their URL) or simply “.*” (intercept all requests/responses).

Intercepted flows will appear orange in the UI. They will not be further processed until you accept them (with or without editing them). To accept a flow, press a. To accept all holding flows, press A.

Intercepted flows can also be inspected in more detail by pressing the return key. In contrast to before, however, they can also be edited by pressing the e key. Depending on which mode you chose, you can edit different parts of the request: header, body, etc. — using vim or other editors.

2.7. Stop mitmproxy and undo iptables changes

Before stopping mitmproxy make sure you don’t want to save any of the flows. If you do, w will do it. To exit, press qand then y.

To undo the iptables changes we’ve done earlier, simply type iptables -t nat -F (flush/delete the whole table). If you have any other ‘nat’ table entries, you might want to remove the entries manually one by one. If unsure, check first with iptables -t nat -L.

If you want, also undo the IP forwarding setting by typing sysctl -w net.ipv4.ip_forward=0.

3. Further thoughts

3.1. Using DNS instead of the standard gateway

The method below requires changing the standard gateway of the device/phone. While this method works quite well, it requires changing the network settings of the device. Instead of using the standard gateway to redirect connections, one could also think of using a fake DNS server to do that.

So the default gateway on the phone stays the same. Instead, the DNS server in the router is changed to a local DNS server that returns the mitmproxy IP address for all domains to be monitored. This should work perfectly fine if mitmproxy and the DNS server do not run on the same machine. If they do, mitmproxy would resolve the domain to itself, which would of course lead to nothing.

I have not yet tried this, however, I am confident that it should work.

3.2. Why this is more scary than you think

I am not much of a conspiracy type of guy, but the recent headlines about the Internet spying activities of various intelligence agencies around the globe (cf. PRISM, Tempora) have really shocked me a bit. Why? Well, consider the following two thoughts:

Access to Internet Exchange Points: If a person (or a government agency) can tap into Internet and phone communications at an Internet Exchange Point (such as DE-CIX or similar) that means that this person can capture all of the connections going through that node. That includes HTTP traffic, mail traffic via SMTP, etc. — simply everything that using TCP or UDP as a transport. Unencrypted traffic (HTTP, SMTP, FTP) is readable by everyone who has access to this node. Encrypted traffic (HTTPS, or any TLS/SSL-secured connection) can, however, not be deciphered by that person.

Access to a trusted certificate authority: If that same person also has access to a certificate authority whose CA certificates are globally trusted (like one of the thirty-something trusted CAs that are shipped with all the browsers and operating systems), that person can use the above described man-in-the-middle attack to decipher any TLS/SSL connection.

In short: access to internet exchange points plus a private key of a major root CA certificate of a trusted certificate authority equals the ability to break every TLS/SSL connection that goes through the respective node. Scary, right?

3.3. Sniffing into non-HTTPS traffic

Sniffing traffic with mitmproxy is limited to HTTP and HTTPS conversations only — meaning that you cannot listen into non-HTTP(S) traffic with mitmproxy. If you’re interested in transparently sniffing plain SSL sockets, you might want to try SSLsplit, a transparent TLS/SSL man-in-the-middle proxy. Also check out my tutorial on how to use SSLsplit to spy on non-HTTPS conversations (e.g. SMTP over SSL or IMAP over SSL).

http://blog.philippheckel.com/2013/07/01/how-to-use-mitmproxy-to-read-and-modify-https-traffic-of-your-phone/