A story around Charles, the debugging proxy application. We see what it offers, why we need it and how to set it up and some regular tools I use.


What is Charles?

Charles Web Debugging Proxy application lets us view the HTTP, HTTPS, HTTP/2 protocol network traffic routing through it.  
In simple words, it lets us debug network calls by sitting in the middle of the client and server, and acting as a proxy to the actual server.

We can use it debug network calls going through it via our macOS or via our devices, if we route their traffic through Charles.


Why you need it?

It offers us many great tools such as:

  1. Breakpoint – Hold a network request or response to analyze or modify it.
  2. Rewrite – automatically modify network request or response based on conditions on variables like path, query parameters etc.
  3. Throttle – change the network speed to emulate real world situations.
  4. And many more!

These tools helps us easily debug, modify and analyze network traffic of application. One doesn't need to recompile the code to change a header, or see different possible response and their handling on the app, or changing network conditions in the real world.  


How Charles works?

Applications communicate with their servers via network calls. We want to intercept them. Most network calls today are HTTPS, which means they are encrypted using the SSL certificate of the server.

What are SSL certificates?

SSL certificates are digital certificates which authenticate the identity of the server and also contain a public encryption key of the server. Applications encrypt any network call they send to the server using this, and  only the servers can decrypt the network calls and process them.

This prevents anyone from reading or modifying our network requests in the middle.  Any mainstream OS like macOS or Android doesn't just trust any certificate of the server but certificates issued by certain "Certificate issuers". Certificate Issuers have their own CA (Certificate Authority) Certificate. This is done to secure any network requests made by the OS applications.

Behind the scenes

With Charles, we want to be in the middle to client and server to intercept, analyze and modify network request or response. For HTTP calls it's easy – the traffic isn't encrypted. But for HTTPS calls, the traffic is encrypted by the SSL certificates of the server.

Charles dynamically generates a certificate for the server and signs it with its own CA certificate. When a call is made, Charles receives the server’s certificate, while your application receives generated server certificate. Charles still communicates via SSL to the server. The communication is SSL (encrypted) from application to Charles and also SSL (encrypted) from Charles to the server.


Setting up Charles

We'll now install and setup Charles in some simple steps.

Download and install charles

Download the DMG from here and then install it.

Mac network preferences

  • Launch Charles and it should ask you to allow it to setup your network preferences. Give it access and it should configure your network preferences. Charles installs it's root CA certificate so the OS trusts the server certificates issued by it as explained above.
  • If  you don't see such an option, don't worry. When Charles is open, go to Help -> SSL proxying -> Install charles root certificate. This will open Keychain access app and  install Charles CA certificate on your OS. You will need to trust the certificate manually like shown below.

Proxy settings

Go to Proxy -> Proxy Settings -> Proxies. Check that Support HTTP/2 is checked and Port is set as 8888. This means that Charles is listening to incoming traffic to it at 8888 port. You can change it to any other port as well.

SSL proxying settings

Now that Charles is listening at the port, we need to enable SSL proxying i.e. the process of how Charles intercepts network calls as explained above. Go to Proxy -> SSL proxying settings -> SSL proxying. Check the Enable SSL Proxying option.
Add *.* in the Include panel. This will enable SSL proxying for all hosts.

Doing this means all application traffic going through Charles will go through the SSL proxying process. This might make many apps not being able to connect to their servers.  This happens because some apps have SSL pinning, i.e. they don't trust just any certificate saying it's their server signed by any CA certificate, they have a copy of their actual server public SSL key and thus reject Charles certificate. To mitigate this we'll remove *.* from Include panel and instead add only the host of our application server e.g. api.mycompany.com


Getting device ready for Charles

Now that Charles is ready, we need to setup our device - make it trust Charles and direct our traffic to it.

WiFi proxy

  • Bring your android device and PC on the same WiFi network. Once that's done we need to re-route entire network traffic to Charles. To do that, first check your PC's local IP address by going to Charles and then Help -> Local IP Address
  • Go to your WiFi settings page. Long press the connected WiFi to see the option to modify it. When the modify popup opens, go to Advanced options -> Proxy -> Set it to manual.
    Then enter the local IP address of PC and port as 8888 and press ok. This will redirect all  traffic on this WiFi to your PC at port 8888, and who's listening at 8888, that's right, Charles! We set that up in it's Proxy Settings.

Charles CA certificate

  • Next we need to download and install the CA certificate of Charles that's running on our mac. Since all traffic is already going through Charles now, we'll download the certificate by visiting http://chls.pro/ssl

Download the certificate and  next we'll need to install it in our device. For that find the install credentials settings in our device settings (Best to search for it).

Click on install a certificate and then on CA certificate

Install the certificate and you'll be able to see it in your user installed certificates like this

Getting project ready for Charles

Now that Charles and our device is ready, we need to setup our project.

  1. Turn off SSL pinning, if you have it in your debug build, so that the project code trusts the Charles issued server certificate.
  2. We need to add a network security config to trust any user CA certificate to issue your server certificates. Create a file called network_security_config.xml and add debug-overrides to allow certificates from system and user.
<network-security-config 
    xmlns:tools="http://schemas.android.com/tools">
    
    <debug-overrides>
        <trust-anchors>
            <certificates src="system" />
            <certificates src="user" />
        </trust-anchors>
    </debug-overrides>
</network-security-config>

Now go to your debug AndroidManifest.xml of the project where you declare application tag and add network security config to it.

 <application
   android:icon="@mipmap/ic_launcher"
   android:networkSecurityConfig="@xml/network_security_config" />

3.   Build the project and install it. You should see your network calls coming in Charles and should see the SSL encrypted traffic as clear-text. This happens since the server certificate was issued by Charles it is able to decrypt the traffic.


Regular tools I use

Now that Charles is running and we can see our applications requests going and the response coming, we have to learn some basic tools.

Before we see the tools, we have to decide a view in which we see the requests. Personally I prefer the Sequence over Structure mode.

Breakpoint tool

Just like a debugger which stops the execution of code where we can analyze values, modify them to simulate different behaviors, the breakpoint tool allows us to see the request headers, body and much more. We can also modify it in flight thus changing our response from the server. We can also breakpoint on the server response and modify it's body or headers.

To enable it , go to Proxy -> Breakpoint Settings. Enable the checkbox "Enable Breakpoints". This is a simple switch for all breakpoints.

To add a new breakpoint, either right click on any request and select breakpoint. Or go to the breakpoint settings and click on Add. It'll ask you the scheme, host, port, path and query parameters against which you may want to check. You also have the option to only stop the request or the response.

Rewrite tool

Sometimes you want to test a specific response lots of time, or maybe do certain changes in request without recompiling. While breakpoint helps us do it, it doesn't automate them based on rules - we have to do it every-time.

To enable it, go to Tools -> Rewrite Settings . Enable the checkbox "Enable Rewrite".

Click on add and you will see on the rules details on the right side. Add a name to the rule to better identify it.

The top half of the dialog is to identify which request should be affected by this rule. Choose the path or params or any other option to your use-case.

The bottom half of the dialog is to show the rewrite logic. You can add, remove, modify headers, query params, path, host, body, pretty much anything in both request and response. It even supports regex!

My most basic use-case is to change something in response to see how my app code handles different responses. How I mostly use is this tool is by replacing the whole API response with something else. To do that, choose Type as "Body", Where as "Response" since we want to rewrite our response and place your whole new response in Replace section "Value" field.

Throttling tool

Sometimes you want to see how your application reacts to changing network bandwidth - you want to throttle the network.

To enable it, go to Proxy -> Throttle settings. Check the "Enable throttling" checkbox. You can even add a specific host if you want. In the throttle settings dialog you can see the throttle preset and a bunch of other options. Choose one options and press ok and your network calls will now throttle.  

Toolbar

All the options we just saw, can be easily accessed from the toolbar at top.
We have a clear current session (Yellow broom), followed by a recording enable/disable button, start/stop SSL proxying (The lock), throttle switch (shown elegantly by a turtle) , breakpoint switch followed by more. Hover mouse over them to check what they are.

Filter

Since Charles will capture all network requests, it may cause a lot of noise. To solve this we can filter the paths we want to see by using the filter bar.


Thanks for reading. If you enjoyed it, leave a comment or reaction.