Writing network proxies for development purposes in C#

If you are developing, testing, or supporting web applications, you probably encounter situations when you need to record or modify HTTP traffic. Quite often, the browser request viewer might be enough, but what if you need to modify the traffic on the fly? Another challenging task is testing how your application behaves when put behind a load balancer or an edge server. There are many great HTTP proxies available in the market, including mitmproxy, Burp Suite, or Fiddler and they may be perfect in diagnosing/testing your applications. In this post, however, I am encouraging you to write small tools for your specific needs. There are many reasons why you may want to do so, such as the need for complex requests modifications, better control over the request processing, or customizations of the certificate creation. Of course, implementing the HTTP protocol could be demanding so, don’t worry; we won’t do that 🙂 Instead, we will use the open-source Titanium Web Proxy. The code samples in this post are meant to be run in LINQPad, which is my favorite tool for writing and running .NET code snippets, but you should have no difficulties in porting the samples to a C# script or a console application.

Record or modify HTTP traffic (HTTP proxy)

We start with the most common scenario, which is recording HTTP traffic. Titanium is excellent for this purpose. To start an HTTP proxy on port 8080, you simply write:

proxyServer = new ProxyServer(userTrustRootCertificate: false); // we don't want to install any certificates

var httpProxy = new ExplicitProxyEndPoint(IPAddress.Any, 8080, decryptSsl: true);

proxyServer.AddEndPoint(httpProxy);
proxyServer.Start();

string command;
while ((command = Console.ReadLine()) != "quit") {
    if (command == "clear") {
        Util.ClearResults();
    }
}

proxyServer.Stop();

We enabled SSL decryption, so Titanium generates the necessary certificates. We will get to this process in a moment, but first, let’s try to record the request and response data. There is a bunch of events we could use, but the most important ones are:

OnBeforeRequest happens after the request reaches the proxy, but before it is sent to the destination server. If you want to modify the request data or skip it, providing a static response, it’s the right place to do so.
OnBeforeResponse happens after the proxy receives the response from the destination server but before the response is sent to the proxy client. If you want to modify or read the response data, it is the hook you need to use.
OnAfterResponse happens after the response from the destination server is sent to the proxy client. It might be the right place to log the event – keep in mind though that the request and response bodies might be already reclaimed.

A simple LINQPad code which logs the requests and responses might look as follows:

proxyServer.BeforeRequest += OnBeforeRequest;
proxyServer.BeforeResponse += OnBeforeResponse;
proxyServer.AfterResponse += OnAfterResponse;

...

async Task OnBeforeRequest(object sender, SessionEventArgs ev)
{
    // Before the request to the remote server
    await Task.CompletedTask;
}

async Task OnBeforeResponse(object sender, SessionEventArgs ev)
{
    // Before the response from the remote server is sent to the 
    // local client. You may read body here: ev.GetRequestBody()

    var request = ev.HttpClient.Request;
    var response = ev.HttpClient.Response;

    Util.Highlight(request.Url).Dump();
    Util.RawHtml("<pre style=\"font-family: Consolas\">" + request.HeaderText + "</pre>").Dump();
    if (request.HasBody) {
        (await ev.GetRequestBodyAsString()).Dump();
    }
    Util.RawHtml("<pre style=\"font-family: Consolas\">" + response.HeaderText + "</pre>").Dump();
    try {
        if (response.HasBody) {
            var resp = (await ev.GetResponseBodyAsString());
            $"A response to a request: {request.RequestUri}".Dump();
            resp.Dump();
        }
    } catch (Exception ex) {
        ex.ToString().Dump();
    }
    await Task.CompletedTask;
}

async Task OnAfterResponse(object sender, SessionEventArgs ev)
{
    // After the response from the remote server was sent to the 
    // local client
    await Task.CompletedTask;
}

The HTTP proxy requires some configuration on the client. Apart from setting the proxy address, you may want to add the proxy root certificate to the trust store. Titanium does not have a nice endpoint for this purpose (like mitmproxy does) so I usually add the necessary code to the OnBeforeRequest event:

async Task OnBeforeRequest(object sender, SessionEventArgs ev)
{
    // Before the request to the remote server
    var request = ev.HttpClient.Request;
    if (!ev.IsHttps && request.Host == "titanium") {
        if (request.RequestUri.AbsolutePath.Equals("/cert/pem", StringComparison.OrdinalIgnoreCase)) {
            // send the certificate
            var headers = new Dictionary<string, HttpHeader>() {
                ["Content-Type"] = new HttpHeader("Content-Type", "application/x-x509-ca-cert"),
                ["Content-Disposition"] = new HttpHeader("Content-Disposition", "inline; filename=titanium-ca-cert.pem")
            };
            ev.Ok(File.ReadAllBytes(concertoCerts.RootCertPath), headers, true);
        } else {
            var headers = new Dictionary<string, HttpHeader>() {
                ["Content-Type"] = new HttpHeader("Content-Type", "text/html"),
            };
            ev.Ok("<html><body><h1><a href=\"/cert/pem\">PEM</a></h1></body></html>");
        }
    }
    await Task.CompletedTask;
}

Now, you simply need to access http://titanium in your browser, click the PEM link (that’s the certificate format) and you should be prompted to install the certificate. There might be some additional steps required to trust the certificate fully, so be aware and search the web, if necessary.

But what if you want to be in full control of the certificate creation? Unfortunately, Titanium does not give you an easy way to do so. They have a predefined set of “certificate engines,” and you may select one by setting the ProxyServer.CertificateManager.CertificateEngine property. At the moment, you can’t use your custom engine without recompiling Titanium. However, we could hack Titanium a bit, and inject our engine by providing a custom certificate cache implementation. In my example, I wanted to use certificates generated by the Concerto library. The first step was to enable caching and set the certificate storage to my custom cache implementation:

using var concertoCerts = new ConcertoCertificateCache(rootCertPath);
var proxyServer = new ProxyServer(userTrustRootCertificate: false);

proxyServer.CertificateManager.SaveFakeCertificates = true;

proxyServer.CertificateManager.CertificateStorage = concertoCerts;

The ConcertoCertificateCache looks as follows:

internal sealed class ConcertoCertificateCache : ICertificateCache, IDisposable
{
    private readonly CertificateChainWithPrivateKey rootCert;
    private readonly Dictionary<string, X509Certificate2> cache = new();

    public ConcertoCertificateCache(string rootCertPath)
    {
        if (File.Exists(rootCertPath)) {
            rootCert = CertificateFileStore.LoadCertificate(rootCertPath);
        } else {
            rootCert = CertificateCreator.CreateCACertificate(name: "Titanium");
            CertificateFileStore.SaveCertificate(rootCert, rootCertPath);
        }
        cache.Add("RootCA", ConvertConcertoCertToWindows(rootCert));
    }

    public CertificateChainWithPrivateKey RootCert => rootCert;

    public X509Certificate2 LoadCertificate(string subjectName, X509KeyStorageFlags storageFlags)
    {
        lock (cache) {
            if (!cache.TryGetValue(subjectName, out var cert)) {
                Console.WriteLine($"Loading cert for {subjectName}");
                subjectName = subjectName.Replace("$x$", "*");
                cert = ConvertConcertoCertToWindows(CertificateCreator.CreateCertificate(new[] { subjectName }, rootCert));
            }
            return cert;
        }
    }

    public X509Certificate2 LoadRootCertificate(string pathOrName, string password, X509KeyStorageFlags storageFlags)
    {
        return cache["RootCA"];
    }

    private X509Certificate2 ConvertConcertoCertToWindows(CertificateChainWithPrivateKey certificateChain)
    {
        const string password = "password";
        var store = new Pkcs12Store();

        var rootCert = certificateChain.PrimaryCertificate;
        var entry = new X509CertificateEntry(rootCert);
        store.SetCertificateEntry(rootCert.SubjectDN.ToString(), entry);

        var keyEntry = new AsymmetricKeyEntry(certificateChain.PrivateKey);
        store.SetKeyEntry(rootCert.SubjectDN.ToString(), keyEntry, new[] { entry });
        using (var ms = new MemoryStream()) {
            store.Save(ms, password.ToCharArray(), new SecureRandom());

            return new X509Certificate2(ms.ToArray(), password, X509KeyStorageFlags.Exportable);
        }
    }

    public void SaveCertificate(string subjectName, X509Certificate2 certificate)
    {
        // we are not implementing it on purpose
    }

    public void SaveRootCertificate(string pathOrName, string password, X509Certificate2 certificate)
    {
        // we are not implementing it on purpose
    }

    public void Clear()
    {
        // we are not implementing it on purpose
    }

    public void Dispose()
    {
        foreach (var c in cache.Values) {
            c.Dispose();
        }
    }
}

Emulate a load balancer or an edge server (transparent/reverse proxy)

In the previous paragraph, we covered proxying HTTP traffic on the client-side. As noted, using the HTTP proxy requires the configuration of the client (or changing the Operating System settings). And what if we can’t or don’t want to do that? A solution could be a transparent (aka reverse) proxy. It runs in front of the web server, and for the clients looks like the web server itself, thus the reason why we call it transparent. There are other benefits of using this type of proxy, including a possibility to test how your application behaves behind a load balancer. Imagine, your site is available at https://example.net, and a load balancer forwards traffic to your application servers. To reproduce this configuration locally, let’s assume we have two application processes listening on localhost on ports 7880 and 7881. A very simple transport-level proxy could use a bunch of socket objects:

async Task Main()
{
    var listenSocket = new Socket(SocketType.Stream, ProtocolType.Tcp);
    listenSocket.Bind(new IPEndPoint(IPAddress.Loopback, 443));

    var remoteEndpoints = new[] { 
        new IPEndPoint(IPAddress.Parse("127.0.0.1"), 7880),
        new IPEndPoint(IPAddress.Parse("127.0.0.1"), 7881),
    };

    Console.WriteLine("Listening on port 443");

    listenSocket.Listen(120);
    int index = 0;

    while (true) {
        var socket = await listenSocket.AcceptAsync();
        Console.WriteLine($"[{socket.RemoteEndPoint}]: connected");

        // random remote socket
        var remoteSocket = new Socket(SocketType.Stream, ProtocolType.Tcp);
        var remoteEndpoint = remoteEndpoints[index & 0x1];
        index = ~index;

        Console.WriteLine($"[{remoteEndpoint}]: forwarding");

        await remoteSocket.ConnectAsync(remoteEndpoint);

        _ = ProcessRequest(socket, remoteSocket);
    }
}

// Define other methods and classes here

private static async Task ProcessRequest(Socket clientSocket, Socket remoteSocket)
{
    var cts = new CancellationTokenSource();
    var tasks = new List<Task>() { 
        Forward(clientSocket, remoteSocket, cts.Token), 
        Forward(remoteSocket, clientSocket, cts.Token) 
    };

    tasks.Remove(await Task.WhenAny(tasks));

    Console.WriteLine($"[{clientSocket.RemoteEndPoint}]: disconnected");

    cts.Cancel();

    // there will be cancel exceptions thrown here, but we swallow them
    await Task.WhenAny(tasks);

    clientSocket.Dispose();
    remoteSocket.Dispose();
}

private static async Task Forward(Socket src, Socket target, CancellationToken ct)
{
    var buffer = new byte[1024];
    using (var sourceStream = new NetworkStream(src))
    using (var targetStream = new NetworkStream(target)) {
        int read;
        while ((read = await sourceStream.ReadAsync(buffer, 0, buffer.Length, ct)) != 0) {
            await targetStream.WriteAsync(buffer, 0, read, ct);
        }
    }
}

When we run it and start posting requests to the https://example.net address, each application instance will serve every second request. A “smarter” proxy should decrypt TLS and maybe also parse HTTP, thus giving us a chance to modify the requests and responses. We can implement such a proxy using Titanium.Web.Proxy (it supports the reverse proxy mode, too!). Our sample flow might look as follows:

void Main()
{
    var proxyServer = new ProxyServer();

    proxyServer.ServerCertificateValidationCallback += OnServerCertificateValidation;

    proxyServer.BeforeRequest += OnBeforeRequest;

    var tcpProxy = new TransparentProxyEndPoint(IPAddress.Loopback, 443, true);

    proxyServer.AddEndPoint(tcpProxy);
    proxyServer.Start();

    string command;
    while ((command = Console.ReadLine()) != "quit") {
        if (command == "clear") {
            Util.ClearResults();
        }
    }

    // Unsubscribe & Quit
    proxyServer.BeforeRequest -= OnBeforeRequest;

    proxyServer.Stop();

}

// Define other methods and classes here

async Task OnServerCertificateValidation(object sender, CertificateValidationEventArgs ev)
{
    // If our destination server has only the domain name in the certificate, we might check it
    // or simply don't care (FOR DEVELOPMENT ONLY).
    ev.IsValid = true;
    await Task.CompletedTask;
}

int index = 0;

async Task OnBeforeRequest(object sender, SessionEventArgs ev)
{
    var request = ev.HttpClient.Request;
    var remotePort = index == 0 ? 7880 : 7881;
    index = ~index;

    // no https
    var destRequestUriString = $"https://127.0.0.1:{remotePort}{request.RequestUri.PathAndQuery}";

    Console.WriteLine($"{request.Method} {request.Url}, redirecting to {destRequestUriString}");

    request.RequestUriString = destRequestUriString;
    request.Host = "example.net";

    await Task.CompletedTask;
}

It works similarly to the previous TCP proxy, but now we can parse and modify HTTP traffic. If in the destRequestUriString we replace https:// with http://, we could emulate a scenario when the edge server performs the TLS handshake and passes unencrypted data to the application.

Final words

I hope you find the techniques presented in this post useful. All the code samples from this post are also available in my Debug Recipes repository, in the network folder.

Debug notes

By Sebastian Solnica