Integrating with Github Webhooks using OWIN

For some reason I got the urge to have a look at webhooks when using GitHub. Since it is a feature that is used extensively by build servers and other applications to do things when code is pushed to GitHub etc, I thought it might be cool to have a look at how it works under the hood. And maybe build some interesting integration in the future…

The basic idea behind it is that you tell GitHub that you want to get notified when things happen in your GitHub repo, and GitHub makes sure to do so. It does so using a regular HTTP call to an endpoint of your choice.

So to start off, I decided to create a demo repo on GitHub and then add a webhook. This is easily done by browsing to your repo and clicking the “Settings” link to the right.

In the menu on the left hand side in the resulting page, you click the link called “Webhooks & Services”.

Then you click “Add webhook” and add the “Payload URL”, the content type to use and a secret. You can also define whether you want more than just the “push” event, which tells you when someone pushed some code, or if that is good enough. For this post, that is definitely enough… Clicking the “Add webhook” button will do just that, set up a webhook for you. Don’t forget that the hook must be active to work…

Ok, that is all you need to do to get webhooks up and running from that end!

There is a little snag though… GitHub will call your “Payload URL” from the internet (obviously…). This can cause some major problems when working on your development machine. Luckily this can be easily solved by using a tool called ngrok.

ngrok is a secure tunneling service that makes it possible to easily expose a local http port on your machine on the net. Just download the program and run it in a console window passing it the required parameters. In this case, tunneling an HTTP connection on port 4567 would work fine.

ngrok http 4567

The important part to note here is the http://e5630ddd.ngrok.io forwarding address. This is what you need to use when setting up the webhook at GitHub. And if you want more information about what is happening while using ngrok, just browse to http://127.0.0.1:4040/.

Ok, so now we have a webhook set up, and a tunnel that will make sure it can be called. The next thing is to actually respond to it…

In my case, I created a Console application in VS2013 and added the NuGet packages Microsoft.Owin.SelfHost and Newtonsoft.Json. Next, I created a new Owin middleware called WebhookMiddleware, as well as a WebhookMiddlewareOptions and WebhookMiddlewareExtensions, to be able to follow the nice AddXXX() pattern for IAppBuilder. It looks something like this

publicclass WebhookMiddleware : OwinMiddleware
{
privatereadonly WebhookMiddlewareOptions _options;

public WebhookMiddleware(OwinMiddleware next, WebhookMiddlewareOptions options)
        : base(next)
    {
        _options = options;
    }

publicoverride async Task Invoke(IOwinContext context)
    {
        await Next.Invoke(context);
    }
}

publicclass WebhookMiddlewareOptions
{
public WebhookMiddlewareOptions()
    {
    }

publicstring Secret { get; set; }
}

publicstaticclass WebhookMiddlewareExtensions
{
publicstaticvoid UseWebhooks(this IAppBuilder app, string path, WebhookMiddlewareOptions options = null)
    {
if (!path.StartsWith("/"))
            path = "/" + path;

        app.Map(path, (app2) =>
        {
            app2.Use<WebhookMiddleware>(options ?? new WebhookMiddlewareOptions());
        });
    }
}

As you can see, we use the passed in path to map when the middleware should be used…

Ok, now that all the middleware scaffolding is there, let’s just quickly add it to the IAppBuilder as well…

class Program
{
staticvoid Main(string[] args)
    {
using (WebApp.Start<Startup>("http://127.0.0.1:4567"))
            Console.ReadLine();
    }
}

publicclass Startup
{
publicvoid Configuration(IAppBuilder app)
    {
        app.UseWebhooks("/webhook", new WebhookMiddlewareOptions { Secret = "12345" });
    }
}

Note: The address passed to the OWIN server. It isn’t the usual localhost. Instead, I use 127.0.0.1, which is kind of the same, but not quite. To get the ngrok tunnel to work, it needs to be 127.0.0.1

That’s the baseline… Let’s add some actual functionality!

The first thing I need is a way to expose the information sent from GitHub. It is sent using JSON, but I would prefer it if I could get it statically typed for the end user. As well as readonly… So I added 4 classes that can represent at least the basic information sent back.

If we start from the top, we have the WebhookEvent. It is the object containing the basic information sent from GitHub. It looks like this

publicclass WebhookEvent
{
protected WebhookEvent(string type, string deliveryId, string body)
    {
        Type = type;
        DeliveryId = deliveryId;

        var json = JObject.Parse(body);
        Ref = json["ref"].Value<string>();
        Before = json["before"].Value<string>();
        After = json["after"].Value<string>();
        HeadCommit = new GithubCommit(json["head_commit"]);
        Commits = json["commits"].Values<JToken>().Select(x => new GithubCommit(x)).ToArray();
        Pusher = new GithubUser(json["pusher"]);
        Sender = new GithubIdentity(json["sender"]);
    }

publicstatic WebhookEvent Create(string type, string deliveryId, string body)
    {
returnnew WebhookEvent(type, deliveryId, body);
    }

publicstring Type { get; private set; }
publicstring DeliveryId { get; private set; }
publicstring Ref { get; private set; }
publicstring Before { get; private set; }
publicstring After { get; private set; }
public GithubCommit HeadCommit { get; set; }
public GithubCommit[] Commits { get; set; }
public GithubUser Pusher { get; private set; }
public GithubIdentity Sender { get; private set; }
}

As you can see, it is just a basic DTO that parses the JSON sent from GitHub and puts it in a statically typed class…

The WebhookEvent class exposes referenced commits using the GitHubCommit class, which looks like this

publicclass GithubCommit
{
public GithubCommit(JToken data)
    {
        Id = data["id"].Value<string>();
        Message = data["message"].Value<string>();
        TimeStamp = data["timestamp"].Value<DateTime>();
        Added = ((JArray)data["added"]).Select(x => x.Value<string>()).ToArray();
        Removed = ((JArray)data["removed"]).Select(x => x.Value<string>()).ToArray();
        Modified = ((JArray)data["modified"]).Select(x => x.Value<string>()).ToArray();
        Author = new GithubUser(data["author"]);
        Committer = new GithubUser(data["committer"]);
    }

publicstring Id { get; private set; }
publicstring Message { get; private set; }
public DateTime TimeStamp { get; private set; }
publicstring[] Added { get; private set; }
publicstring[] Removed { get; private set; }
publicstring[] Modified { get; private set; }
public GithubUser Author { get; private set; }
public GithubUser Committer { get; private set; }
}

Once again, it is just a JSON parsing DTO. And so are the last 2 classes, the GitHubIdentity and GitHubUser…

publicclass GithubIdentity
{
public GithubIdentity(JToken data)
    {
        Id = data["id"].Value<string>();
        Login = data["login"].Value<string>();
    }

publicstring Id { get; private set; }
publicstring Login { get; private set; }
}

publicclass GithubUser
{
public GithubUser(JToken data)
    {
        Name = data["name"].Value<string>();
        Email = data["email"].Value<string>();
if (data["username"] != null)
            Username = data["username"].Value<string>();
    }

publicstring Name { get; private set; }
publicstring Email { get; private set; }
publicstring Username { get; private set; }
}

Ok, those are all the boring scaffolding classes to get data from the JSON to C# code…

Let’s have a look at the actual middleware implementation…

The first thing it need to do is to read out the values from the request. This is very easy to do. The webhook will get 3 headers and a body.  And they are read like this

publicoverride async Task Invoke(IOwinContext context)
{
    var eventType = context.Request.Headers["X-Github-Event"];
    var signature = context.Request.Headers["X-Hub-Signature"];
    var delivery = context.Request.Headers["X-Github-Delivery"];

string body;
using (var sr = new StreamReader(context.Request.Body))
    {
        body = await sr.ReadToEndAsync();
    }
}

Ok, now that we have all the data, we need to verify that the signature passed in the X-Hub-Signature header is correct.

The passed value will look like this sha1=XXXXXXXXXXX, and the XXXXXXXXX is a HMAC SHA1 hash generated using the body and the secret. To validate the hash, I add a method to the WebhookMiddlewareOptions class, and make it private. In just a minute I will explain how I can still let the user make modifications to it even if it is private…

It looks like this

privatebool ValidateSignature(string body, string signature)
{
    var vals = signature.Split('=');
if (vals[0] != "sha1")
returnfalse;

    var encoding = new System.Text.ASCIIEncoding();
    var keyByte = encoding.GetBytes(Secret);

    var hmacsha1 = new HMACSHA1(keyByte);

    var messageBytes = encoding.GetBytes(body);
    var hashmessage = hmacsha1.ComputeHash(messageBytes);
    var hash = hashmessage.Aggregate("", (current, t) => current + t.ToString("X2"));

return hash.Equals(vals[1], StringComparison.OrdinalIgnoreCase);
}

As you can see, it is pretty much just a matter of generating a HMAC SHA1 hash based on the body and secret, and then verifying that they are equal. I do this case-insensitive as the .NET code will generate uppercase characters, and the GitHub signature is lowercase.

Now that we have this validation in place, it is time to hook it up. I do this by exposing a OnValidateSignature property of type Func<string, string, bool> on the options class, and assign it to the private function in the constructor.

publicclass WebhookMiddlewareOptions
{
public WebhookMiddlewareOptions()
    {
        OnValidateSignature = ValidateSignature;
    }

    ...

publicstring Secret { get; set; }
public Func<string, string, bool> OnValidateSignature { get; set; }
}

This way, the user can just leave that property, and it will verify the signature as defined. Or he/she can replace the func with their own implementation and override the way the validation is done.

The next step is to make sure that we validate the signature in the middleware

publicoverride async Task Invoke(IOwinContext context)
{
    ...

if (!_options.OnValidateSignature(body, signature))
    {
        context.Response.ReasonPhrase = "Could not verify signature";
        context.Response.StatusCode = 400;
return;
    }
}

And as you can see, if the validation doesn’t approve the signature, it returns an HTTP 400.

So why are we validating this? Well, considering that you are exposing this endpoint on the web, it could get compromised and someone could be sending spoofed messages to your application…

Ok, the last thing to do is to make it possible for the middleware user to actually do something when the webhook is called. Once again I expose a delegate on my options class. In this case it is an Action of type WebhookEvent, called OnEvent. And once again I add a default implementation in the options class itself. However, the implementation doesn’t actually do anything in this case. But it means that I don’t have to do a null check…

publicclass WebhookMiddlewareOptions
{
public WebhookMiddlewareOptions()
    {
        ...
        OnEvent = (obj) => { };
    }

    ...

public Action<WebhookEvent> OnEvent { get; set; }
}

And now that we have a way to tell the user that the webhook has been called, we just need to do so…

publicoverride async Task Invoke(IOwinContext context)
{
    ...

    _options.OnEvent(WebhookEvent.Create(eventType, delivery, body));

    context.Response.StatusCode = 200;
}

The last thing to do is also to send an HTTP 200 back to the server, telling it that the request has been accepted and processed properly.

Now that we have a callback system in place, we can easily hook into the webhook and do whatever processing we want. In my case, that means doing a very exciting Console.WriteLine

publicvoid Configuration(IAppBuilder app)
{
    app.UseWebhooks("/webhook", new WebhookMiddlewareOptions
    {
        Secret = "12345",
        OnEvent = (e) =>
        {
            Console.WriteLine("Incoming hook call: {0}\r\nCommits:\r\n{1}", e.Type, string.Join("\r\n", e.Commits.Select(x => x.Id)));
        }
    });

    app.UseWelcomePage("/");
}

That’s it! A fully working and configurable webhook integration using OWIN!

And as usual, there is code for you to download! It is available here: DarksideCookie.Owin.GithubWebhooks.zip (13KB)

Cheers!