Integrating a front-end build pipeline in ASP.NET builds

A while back I wrote a couple of blog posts about how to set up a “front-end build pipeline” using Gulp. The pipeline handled things like less to css conversion, bundling and minification, TypeScript to JavaScript transpile etc. However, this pipeline built on the idea that you would have Gulp watching your files as you worked, and doing the work as they changed. The problem with this is that it only runs on the development machine, and not the buildserver.… I nicely avoided this topic by ending my second post with “But that is a topic for the next post”. I guess it is time for that next post…

So let’s start by looking at the application I will be using for this example!

The post is based on a very small SPA application based on ASP.NET MVC and AngularJS. The application itself is really not that interesting, but I needed something to demo the build with. And it needed to include some things that needed processing during the build. So I decided to build the application using TypeScript instead of JavaScript, and Less instead of CSS. This means that I need to transpile my code, and then bundle and minify it.

I also want to have a configurable solution where I can serve up the raw Less files, together with a client-side Less compiler, as well as the un-bundled and minified JavaScript files, during development, and the bundled and minified CSS and JavaScript in production. And on top of that, I want to be able to include a CDN in the mix as well if that becomes a viable option for my app.

Ok, so let’s get started! The application looks like this

As you can see, it is just a basic ASP.NET MVC application. It includes a single MVC controller, that returns a simple view that hosts the Angular application. The only thing that has been added on top of the empty MVC project is the Microsoft.AspNet.Web.Optimization NuGet package that I will use to handle the somewhat funky bundling that I need.

What the application does is fairly irrelevant to the post. All that is interesting is that it is an SPA with some TypeScript files, some Less files and some external libraries, that needs transpiling, bundling, minification and so on during the build.

As I set work on my project, I use Bower to create a bower.json file, and then install the required Bower dependencies, adding them to the bower.json file using the –save switch. In this case AngularJS, Bootstrap and Less. This means that I can restore the Bower dependencies real easy by running ”bower install” instead of having to check them into source control.

Next I need to set up my bundling using the ASP.NET bundling stuff in the Microsoft.AspNet.Web.Optimization NuGet package. This is done in the BundleConfig file.

If you create an ASP.NET project using the MVC template instead of the empty one that I used, you will get this package by default, as well as the BundleConfig.cs file. If you decided to do it like me, and use the empty one, you need to add a BundleConfig.cs file in the App_Start folder, and make sure that you call the BundleConfig.RegisterBundles() method during application start in Global.asax.

Inside the BundleConfig.RegisterBundles I define what bundles I want, and what files should be included in them… In my case, I want 2 bundles. One for scripts, and one for styles.

Let’s start by the script bundle, which I call ~/scripts. It looks like this

publicstaticvoid RegisterBundles(BundleCollection bundles)
{
    bundles.Add(new ScriptBundle("~/scripts", ConfigurationManager.AppSettings["Optimization.PathPrefix"] + "scripts.min.js") { Orderer = new ScriptBundleOrderer() }
        .Include("~/bower_components/angularjs/angular.js")
        .Include("~/bower_components/less/dist/less.js")
        .IncludeDirectory("~/Content/Scripts", "*.js", true));
    ...
}

As you can see from the snippet above, I include AngularJS and Less.js from my bower_components folder, as well as all of my JavaScript files located in the /Content/Scripts/ folder and its subfolders. This will include everything my application needs to run. However, there are 2 other things that are very important in this code. The first one is that second parameter that is passed to the ScriptBundle constructor. It is a string that defines the Url to use when the site is configured to use CDN. I pass in a semi-dynamically created string, by concatenating a value from my web.config, and “scripts.min.js”.

In my case, the Optimization.PathPrefix value in my web.config is defined as “/Content/dist/” at the moment. This means that if I were to tell the application to use CDN, it would end up writing out a script-tag with the source set to “/Content/dist/scripts.min.js”. However, if I ever did decide to actually use a proper CDN, I could switch my web.config setting to something like “//mycdn.cdnnetwork.com/” which would mean that the source would be changed //mycdn.cdnnetwork.com/scripts.min.js, and all of the sudden point to an external CDN instead of my local files. This is a very useful thing to be able to do if one were to introduce a CDN…

The second interesting thing to note in the above snippet is the { Orderer = new ScriptBundleOrderer() }. This is a way for me to control the order of which my files are added to the page. Unless you use some dynamic resource loading strategy in your code, the order of which your scripts are loaded is actually important… So for this application, I created this

privateclass ScriptBundleOrderer : DefaultBundleOrderer
{
publicoverride IEnumerable<BundleFile> OrderFiles(BundleContext context, IEnumerable<BundleFile> files)
    {
        var defaultOrder = base.OrderFiles(context, files).ToList();

        var firstFiles = defaultOrder.Where(x => x.VirtualFile.Name != null&& x.VirtualFile.Name.StartsWith("~/bower_components", StringComparison.OrdinalIgnoreCase)).ToList();
        var lastFiles = defaultOrder.Where(x => x.VirtualFile.Name != null&& x.VirtualFile.Name.EndsWith("module.js", StringComparison.OrdinalIgnoreCase)).ToList();
        var app = defaultOrder.Where(x => x.VirtualFile.Name != null&& x.VirtualFile.Name.EndsWith("app.js", StringComparison.OrdinalIgnoreCase)).ToList();

        var newOrder = firstFiles
            .Concat(defaultOrder.Where(x => !firstFiles.Contains(x) && !lastFiles.Contains(x) && !app.Contains(x)).Concat(lastFiles).ToList())
            .Concat(lastFiles)
            .Concat(app)
            .ToList();

return newOrder;
    }
}

It is pretty much a quick hack to make sure that all libraries from the bower_components folder is added first, then all JavaScript files that are not in that folder and do not end with “module.js” or “app.js”, then all files ending with “module.js” and finally “app.js”. This means that my SPA is loaded in the correct order. It all depends on the naming conventions one use, but for me, and this solution, this will suffice…

Next it is time to add my styles. It looks pretty much identical, except that I add Less files instead of JavaScript files…and I use a StyleBundle instead of a ScriptBundle

publicstaticvoid RegisterBundles(BundleCollection bundles)
{
    ..
    bundles.Add(new StyleBundle("~/styles", ConfigurationManager.AppSettings["Optimization.PathPrefix"] + "styles.min.css")
        .Include("~/bower_components/bootstrap/less/bootstrap.less")
        .IncludeDirectory("~/Content/Styles", "*.less", true));
    ...
}

As you can see, I supply a CDN-path here as well. However, I don’t need to mess with the ordering this time as my application only has 2 less files that are added in the correct order. But if you hade a more complex scenario, you could just create another orderer.

And yes, I include Less files instead of CSS. I will then use less.js, which is included in the script bundle, to convert it to CSS in the browser…

Ok, that’s it for the RegisterBundles method, except for 2 more lines of code

publicstaticvoid RegisterBundles(BundleCollection bundles)
{
    ...
    bundles.UseCdn = bool.Parse(ConfigurationManager.AppSettings["Optimization.UseBundling"]);
    BundleTable.EnableOptimizations = bundles.UseCdn;
}

These two lines makes it possible to configure whether or not to use the bundled and minified versions of my scripts and styles by setting a value called Optimization.UseBundling in the web.config file. By default, the optimization stuff will serve unbundled files if the current compilation is set to debug in web.config, dynamically bundled files if set to “not debug” and the CDN path if UseCdn is set to true. In my case, I short circuit this and make it all dependent on the web.config setting…

Tip: To be honest, I generally add a bit more config in here, making it possible to not only use the un-bundled and minified JavaScript files or the bundled and minified versions. Instead I like being able to use bundled but not minified files as well. This can make debugging a lot easier some times. But that is up to you if you want to or not…I just kept it simple here…

Ok, now all the bundling is in place! Now it is just a matter of adding it to the actual page. Something that is normally not a big problem. You just call Styles.Render() and Scripts.Render(). Unfortunately, Iwhen adding Less files instead of CSS, the link tags that added need to have a different type defined. So to solve that, I created a little helper class called LessStyles. It looks like this

publicstaticclass LessStyles
{
privateconststring LessLinkFormat = "<link rel=\"stylesheet/less\" type=\"text/css\" href=\"{0}\" />";

publicstatic IHtmlString Render(paramsstring[] paths)
    {
if (!bool.Parse(ConfigurationManager.AppSettings["Optimization.UseBundling"]) & HttpContext.Current.IsDebuggingEnabled)
        {
return Styles.RenderFormat(LessLinkFormat, paths);
        }
else
        {
return Styles.Render(paths);
        }
    }
}

All it does is verifying whether or not it is set to use bundling or not, and if it isn’t, it renders the paths by calling Styles.RenderFormat(), passing along a custom format for the link tag, which sets the correct link type. This will then be picked up by the less.js script, and converted into CSS on the fly.

Now that I have that helper, it is easy to render the scripts and styles to the page like this

@using System.Web.Optimization
@using DarksideCookie.AspNet.MSBuild.Web.Helpers
<!DOCTYPEhtml>

<htmldata-ng-app="MSbuildDemo">
<head>
<title>MSBuild Demo</title>
    @LessStyles.Render("~/styles")
</head>
<bodydata-ng-controller="welcomeController as ctrl">
<div>
<h1>{{ctrl.greeting('World')}}</h1>
</div>
    @Scripts.Render("~/scripts")
</body>
</html>

There you have it! My application is done… Running this in a browser, with Optimization.UseCdn set to false, returns

<!DOCTYPEhtml>

<htmldata-ng-app="MSbuildDemo">
<head>
<title>MSBuild Demo</title>
<linkrel="stylesheet/less"type="text/css"href="/bower_components/bootstrap/less/bootstrap.less"/>
<linkrel="stylesheet/less"type="text/css"href="/Content/Styles/site.less"/>

</head>
<bodydata-ng-controller="welcomeController as ctrl">
<div>
<h1>{{ctrl.greeting('World')}}</h1>
</div>
<scriptsrc="/bower_components/angularjs/angular.js"></script>
<scriptsrc="/bower_components/less/dist/less.js"></script>
<scriptsrc="/Content/Scripts/Welcome/GreetingService.js"></script>
<scriptsrc="/Content/Scripts/Welcome/WelcomeController.js"></script>
<scriptsrc="/Content/Scripts/Welcome/Module.js"></script>
<scriptsrc="/Content/Scripts/Welcome/App.js"></script>

</body>
</html>

As expected it returns the un-bundled Less files and JavaScript files. And setting Optimization.UseCdn to true returns

<!DOCTYPEhtml>

<htmldata-ng-app="MSbuildDemo">
<head>
<title>MSBuild Demo</title>
<linkhref="/Content/dist/styles.min.css"rel="stylesheet"/>
</head>
<bodydata-ng-controller="welcomeController as ctrl">
<div>
<h1>{{ctrl.greeting('World')}}</h1>
</div>
<scriptsrc="/Content/dist/scripts.min.js"></script>
</body>
</html>

Bundled JavaScript and CSS from a folder called dist inside the Content folder.

Ok, sweet! So my bundling/CDN hack thingy worked. Now I just need to make sure that I get my scripts.min.js and styles.min.css created as well. To do this, I’m going to turn to node, npm and Gulp!

I use npm to install my node dependencies, and just like with Bower, I use the “--save” flag to make sure it is saved to the package.json file for restore in the future…and on the buildserver…

In this case, there are quite a few dependencies that needs to be added… In the end, my package.json looks like this

{
  ...
"dependencies": {
"bower": "~1.3.12",
"del": "^1.2.0",
"gulp": "~3.8.8",
"gulp-concat": "~2.4.1",
"gulp-less": "~1.3.6",
"gulp-minify-css": "~0.3.11",
"gulp-ng-annotate": "^0.5.3",
"gulp-order": "^1.1.1",
"gulp-rename": "~1.2.0",
"gulp-typescript": "^2.2.0",
"gulp-uglify": "~1.0.1",
"gulp-watch": "^1.1.0",
"merge-stream": "^0.1.8",
"run-sequence": "^1.1.1"
  }
}

But it all depends on what you are doing in your build…

Now that I have all the dependencies needed, I add a gulfile.js to my project.

It includes a whole heap of code, so I will just write it out here, and then try to cover the main points

var gulp = require('gulp');
var typescript = require('gulp-typescript');
var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var rename = require('gulp-rename');
var less = require('gulp-less');
var minifycss = require('gulp-minify-css');
var watch = require('gulp-watch');
var del = require('del');
var runSequence = require('run-sequence');
var ngAnnotate = require('gulp-ng-annotate');
var mergeStream = require('merge-stream');
var order = require('gulp-order');
var exec = require('child_process').exec;

var settings = {
    contentPath: "./Content/",
    buildPath: "./Content/build/",
    distPath: "./Content/dist/",
    bowerPath: "./bower_components/",
    bower: {
"bootstrap": "bootstrap/dist/**/*.{map,css,ttf,svg,woff,eot}",
"angular": "angularjs/angular.js"
    },
    scriptOrder: [
'**/angular.js',
'**/*Service.js',
'!**/*(App|Module).js',
'**/*Module.js',
'**/App.js'
    ],
    stylesOrder: [
'**/normalize.css',

'**/*.css'
    ]
}

gulp.task('default', function (callback) {
    runSequence('clean', 'build', 'package', callback);
});
gulp.task('Debug', ['default']);
gulp.task('Release', ['default']);

gulp.task('build', function () {
var lessStream = gulp.src(settings.contentPath + "**/*.less")
        .pipe(less())
        .pipe(gulp.dest(settings.buildPath));

var typescriptStream = gulp.src(settings.contentPath + "**/*.ts")
        .pipe(typescript({
            declarationFiles: false,
            noExternalResolve: false,
            target: 'ES5'
        }))
        .pipe(gulp.dest(settings.buildPath));

var stream = mergeStream(lessStream, typescriptStream);

for (var destinationDir in settings.bower) {
        stream.add(gulp.src(settings.bowerPath + settings.bower[destinationDir])
                    .pipe(gulp.dest(settings.buildPath + destinationDir)));
    }

return stream;
});

gulp.task('package', function () {
var cssStream = gulp.src(settings.buildPath + "**/*.css")
        .pipe(order(settings.stylesOrder))
        .pipe(concat('styles.css'))
        .pipe(gulp.dest(settings.buildPath))
        .pipe(minifycss())
        .pipe(rename('styles.min.css'))
        .pipe(gulp.dest(settings.distPath));

var jsStream = gulp.src(settings.buildPath + "**/*.js")
        .pipe(ngAnnotate({
            remove: true,
            add: true,
            single_quotes: true,
            sourcemap: false
        }))
        .pipe(order(settings.scriptOrder))
        .pipe(concat('scripts.js'))
        .pipe(gulp.dest(settings.buildPath))
        .pipe(uglify())
        .pipe(rename('scripts.min.js'))
        .pipe(gulp.dest(settings.distPath));

return mergeStream(cssStream, jsStream);
});


 
gulp.task('clean', function () {
    del.sync([settings.buildPath, settings.distPath]);
});

It starts with a “default” task which runs the “clean”, “build” and “packages” tasks in sequence. It then has one task per defined build configuration in the project. In this case “Debug” and “Release”. These in turn just run the “default” in this case, but it makes I possible to run different tasks during different builds.

Note: Remember that task names are case-sensitive, so make sure that the task names use the same casing as the build configurations in you project

The “build” task transpiles Less to CSS and TypeScript to JavaScript and puts them in the folder defined as “/Content/build/”. It also copies the defined Bower components to this folder.

The “package” task takes all the CSS and JavaScript files generated by the “build” task, bundles them into a styles.css and a scripts.js file in the same folder. It then minifies them into a styles.min.css and scripts.min.css file, and put them in a folder defined as /Content/dist/”. It also makes sure that it is all added in the correct order. Just as the BundleConfig class did.

The “clean” task does just that. It cleans up the folders that the other tasks have created. Why? Well, it is kind of a nice feature to have…

Ok, now I have all the Gulp tasks needed to generate the files needed, as well as clean up afterwards. And these are easy to run from the command line, or using the Task Runner Explorer extension in Visual Studio. But this will not work on a buildserver unfortunately… 

Note: Unless you can get your buildserver to run the Gulp stuff somehow. In TFS 2015, and a lot of other buildservers, you can run Gulp as a part of the build. In TFS 2013 for example, this is a bit trickier…

So how do we get it to run as a part of the build (if we can’t have the buildserver do it for us, or we just want to make sure it always runs with the build)?

Well, here is where it starts getting interesting! One way is unload the .csproj file, and start messing with the build settings in there. However, this is not really a great solution. It gets messy very fast, and it is very hard to understand what is happening when you open a project that someone else has created, and it all of the sudden does magical things. It is just not very obvious to have it in the .csproj file… Instead, adding a file called <PROJECTNAME>.wpp.targets, will enable us to do the same things, but in a more obvious way. This file will be read during the build, and work in the same way as if you were modifying the .csproj file.

In my case the file is called DarksideCookie.AspNet.MSBuild.Web.wpp.targets. The contents of it is XAML, just like the .csproj file. It has a root element named Project in the http://schemas.microsoft.com/developer/msbuild/2003 namespace. Like this

<?xmlversion="1.0"encoding="utf-8"?>
<ProjectToolsVersion="4.0"xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<ExcludeFoldersFromDeployment>.\Content\build\</ExcludeFoldersFromDeployment>
</PropertyGroup>
</Project>

Next, is the definition of the tasks, or targets as they are called in a targets file. They define what should happen, and when.

A target is defined by a Target element. It includes a name, when to run, and whether or not it depends on any other target before it can run, and if there are any conditions to take into account. Inside the Target element, you find the definition of what should happen.

In this case, I have a a bunch of different targets, so let’s look at what they do

The first on is one called “BeforeBuild”.

Note: Oh…yeah…by naming your target to some specific names, they will be run at specific times, and do not need to be told when to run… In this case, it will run before the build…

<TargetName="BeforeBuild">
<MessageText="Running custom build things!"Importance="high"/>
</Target>

All it does is print out that it is running the custom build. This makes it easy to see if everything is running as it should by looking at the build output.

The next target is called “RunGulp”, and it will do jus that, run Gulp as a part of the build.

<TargetName="RunGulp"AfterTargets="BeforeBuild"DependsOnTargets="NpmInstall;BowerInstall">
<MessageText="Running gulp task $(Configuration)"Importance="high"/>
<ExecCommand="node_modules\.bin\gulp $(Configuration)"WorkingDirectory="$(ProjectDir)"/>
<OnErrorExecuteTargets="DeletePackages"/>
</Target>

As you can see, it is set to run after the target called “BeforeBuild”, and depends on the targets called “NpmInstall” and “BowerInstall”. This will make sure that the “NpmInstall” and “BowerInstall” targets are run before this target. In the target, it prints out that it is running the specified Gulp task, which once again simplifies debugging things using the build output and logs, and the runs Gulp using an element called “Exec”. “Exec” is basically like running a command in the command line. In this case, the “Exec” element is also configured to make sure the command is run in the correct working directory. And if it fails, it executes the target called “DeletePackages”.

Ok, so that explains how the Gulp task is run. But what do the “NpmInstall” and “BowerInstall” targets do? Well, they do pretty much exactly what they are called. They run “npm install” and “bower install” to make sure that all dependencies are installed. As I mentioned before, I don’t check in my dependencies. Instead I let my buildserver pull them in as needed.

Note: Yes, this has some potential drawbacks. Things like, the internet connection being down during the build, or the bower or npm repos not being available, and so on. But in most cases it works fine and saves the source control system from having megs upon megs of node and bower dependencies to store…

<TargetName="NpmInstall"Condition="'$(Configuration)' != 'Debug'">
<MessageText="Running npm install"Importance="high"/>
<ExecCommand="npm install --quiet"WorkingDirectory="$(ProjectDir)"/>
<OnErrorExecuteTargets="DeletePackages"/>
</Target>


<TargetName="BowerInstall"Condition="'$(Configuration)' != 'Debug'">
<MessageText="Running bower install"Importance="high"/>
<ExecCommand="node_modules\.bin\bower install --quiet"WorkingDirectory="$(ProjectDir)"/>
<OnErrorExecuteTargets="DeletePackages"/>
</Target>

As you can see, these targets also define some conditions. They will only run if the current build configuration is something else than “Debug”. That way they will not run every time you build in VS. Instead it will only run when building for release, which is normally done on the buildserver.

The next two targets look like this

<TargetName="DeletePackages"Condition="'$(Configuration)' != 'Debug'"AfterTargets="RunGulp">
<MessageText="Downloaded packages"Importance="high"/>
<ExecCommand="..\tools\delete_folder node_modules"WorkingDirectory="$(ProjectDir)\"/>
<ExecCommand="..\tools\delete_folder bower_components"WorkingDirectory="$(ProjectDir)\"/>
</Target>

<TargetName="CleanGulpFiles"AfterTargets="Clean">
<MessageText="Cleaning up node files"Importance="high"/>
<ItemGroup>
<GulpGeneratedInclude=".\Content\build\**\*"/>
<GulpGeneratedInclude=".\Content\dist\**\*"/>
</ItemGroup>
<DeleteFiles="@(GulpGenerated)"/>
<RemoveDirDirectories=".\Content\build;.\Content\dist;"/>
</Target>

The “DeletePackages” target is set to run after the “RunGulp” target. This will make sure that it removes the node_modules and bower_components folders when done. However, once again, only when not building in “Debug”. Unfortunately, the node_modules folder can get VERY deep, and cause some problems when being deleted on a Windows machine. Because of this, I have included a little script called delete_folder, which will take care of this problem. So instead of just deleting the folder, I call on that script to do the job.

The second target, called “CleanGulpFiles”, deletes the files and folders generated by Gulp, and is set to run after the target called “Clean”. This means that it will run when you right click your project in the Solution Explorer and choose Clean. This is a neat way to get rid of generated content easily.

In a simple world this would be it. This will run Gulp and generate the required files as a part of the build. So it does what I said it would do… However, if you use MSBuild or MSDeploy to create a deployment package, or deploy your solution to a server as a part of the build, which you normally do on a buildserver, they newly created files will not automatically be included. To get this solved, there is one final target called “AddGulpFiles” in this case.

<TargetName="AddGulpFiles"BeforeTargets="CopyAllFilesToSingleFolderForPackage;CopyAllFilesToSingleFolderForMsdeploy">
<MessageText="Adding gulp-generated files"Importance="high"/>
<ItemGroup>
<CustomFilesToIncludeInclude=".\Content\dist\**\*.*"/>
<FilesForPackagingFromProjectInclude="%(CustomFilesToInclude.Identity)">
<DestinationRelativePath>.\Content\dist\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</FilesForPackagingFromProject>
</ItemGroup>
<OnErrorExecuteTargets="DeletePackages"/>
</Target>

This target runs before “CopyAllFilesToSingleFolderForPackage” and “CopyAllFilesToSingleFolderForMsdeploy”, which will make sure that the defined files are included in the deployment.

In this case, as all the important files are added to the “/Content/dist/” folder, all we need to do is tell it to include all files in that folder…

That is it! Switching the build over to “Release” and asking VS to build, while watching the Output window, will confirm that our targets are running as expected. Unfortunately it will also run “npm install” and “bower install”, as well as delete the bower_components and npm_modules folders as part of the build. So once you have had the fun of watching it work, you will have to run those commands manually again to get your dependencies back…

And if you want to see what would actually going to be deployed to a server, you can right-click the project in the Solution Explorer and choose Publish. Publishing to the file system, or to a WebDeploy package, will give you a way to look at what files would be sent to a server in a deployment scenario.

In my code download, the publish stuff is set to build either to the file system or a web deploy package, in a folder called DarksideCookie on C:. This can obviously be changed if you want to…

As usual, I have created a sample project that you can download and play with. It includes everything covered in this post. Just remember to run “npm install” and “bower install” before trying to run it locally.

Code available here: DarksideCookie.AspNet.MSBuild.zip (65.7KB)