Integrating Gulp into your TFS builds and Web Deploy

46
March 15, 2015 // build

As I’m starting to dabble with Gulp, it is wonderful to see support for it already well underway within Visual Studio.  Kudos to Microsoft/Mads Kristensen for embracing a community solution and making it compatible with the IDE we love.  There’s a lot of wonderful articles out there by Scott Hanselman, John Papa, Dave Paquette and others talking about what these tools are and how to get them up and running in Visual Studio.  I’m not going to recap any of it here as it’s already been well stated. 🙂

What I didn’t find out there was how do I take Gulp the rest of the way out to a production server?  Specifically, how can I get Gulp to run via MSbuild on a TFS2013 CI build server, and then make sure that output can be picked up and deployed via Web Deploy?  Works on my machine is great, but I <3 my CI and automated deployments.  Here’s what I learned that can take us the rest of the way to Works in my build and deployment pipeline.

Note: With the solution below, you don’t need to have the Task Runner Explorer extension by Mads Kristensen as referenced in the articles – but it can still be pretty helpful for visualizing output or ad hoc runs.

Phase 1: Conquer the universe

Whoa, whoa, slow down.  Can’t I just simply use pre-build events or post-build events?

Excellent question, let’s try it.

Pre-build events (spoiler: too soon)

Pre-build events

The commands are pretty straight-forward. We make sure we’re in the project directory on the build server, we call npm install to pull down our dependencies, and then we call out to Gulp.  This looks great, and it seems to work locally.. sometimes.

Wait, sometimes?

In my case I’m utilizing TypeScript.  What’s happening is that on a rebuild (or clean build), TypeScript has removed all the JavaScript files (or they haven’t been built yet) and there’s nothing for Gulp to find.  On a second build, Gulp finds the old JavaScript output and then it works.  Two compile passes isn’t going to fly.

Alright, pre-build events are out – let’s wait until after the build so we know all the TypeScript->JavaScript compilation is complete.

Post-build events (spoiler: too late)

Post-build events

Same setup, we’ll just try it a little later.

Now our Gulp output is getting generated (yay!).. but it happens too late to be picked up by Web Deploy which actually occurs prior to post-build (boo!).

So pre-build is too soon, and post-build is too late.  It’s time to turn to custom targets in our .csproj file.  Trust me, it sounds scarier than it really is – we’re devs, we’ve got this. 😉

Extending our .csproj with custom targets (just right)

Manually edit your .csproj file and at the bottom (must be after all Import lines) we’re going to add some new content.

  <PropertyGroup>
    <CompileDependsOn>
      $(CompileDependsOn);
      GulpBuild;
    </CompileDependsOn>
  </PropertyGroup>
  <Target Name="GulpBuild" DependsOnTargets="CompileTypeScript">
    <Exec Command="npm install" />
    <Exec Command="gulp" />
  </Target>

What we have done here is specify a PropertyGroup to extend the existing CompileDependsOn target to add our own custom target called ‘GulpBuild’.  On our target, we can specify other targets that need to happen first – such as CompileTypeScript.  Now we know that TypeScript compilation will happen first, and we’re in the pipeline long before deployment occurs.  For more details, check out this article on MSDN or Sayed Ibrahim Hashimi’s wonderful book on MSBuild.

What about triggering a gulp cleanup script when we clean our solution in Visual Studio?  It’s a very similar extension:

  <PropertyGroup>
    <CompileDependsOn>
      $(CompileDependsOn);
      GulpBuild;
    </CompileDependsOn>
    <CleanDependsOn>
      $(CleanDependsOn);
      GulpClean
    </CleanDependsOn>
  </PropertyGroup>
  <Target Name="GulpBuild" DependsOnTargets="CompileTypeScript">
    <Exec Command="npm install" />
    <Exec Command="gulp" />
  </Target>
  <Target Name="GulpClean">
    <Exec Command="gulp clean" />
  </Target>

Fantastic.  At this point, we’re now able to run Gulp both locally within Visual Studio as well as part of our build process!

Hooking into Web Deploy

So what’s left?  Well, by default Web Deploy isn’t going to pick up any of our output from the Gulp pipeline since it isn’t a part of the project.  So we need to explicitly point it at our build output.  There’s a couple helpful articles that talk about the approach in more detail on ASP.net and by Sam Stephens.

  <PropertyGroup>
    <CopyAllFilesToSingleFolderForPackageDependsOn>
      $(CopyAllFilesToSingleFolderForPackageDependsOn);
      CollectGulpOutput;
    </CopyAllFilesToSingleFolderForPackageDependsOn>
    <CopyAllFilesToSingleFolderForMsdeployDependsOn>
      $(CopyAllFilesToSingleFolderForMsdeployDependsOn);
      CollectGulpOutput;
    </CopyAllFilesToSingleFolderForMsdeployDependsOn>
  </PropertyGroup>
  <Target Name="CollectGulpOutput">
    <ItemGroup>
      <_CustomFiles Include="build\**\*" />
      <FilesForPackagingFromProject Include="%(_CustomFiles.Identity)">
        <DestinationRelativePath>build\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
      </FilesForPackagingFromProject>
    </ItemGroup>
    <Message Text="CollectGulpOutput list: %(_CustomFiles.Identity)" />
  </Target>

This is a bit harder to read, but in essence again we are extending existing targets with our own custom target. That target is collecting any files that exist in a ‘build’ folder (you can replace the 2 occurrences of ‘build’ with ‘dist’ or whatever folder convention you use), and synchronizing them across into the deployment package with the same folder structure.

Bringing it all together

Combining those pieces we’ve now got a project setup that will run Gulp both locally and on the build server, is compatible with TypeScript generating JavaScript, and is compatible with Web Deploy. If you’re raising your arms in triumph here.. thanks for joining me. 😉

  <PropertyGroup>
    <CompileDependsOn>
      $(CompileDependsOn);
      GulpBuild;
    </CompileDependsOn>
    <CleanDependsOn>
      $(CleanDependsOn);
      GulpClean
    </CleanDependsOn>
    <CopyAllFilesToSingleFolderForPackageDependsOn>
      $(CopyAllFilesToSingleFolderForPackageDependsOn);
      CollectGulpOutput;
    </CopyAllFilesToSingleFolderForPackageDependsOn>
    <CopyAllFilesToSingleFolderForMsdeployDependsOn>
      $(CopyAllFilesToSingleFolderForMsdeployDependsOn);
      CollectGulpOutput;
    </CopyAllFilesToSingleFolderForMsdeployDependsOn>
  </PropertyGroup>
  <Target Name="GulpBuild" DependsOnTargets="CompileTypeScript">
    <Exec Command="npm install" />
    <Exec Command="gulp build --mode $(ConfigurationName)" />
  </Target>
  <Target Name="GulpClean">
    <Exec Command="gulp clean" />
  </Target>
  <Target Name="CollectGulpOutput">
    <ItemGroup>
      <_CustomFiles Include="build\**\*" />
      <FilesForPackagingFromProject Include="%(_CustomFiles.Identity)">
        <DestinationRelativePath>build\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
      </FilesForPackagingFromProject>
    </ItemGroup>
    <Message Text="CollectGulpOutput list: %(_CustomFiles.Identity)" />
  </Target>

Bonus round – conditional gulp

    <Exec Command="gulp build --mode $(ConfigurationName)" />

You may have noticed I snuck something else in the final version – passing the build configuration into Gulp as a command line argument called mode. Using tools like yargs and gulpif you can then tailor your Gulp pipeline (e.g. not running uglify in ‘Debug’ mode).  If you want more details please post a comment.

Other approaches

After posting this article, I ran into two other approaches solving the same problem in slightly different ways.  All three of our posts have been in March, so clearly this is the month for automating builds. 😉  Here’s links to those alternative ways:

Summary

By utilizing custom targets in our .csproj file, we’re able to utilize npm and Gulp consistently both on our local Visual Studio instance and on our TFS build server.  We can also make sure our generated build output is picked up by Web Deploy and lands on our servers.

Thanks again to all the referenced authors and articles for guiding me to this solution.  No promises this is the only way (or even a good way) to do it, but it works for me and hopefully gives you some ideas too. 🙂  Please chime in on the comments any thoughts, questions or improvements!

About the author

Steve Cadwallader is a software developer who geeks out on user interfaces, clean code and making things easier.

46 Comments

  1. Great post Steve! I solved the problem using PowerShell to invoke npm and gulp. I prefer to use PowerShell as much as I can for build customizations – but I also had to make changes to the csproj file for WebDeploy.

    http://colinsalmcorner.com/post/jspm-npm-gulp-and-webdeploy-in-a-teambuild

    • Thanks Colin. 🙂 I really like your approach for taking the build customizations further, and will add your link directly into the post to make it easier for others to find as well as another source.

  2. Oh that’s fantastic

    I wrote a piece which covered similar ground here: http://blog.icanmakethiswork.io/2015/02/using-gulp-in-asp-net-instead-of-web-optimization.html

    However – I submit yours is a much better approach. Particular kudos for this:

    Much better than my own (admittedly hacky) approach of reading the web.config and checking the debug flag therein. (Lest anyone worry I am aware of the distinction between the web.config debug flag and the build configuration but they were fulfilling a similar purpose in my case.)

    In summary – I intend to be moving my code closer to using some of your approaches here. Well done!

  3. Angle brackets 🙁

    The bit of missing code above should be:

    <Exec Command=”gulp build –mode $(ConfigurationName)” />

  4. Finally. This is the first article I’ve seen that actually “gets” what it takes to integrate gulp in with the rest of the build process. Task Runner Explorer is great and all, but it simply doesn’t play with the overall build system because it’s simply hooking VS events. This is the only way to do this “right”. Then, as you’ve pointed out, the real gotcha is the fitting in with the web deployment stuff and you’ve done that very well too. I usually just introduce new items into the mix dynamically from my targets and then I don’t have to know anything about web deploy at all, it just sees them as if they were part of the original project. You seem to have gone deeper with the FilesForPackagingFromProject which is cool cause you technically get a little more control over things that way.

    Great stuff.

  5. Oops, that should have read:

    “I usually just introduce <Content /> items into the mix dynamically …”

  6. I’ve now used this approach myself and really like it. The thing I hadn’t expected (but really appreciate) is that the gulp output now shows up inside the build output. This makes a lot of sense and it’s much nicer than the previous scenario where I jumped back and forth between Visual Studio’s build output window and the task runner explorer.

    Thanks again chap!

  7. Oh and I just read Drew’s comment above. I was using the dynamic content approach for including gulp built artefacts in my project (in fact I still am). However, AfterBuild inclusion doesn’t play well with TypeScript when it comes to deployment. So I ended up using the same approach that you are using. I think we both half-inched it off of Sayed.

    So to avoid deployment troubles I’d stick with what you’ve got. If interested in my travails with the After Build you can see here:

    http://help.appveyor.com/discussions/problems/1429-afterbuild-not-respected-for-webdeploy

    Feodor set me on the right path…

  8. Thanks for this article, it has really helped us solve our issue with deployment builds. Is there a way to configure the build so that the bundling task does not run on local development builds? Our bundle task takes 20 seconds every time we build our solution.

    • I’m glad to hear it has helped out. Yes, what I’ve done for conditional gulp statements is a combination of using yargs and gulpif.

      Here’s a code excerpt:


      var gulpif = require('gulp-if');
      var yargs = require('yargs');

      var mode = yargs.argv.mode;
      var isLocalBuild = 'Debug' === mode;
      ...
      .pipe(gulpif(!isLocalBuild, uglify()))

      When you invoke gulp, you’ll want to pass the build mode and then the conditional logic will run or not. In the post I show this as:


      <Exec Command="gulp build --mode $(ConfigurationName)" />

      Hope it helps!

  9. Thanks for the excellent post Steve!

    Have you run into the 260 character limit with this approach at all? I’m running into this issue when TeamCity is doing an MSBuild.

    • You’re very welcome. 🙂

      Yes, I have run into the npm on Windows 260 character limit frequently. My ugly workaround has been to add the dependency of my dependencies to the package.json (e.g. if I need project X and it depends on project Y, I add project Y to my dependencies). You may also have a little luck using the ‘npm dedupe’ command, but that only kicks in if two different projects have the same dependency. Rumor has it future versions of node/npm are supposed to have a flatter hierarchy, more like bower.

      • Thanks Steve, that’s super helpful! I agree it’s not the prettiest but hopefully the next version of node/npm will make this easier!

  10. Pingback: MVC + Angular + Less + Gulp + TFS + VSO | miczdem

  11. Hi guys

    Trying to use this approach, I find that every time I put the custom sections into my project file, it causes vs to hang when i open the solution?? I remove them, no problem, put them back in again, problem reappears. I try various combinations of the configs, at the moment, i dont bother with clean etc, just build and bundling of files – and i am continually experiencing VS crashing – any ideas???

  12. So I solved the issue of VS hanging on load – I am experimenting with SmartAdmin 1.7 Beta angular project, and in the gulp “default” build task, there was a call to gulp.watch(etcetc) which caused Visual Studio to hang when opening the solution, even when NOT triggering a build!

    I created a new gulp task “MsBuild” which has all the previous gulp tasks (minify, copy, etc etc as required) EXCEPT the watch task… and so several hours and much frustration later, it all works perfectly. Hope this helps someone else.

    • I’m glad to hear you got it figured out, and thanks for sharing it back. 🙂

      One other thing to share on the topic, is that the approach in this blog post does result in the commands (npm install, etc.) all running when the solution is initially opened. So the first load in a new workspace will run slower as those commands are all executed by Visual Studio (you can watch node running via a tool like Process Explorer).

  13. Steve – great post. How are you handling node package paths that are greater than 260 characters?

  14. Clever approach. This Windows limitation creates a lot of outside the box thinking. You’re taking advantage of how nodes cumbersome dependency management works, Colin is modifying the build template and the approach I have thought about implenenting is a simple console app that recursively deletes the packages folder that we’d run before queuing the build with the limitation being we would to add the app to a scheduled task for scheduled builds and of course giving permissions to the build server. I really wish TF Build let you run scripts prior to the workflow without modifying the template. Pre process arguments for example to compliment pre build arguments. This would allow us to plug a simple script without getting into the non trivial world of windows workflow editing.

    • TFS2013 and newer templates do allow you to specify pre-build, post-build, pre-test, post-test and other extension points that weren’t available in TFS2012 and earlier build templates.

  15. Steve thanks for the quick reply – we are using 2013 but I was under the impression the build process tries to delete eveything in the builds folder (we are doing a clean build by setting Clean workspace to true) prior to the prebuild build script being fired and that’s why Colin had to modify the actual build process template.

    I know it will fire before msbuild compiles the code but I’m not sure about cleaning the workspace.

    Am I misunderstanding the pre build arguments?

    All that being said I’m going to test your solution tomorrow.

    Again thanks for the quick reply.

    • Ok, I understand now that you’re trying to solve the problem by controlling how the folder is cleaned up prior to files being pulled from source control. You’re right, the pre-build extension points I was referencing run later in the process than that. I’ve been more focused on preventing the long paths from being created than forcing them to get deleted after the exist, so unfortunately I’m not sure about extension points that would work for you. Good luck on your test.

  16. Great article. Do you know if this method is this still necessary when using TFS 2015/VS 2015 on the build agent?

    • Usually on the build agent, having Visual Studio installed just sets up the environment and command line tools. The actual compilation happens with MSBuild outside of Visual Studio – so there’s no way for extensions like the Task Runner Explorer to kick in and invoke gulp scripts during automated builds.

      So yes, to my knowledge, it’s still necessary with TFS 2015.. but I’d be happy if anybody could prove me wrong!

  17. The only actually useful article I’ve yet to see on integrating Gulp tasks with VS deployment.

    BTW, why the underscore in ?

  18. Great solution! Just helped us to review some of our problems with deployment and set it to a one click process! Thanks!

  19. Awesome article. One issue that several people mentioned and I experienced is that this causes VS to hang often and for long periods of time, especially when opening solution and also when working with TFS. Process Monitor shows a lot of recursive work with node.exe process accessing node_modules, and shows its calling process is the gulp build command. I moved the gulp build command out to a prebuild/postbuild event as you showed (I’m not relying on VS for typescript as gulp can handle that) and it resolved my performance issues.

    • Thanks, and I appreciate the extra detail you’ve offered. Yes, I’ve seen the performance hits as Visual Studio is triggered on startup with this approach to wait for node to run its installs, download packages, etc.

      I would think that moving to a pre-build event would simply change the timing of when that initial node hit occurs, but it’s probably more comfortable to wait during a build than during solution open and possibly there’s a little less context switching vs. solution open. In your scenario where you don’t need TypeScript compiling – sounds like a better approach!

  20. Thanks for the great post!

    I am however getting the following error:
    The command “gulp build” exited with code 1.

    Has anyone come across this problem?

    I do not have DependsOnTargets=”CompileTypeScript” on the GulpBuild Target as I am not using TS.
    Is Exec running the local gulp node module downloaded by npm or is it running the global gulp which I have installed for the service account running the TFS Build Host service?
    Do any one know how I get the log output of npm and gulp in TFS build log?

    • You’re welcome. 🙂

      It doesn’t have to be dependent on TypeScript, but be sure you have some other DependsOnTargets specified so it knows when to run in the build process.

      Exec is basically just a command line run, so I would expect the same instance to run as when you run at the command line.

      Have you turned up the build log verbosity to detailed?

  21. It might be worth mentioning that VSTS now has handy, easily-configurable build steps for running npm and gulp. I was able to get the build set up without ever needing to mess with my csproj file.

    • Cool, glad to hear it. VSTS builds are a drastically better experience than the old TFS workflow based approaches. Do you use Task Runner Explorer then locally to execute all the npm/gulp events before build? Part of the reason for the approach I took was to ensure that the same steps were happening locally as on the build server.

  22. Something to mention that I’ve discovered.

    If you use CompileDependsOn, that runs whenever you add a file to a project as well. Having Gulp build my css and javascript whenever I added a file to the project took forever and would cause VS2015 to freeze up. All I changed was CompileDependsOn to target AfterBuild and that resolved my problems and still bundled up all the files for msdeploy.

  23. Hi Steve

    Can you explain the process for automating gulp for building with TeamCity.
    I am using VCS as TFS and having Team City Continuous Integration.
    As of now I am manually running Gulp in Visual Studio.

    Is there any process that Team City take care of gulp?

  24. Hi Steve,
    Quick question for you. We’re running a similar setup via a build.proj, pre/post build settings, etc. I’m running into an issue where a gulp plugin error is not being picked up by MSBuild. This leads to MSBuild saying the build completed with 0 errors even though errors are present in the log.

    Any thoughts?

    Regards

    • Hi Owen –

      If you’re using pre/post build events, then are you using the “call gulp” style syntax? If so you’ll need to add some code to catch the return of that call and determine if the exit code was an error.

      Hope it helps,
      -Steve

  25. Hi Steve,
    I have added the code under “Bringing It All Together” section into my .csproj file at the end. But, when the PowerShell script is used to deploy the code files to deployment server the gulp files are not uploaded into it. Please help me for the same.

    Thanks and Regards,
    Avinash.

Leave a Comment