Swapping Out Sprockets For Gulp

Without revolt: this is, and is not, Cressid.
Within my soul there doth conduce a fight
Of this strange nature that a thing inseparate
Divides more wider than the sky and earth

Ah Sprockets, you made asset management a thing and you made it easy, why would anyone want to use anything else? Well, if you ever had to build anything which tried to extend or interface with Sprockets you might not be so overwhelmed by it, if you’ve been writing Javascript on Node and can’t go back to shoving everything into the global namespace and your idea of reuse is more than cut and paste, or maybe you want more fine control of your asset publication process including testing and linting, or even if you just want to cut asset compilation down to a few seconds.

Sprockets is awesome, one of the best off the shelf solutions I’ve seen, but the surge in javascript tools is not particularly easy to incorporate inside Rails, even with ExecJS or RubyRacer. Sometimes you need to go native, especially when more and more of your UI is in javascript and you’d like to make some effort to modularize code.

So what do we need to do? Well, we need to move files around and manipulate their contents, then we need to handle accessing digest appended files and the other tasks Rails built-in helpers do now. I am going to split the work up into multiple parts, this article will focus on building a replacement pipeline and looking at those results working in the browser.

What Is Gulp?

It’s Rake with a Stream paradigm. If you’re not clear on streams, think of a flow of data, like an mp4 playing remotely, only you can modify it in place, filter it, combine it with others and do pretty much any editing you want. No temporary files, no pausing, and running lightning fast in Node.

What can we do with Gulp? It’s just a toolset with a stream paradigm, so just about anything but streams really shine at manipulating files for conversion, translation or minification; so perfect to build your own asset pipeline.

Why bother reading this, I can just get a recipe can’t I? Of course, though I find the recipes and brief tutorials really don’t go into enough detail to get anything real done. The code isn’t complex, it’s brutally simple, but where I found difficulty was changing my thought process and adapting from structured build processes like Ant, Maven or Rake, to asynchronous Javascript and Gulp streams to get things done efficiently and clearly.

A Fairly Thorough Pipeline

I don’t know about you, but my starting point with Rails is to create an app, add in the gems I use every time, hook up bootstrap and copy across the growing mush of semi-useful javascript files I’ve used before and pretty certain I will use again. It’s a fairly mind-numbing couple of hours interspersed with me cursing the missing fonts and promising that this time I will write better set up notes, and finally get around to writing that Generator I’ve been promising these last 5 years. Specifically, I would like to

1. Use vendor css/js templates, both Bootstrap and FontAwesome
2. Modularize my javascript code, and include the common packages which make Rails tick
3. Minify, Digest and Cache-bust-enable my files

I don’t use Coffeescript, but I will show how easily its handled alongside Javascript compilation. For now I’m going to skip linting, testing and automated reloading, but I will scaffold them in for use later.

Your Opinion Counts?

After all this time being spoiled and completely ignoring assets until they break and you lose a weekend, I have had to put some thought into workflow and into how I need to organize things. One of the phenomenon I find most unique about Rails is when there is something not covered in the guides there is much hand-wringing about where to put things. Well, you’re going to hate this. You need to be opinionated and make decisions about your asset workflow. There are a few things you should keep in mind.

1. You need a node project at the same level as the directories which will hold your javascript.
2. You will be able to manipulate files outside the working path, but it is much easier doing it from the working path down.
3. You’re going to be using your files, 2 different storage mechanisms for Vendor files, a require syntax which is wholly reliant on relative paths, and you’re going to need a solution for images, fonts, css, javascripts, movies and whatever media you need to deploy.

I’ve made decisions which work best for me, you will probably find you want to tweak things, go ahead it’s encouraged. There is no single solution which is going to suit everyone and you should not feel the need to find out what everyone else is doing, there is no Rails way, you will understand the pros and cons of organization schemes by how convoluted your process is, or how ugly your requires.

Workflow

Although it can be about as exciting as sitting in on a board meeting of the local water treatment executive, you really, really need to figure out your workflow. Almost all the gulp articles and demos I read were vastly simplified compared with delivering a project. After floundering around for a while, I started thinking in terms of workflow, from asset generation to publication. Remember, you need to eventually push files to the public directory in Rails, but how you get there can depend a lot on . Like directory layout, you need to be opinionated, and you need to learn quickly from your mistakes. It’s ok though, you won’t be unpicking 50 K-LOCS of code, you are working with a few dozen lines of javascript and tweaking it until it works.

Here’s my initial thoughts on workflow,:

1. Clean up
2. Make sure the latest 3rd party components are downloaded
3. Copy 3rd party files into directories designed for building
4. Build my images, fonts, less files and javascripts
5. Move everything to a staging directory
6. Move everything to my environment’s public directory

Now in theory you can do all of this in 1 shot, and I am sure that using a staging directory gets me a black mark at the next Gulp Annual General Meeting, but that’s life. I find having a copy of what you just pushed to production will come in useful, if nothing else than to determine that you did not overwrite the company logo with cat pictures. So my workflow is different from the gulp examples I have read, but again, so what. If you want to be moving 3 javascript files to production in 3 lines you can google Gulp and find out how, this is going to be a little more involved. Hey, I said at the beginning you are going to need to be opinionated.

Start!

It should take no more than 5 minutes, I promise.

The final gulpfile and html sample are available from github

Create a new Rails project

Add this to your config/application.rb

Go through your environments/test, environments/production and environments/development files and comment out or delete all references to assets.

Go to your gemfile and comment out, or delete the following

That is all you need to disable the asset pipeline. You should go into application.rb and change the line require rails/all to require individual components and leave out sprockets, but for now it’s not important.

Go to views/layouts/application.html.erb and make the following changes

Create a controller and view. Just 1 will do, you don’t need to scaffold:

For now that’s it. Start your server and go to http://localhost:3000. You should see some text, no formatting and a missing image. Later we’ll copy/pasta some bootstrap and font-awesome sample code, but for now we’ve enough to do just to get that image up and running.

Set up Node, Bower and Gulp

If you haven’t used Node and know nothing about it, don’t worry. I won’t be asking for a legacy database API with native sql, you just need to install it and be able to run some command line tools. When you have time, do some things with Node, trust me, you’ll love it. It’s Javascript on your machine, available from the command line with 100,000 plugins of everything bored CompSci undergraduates could think they may need some day. If you’re on Linux you can use your package manager and be up and running in a few minutes, if you’re on a Mac, follow these instructions and use the Homebrew method. If you’re on Windows, pray. The last 5 years of development tools have been some sort of revenge on Wintel for destroying Apple v 1.o. I have no idea if Node, or anything other than VS run on Windows any more.

Along with Node you need to install NPM. It’s a package manager for node, very gem-esque.

Now its time to set up a directory structure, initialize Node, and start building. From the root of your application, create a directory called frontend. We will eventually have this directory structure created by the tooling.

For now, delete your vendor and app/assets directories, you won’t be using them.

You can now either copy/paste the below into a file called package.json, or from the frontend directory run npm init and give whatever answers you feel appropriate and then copy the line from devDependencies down to the penultimate line and paste into package.json

It’s a decent starter file which should contain all we need. Next, within the frontend directory run npm install. All those packages referred to above will be installed. Next run the following;

Keen eyed observers will notice that we’ve installed those two packages before. Why do we install twice? Well, the -g means install globally, that does some nice automated things to make using command line tools easy. Why global doesn’t mean the package is available globally? One of the many quirks you’re going to become familiar with with Node.

Almost finished, next run the following:

If you’ve been on retreat in Bhutan the last two years, you probably haven’t heard of Bower. It’s essentially Bundler for front end things. So, forget the hardship and torment of googling sites, pressing download buttons and opening up zip files, Bower does that for you. As should be seriously obvious, I am installing bootstrap and fontawesome. The –save directive adds those to a configuration file which we will reuse later.

Finally, let’s create the directories. Create a file called gulpfile.js and add the following to it. When you’re done, run gulp setup-directories from the command line and we should have initial setup complete.

With that, we’re finally ready to start writing our pipeline. It may seem like a lot of overhead, but one of the beautiful things about gulp is that when you’re sure you have a repeatable pipeline, it’s just a handful of lines to generate all that stuff for you.

Start With Something Easy

Ok, fonts and images. Copy some image into frontend/assets/images and name it MySample.jpg. Now it’s time to write some gulp.

You have already seen the mountain of requires you’ll find in a Gulpfile – think of straight Ruby, but you have to require every other class. Those packages will be used as we incrementally add functionality, but for now we are going to do the simple things; copy images and fonts from assets to public.

We’ve created 2 tasks, one which accumulates images, and one fonts. I have commented out the code to create a digest. A digest is a file fingerprint appended to file name, for example my_image_6a4d90f.jpg which Sprockets lovingly creates) as it will take until part 2 to create code to use them. For now, we’ll just use the regular image names. All we’ve done is create tasks which move 2 input directories to an output directory. The getAssets(name) and getVendorAssets(name) are just convenience methods I created to return a path instead of needing dozens of configuration entries.

I have left livereload hooked up, but I’m not doing anything with it. More on that much later because for now, there are a bunch more things we need to get done before thinking about continuous integration.

Now, to publish them, I will create a new publish task

That task will take the contents of my distribution directory and push them to public. It may seem like overkill at this point, but it goes back to the concept of workflow which will hopefully become apparent later. Accumulating and creating a bunch of files from many places can sometimes need a break before publishing to the application. Although Gulp has ways to skip this step, I prefer having a physical location for my distribution files.

Now, run gulp images then gulp publish. Refresh your browser, you should see the broken image replaced by a real one. Hooray, you are a living Sprocket.

Synchronicity

So we need to talk about Node very briefly. Node is like running a manufacturing plant in low earth orbit; yes the efficiency is incredible, but you need to throw out many of your tried and trusted ways of doing things; you need to work with the environment and embrace some new ways of doing the things you may have taken for granted for years.

One of the reasons Node is so fast is that many things are happening at the same time. This is not the place for an essay about asynchronous Javascript, but the paradigm you may be used to with Ruby is not applicable when you start working with Node. There are many resources on this subject, but in brief; multiple commands run in parallel, immediately and finish in no defined order….except when they don’t.

Paste this into a file called whatIsHappening.js and run it with node whatIsHappening on the command line. You should see the output as below

What happened? When we call longOperation, it takes some time to complete, while that’s going on Node continues to process your script and calls the next line which logs that the process has finished. While this is a very trivial example, this is sincerely one of the joys of Node: having to think before coding and planning how events should be orchestrated. It’s also one of the pain points which can potentially wreck choreographing workflows. You need bootstrap to be downloaded before processing, or file directories to be cleaned before running less, you need to think of event handling and not sequential running code line by line.

Except when they don’t? There are some commands which are ‘blocking’, they don’t allow Node to move on until they have completed in their entirety. Generally, these commands either have sync in the name.

While this is absolutely great for kicking off multiple compilers at once, it makes running things in sequence a little harder, and the code is about to get a little messy.

How is that opinionated workflow going?

Remember we set up Bower, it’s pulling down Bootstrap and Font Awesome. I also have my own Less files and we need to compile them all to create a single stylesheet package. I find Less works better if your codebase is together, so rather than move these files over to the Vendor directory, I will copy them to my asset directory and then run Less. This is where Workflow meets lack of orchestration and will require a re-think.

First time through, I created a copy task like below:

What’s all that doing? Well, I want 3 copy tasks to run and report when they are finished, otherwise I can’t know when to start my build task. I also want to make sure that Bower is up to date, so I add a dependency on bower to run that first. Seems logical, but the first indicator that something is wrong is that it looks like crap. I remind myself that I’m manufacturing in low earth orbit and need to stop following old habits. So its time to take a long hard look at my proposed workflow.

The theory that building takes 7 or 8 sequential steps is fine, except that paradigm will drive you nuts and make your gulpfile look hideous. There’s a build tool called Grunt designed for exactly that kind of build process. Gulp is not and does not provide the tools for it, which is why I’m using Node’s async library to achieve some semblance of sequencing. It’s time to ditch the workflow and look at what I want to do.

I want all my sources to be moved to the correct directories before building, then I want to build and deploy. I don’t need to execute every step for every build, Bootstrap are not releasing new versions every 20 minutes, so I start thinking of my build as being multiple different flows. I should not be coding for rare and frequent operations in the same flow. So, I’m going to adjust and break down my workflow from a sequential series of steps, to multiple different events working independently.

I can set up my build tasks to trigger automatically when file changes are detected, or when I run them manually, but it becomes obvious that they should have no prior dependencies except for cleaning out the dist directory beforehand.

I should not automate grabbing new versions of 3rd party libraries, and I should not update my source directories from Bower unless I am positively deciding to change something, so those both need to be out of the regular publishing workflow.

I have made a decision to separate compilation and aggregation from deployment, so those are separate flows which must not be dependent on one another.

It becomes clear, that instead of my original 6 step process, what I really need is:

Flow 1

1. Update 3rd party libraries

Flow 2

1. Copy 3rd party files to build directories

Flow 3

1. Clean dist directory
2. Build with output to dist

Flow 4

1. Deploy to public directory

So let’s revisit the task to copy over 3rd party files.

I still start these 3 tasks in parallel, but I don’t care when they finish so I don’t have to add all that extra callback scaffolding. Let’s add in the code to copy Bootstrap from Bower:

Works pretty much the same way, I also add in a task which will perform both copies, again without me caring what order or when they finish. So I trigger off a bunch of copy commands and do not care in what order or when they are completed. Note the task ‘copy-libraries’ with what looks like dependencies in sequence; they’re not. Those dependencies will run in parallel. There is a specific workaround to get 1 dependency chain working, but whenever you see code like this in Gulp, recognize that all of those tasks running independently.

Speaking of the Bower task, here’ it is;

Some integrations are a little clunky, Bower doesn’t really fit into the paradigm of streams, so Gulp is just calling it similarly to running Bower on the command line. I add in a callback so that Gulp can figure out when the task is complete, but it’s not really necessary. An empty install() command will trigger Bower to use bower.json and to install all saved dependencies. This eliminates adding yet more configuration to Gulp and allows me to use Bower as intended, running with it’s own configuration.

Lessen The Pain

So we have the fonts, images and Bower files are ready for processing. Next step is to run Less. If everything has gone well you should be able to run the following

gulp bower
gulp copy-libraries

and by magic, you have a folder structure like this

I’ll be honest, I’ve been using Sass so long I didn’t think I would want to change, but Less turned out to compile about 10 times faster, so I thought I would give it a shot. We need  master file which acts as an entr point and manages includes. Here’s the one we need for this example;

Those 4 lines will pull in my vendor files and my custom file. Sure, it’s no more and no less exciting than a sprockets manifest, but using Less (or Sass if you prefer) is incredibly flexible.

My custom Less file is remarkably simple, just enough to allow you to instantly know if its working by the revolting colors which will show up in our example page. It’s deliberately convoluted so that I can verify the Less compiler is working.

Ok, all our files are ready, time for what must be a dramatically difficult build process….

Sourcemaps, autoprefixer and the combined genius of Google, Font-Awesome and my artistic pallette. It runs on my laptop in about 8ms. Waiting for Heroku to run assets.precompile will never be the same again.

Tying It All Together

So we just need a clean task, followed by orchestrated building of less, images and fonts.

I thought this was going to be easy until I started getting deprecation errors and trying to follow a several hundred post thread in Gulp issues. Apparently, there’s no Gulp-way to have 1 task be run, followed by 2 or more others. I could run 1 with a dependency, but that’s not what I need, I really do need this

After much shaking of head it appears as though I’m forgetting my own gravity analogy and that things are different. Gulp doesn’t magically replace Javascript, it enhances it, so instead of defining everything as a task, I need to refactor somewhat and push code into functions which are then called from tasks. It’s easier demonstrated than explained:

So now Clean runs first, then images, fonts and less run in parallel afterwards. I have moved what used to be tasks into functions, and those functions are called from the build task itself.

If you run this as coded, you will notice some odd output on the console. Build never finishes; that’s because we haven’t told Gulp it’s over by invoking the callback. I’m going to leave it like this for a while, we have more important work for now, but added to the growing list of todos is making sure we invoke callbacks at the correct time.]

Let’s verify that font-awesome, bootstrap and our color pallete are all working. Run gulp build followed by gulp deploy

Copy the following into examples/index.html.erb

Reload and you should see something like the image below. An image, some colors, a sample of font-awesome and a smattering of bootstrap stuff. Not bad, and after doing this once, it’s a build system which will travel from project to project. We have forgotten one thing though, time to add in the javascript.

Javascript

I find the state of Javascript to be quite amusing. There are multiple competing ways of creating fake modules, a ton of code which was written long beforehand in single files, and a codebase growing in every possible direction with few glimmers of standards.

I am going to simulate my own javascript bouillabaisse by doing the following; I will write new code according to CommonJS, include some older style code,  some Coffeescript, plus  JQuery, JQuery-UJS and Bootstrap. Hopefully they will all work together.

Here are three javascript files and a coffescript file to put in /frontend/assets/javascripts

While we’re here, add the following at the bottom of app/views/examples/index.html.erb

So I want an application.js to do roughly the same thing as a sprockets manifest file, I want to make sure the tree of javascripts are added along with Jquery, JQuery-UJS and Bootstrap. What I really need is something to figure out the dependencies, not duplicate includes and generally turn my rubbish into something useable. Oh hai Browserify. Although not perfect, Browserify comes close to it by packaging up disparate strands of javascripts from many sources and making it run in a browser.

Let’s start with the frontend/assets/javascripts/application.js file. This is our manifest replacement.

That’s all it is. It’s ugly, it’s effective and has the comforting Javascript sameness of trying to figure out which libraries need to be shimmed, which ones need to go in order. This small sample was a combination of trial and error, then reading stacks about browserify and trying a bunch of recommended solutions which didn’t work. I suppose some parts of Javascript are beyond any utility’s ability to clean up, but as long as you remember to put Jquery first, bind it to $ and jQuery you should be mostly ok. The ugly require statement for bootstrap is because the source file is in another path and that’s just how CommonJS works. It’s definitely not as clean as Sprockets, but it works.

Adding to the todos, it’s clear that my original directory structure is less than ideal, just with Less, Javascript from 3rd parties should be added to my assets/javascripts path so that requires stop being so ugly.

Next, our gulpfile. Sadly, Browserify doesn’t immediately get along with Gulp, so there are two additional packages required to turn output from Browserify into a Gulp stream to be processed. Here’s my compilation function.

I don’t use Coffeescript, but I added one just so you can be as impressed as I am when you see the source file written in coffeescript in Chrome and set breakpoints. I still have trouble believing how smoothly this works; remember this is a bunch of Javascripts written in multiple ways, and a Coffeescript file, it’s all combined, minified and accurately accompanied by sourcemaps.

Wrapping It Up

After writing those build functions I decide to wrap as individual tasks to that I can either run the whole build or individual pieces of it.

you can then run gulp javascripts followed by  gulp deploy and refresh the browser.

Pretty neat, but that’s when it dawns on me that this can get better. Those build functions are really strategies which can be moved out of the gulpfile. I can create multiples depending on command line switches, or a configuration file. From a simple series of deployment tasks to continuous integration with linting, testing and whatever else I need, customized to individual project needs and web hosting specifics., but reusable across projects. Kind of like the way you can run rake db:migrate in any project and it does exactly what you can expect. However, those changes are for another time.

Also for another time will most probably be ditching the dist directory. It’s funny, but every preconception I began with has been challenged and altered to adapt to the tooling, there are enough options for automated testing to make any manual step little more than an inconvenience, so that will need to be addressed in the future.

Where are we

I wanted to demonstrate swapping out sprockets and we’re about halfway there. So far we have build tools to deploy code on demand, a project structure which should be reusable, and we’ve verified that assets are being correctly accumulated with some simple browser functionality. But there are some pieces missing. Automated deployment, accessing assets with their asset digest, Rails helpers and being able to handle Heroku’s rake assets:precompile without disaster. Those are for part II.

If you haven’t grabbed the files, gulpfile.js and example.html.erb are available here.