If you’ve been around modern tooling in front-end web development, you’ve likely run into task runners like Grunt and Gulp. Grunt uses a declarative way of defining tasks via a large configuration JavaScript object that gets fed into the Grunt engine. Gulp takes the more programmatic approach, using JS stream-centric code to define tasks. The battle comes down to configuration vs code when comparing Grunt to Gulp, respectively.
Personally, I’d prefer to have no configuration and no code. Ideally, tooling should help me avoid having to write that boilerplate. In this post, I’ll talk about the feasibility of automating front-end build processes and auto-generating Grunt tasks. The ideas are used in an experimental tool called YA that explores solving this problem.
Can’t I just copy and paste my Gruntfile/Gulpfile from another project?
You could do that – assuming you adhere to the previous project’s folder structure, set of preprocessors used (SASS, Jade, etc), JS module type (AMD or CommonJS), client-side templating language, and file-naming conventions (for r.js or browserify entry points).
If that were the case, you might use a great tool like Yeoman to generate most of this stuff for you. You could create a generator that auto-creates your set directory structure and Gruntfile. This feels a bit restrictive and a bit like pulling in the kitchen sink when you only need a fork and spoon.
For instance, if you only needed to mock a simple HTML prototype, and wanted to use SASS (because it’s awesome), then you’d need to have Yeoman set up the kitchen sink and install libraries that you don’t want or need. Or… or, you could have a generator per use case: one for simple prototypes that use no JS, one for a prototype that does use JS, one for…
Alternatively, our tools could be smarter and figure out what tools we’re using while we use the tools and take care of Grunt or Gulp behind the scenes. Seriously, I just want to use SASS. I don’t give a shit about writing a custom, 30-line Grunt config. I don’t give a shit about writing more irrelevant JS code to configure Gulp. I really don’t want to care about those tools; I just want to build my app.
How could we auto-detect that we’re using SASS and generate a Grunt task for it?
We’ll use Grunt as the primary build engine in this discussion since it’s the most popular task runner, is pretty functionally equivalent to Gulp, and gets the point across.
Grunt has a watch functionality that monitors a directory for filesystem modifications (add/remove/modify files). If we create a new SASS file (example: styles.scss), Grunt will fire an event indicating that styles.scss has been added.
Once you know the extension of the new file, you know the preprocessor being used. Once you know the preprocessor, you can associate that with an existing grunt plugin that should be installed. Once installed, you can then auto-generate a compile and watch task for files of that extension.
YA provides a custom listener for that watch event. Once YA is started, it runs Grunt’s watcher in a new process. On the addition of a new file, we need to do the following:
- Get the file extension of the newly added file
- Tell YA that a new extension was used. Since Grunt is being run in a different process than YA, there’s no straightforward way to communicate between the two. Hence, we instruct Grunt to log the addition of the new file to stdout. We then tell YA to listen to the ‘data’ event of stdout and search for text that indicates the addition of a new file.
- Once YA has the newly added extension, it dynamically installs the appropriate Grunt plugin (grunt-contrib-sass, in this case), generates a task to compile and watch SASS files, generates a new Gruntfile to disk, compiles the added SASS file, and restarts the watch process.
Now, anytime you make changes to that SASS file, grunt will take over and execute the SASS compile task that YA set up. Anytime you add an additional SASS file, Grunt takes over and recompiles all SASS files (or only the ones that changed since YA utilizes the grunt-newer plugin).
That custom watch listener looks like the following:
grunt.event.on('watch', function(action, filepath) { var ext = path.extname(filepath); // Ignore directories if (fs.lstatSync(filepath).isDirectory()) return; if (action === 'added') { // This is a special message that's parsed by YA // to determine if support for an additional preprocessor is necessary console.log('EXTADDED:' + ext); } });
In fact, all preprocessors have the same flow for auto-generating tasks. We’ve basically automated the use of SASS, Compass, Coffeescript, Jade, Slim, JSX, Typescript, and more. You didn’t have to do anything more than start up YA once (and keep it running in the background), create one of those preprocessor files, and let YA take care of the gruntwork for you 😉 See the list of preprocessors that YA currently supports for a more comprehensive list.
Okay, that was an easy case
Preprocessors aside, I was more interested in seeing how much of the set of front-end build tooling processes could be automated.
First, it’s necessary to quantify the common steps around building front-end apps:
- Compiling and watching preprocessors (done)
- Creating JavaScript application bundles using r.js or browserify
- Installing vendor libaries like Backbone, jQuery, etc
- JSHinting
- Running test suites (like Jasmine or Mocha)
- Compiling (mustache/handlebars) templates
- Image optimization/compression
- LiveReload
- and probably others
Getting the easy stuff out of the way, JSHinting all of your changed JS files is pretty similar to handling preprocessors. If there’s a .jshintrc file in the directory that you tell YA to manage, YA will set up Grunt to jshint your files on save. Compiling handlebars/mustache templates and detecting the use of images that should be optimized could be done similarly.
I’ll focus in on Step 2 and mention the other steps that use similar techniques.
Creating JavaScript application bundles using r.js or browserify
I find it really silly that we need to specify the entry point(s) to our JavaScript apps when writing a Grunt configuration for requirejs or browserify. To automate this, we need to determine the “roots” of the dependency trees of the apps in a given folder using static analysis (i.e., analyzing abstract syntax trees, AST).
A root is a file/module that is not depended upon (i.e., no other module requires it). Think about it. The entry point that you supply to requirejs or browserify is a module that requires in a bunch of other modules and initializes them – starting up your application.
Finding the roots entails:
- Find all JS files about a given directory (the directory that you told YA to manage).
- Ignore all modules that don’t use AMD or CommonJS or UMD. I wrote a library for determining the module type of a given file.
- Add all module names to a look-up-table (LUT) that marks all files that have been depended upon.
- For each file, statically analyze the module to get its dependencies using detective-amd or detective, and mark each dependency as “used” in the LUT.
- After all of the modules have been processed, the files in the LUT that haven’t been marked as used are the roots.
Once YA finds the roots, it will generate the appropriate Grunt task (requirejs or browserify) depending on the module’s definition (AMD or CommonJS, respectively), flush the new gruntfile to disk, generate the bundles, and rewatch the directory for changes to JS files.
If a new JS file gets added, the computation of the roots needs to be repeated since that new module could potentially require the old root. There’s a similar argument for removing JS files.
The other remaining steps
Installing vendor libraries like backbone, jquery, and others could be achieved using static analysis (on the roadmap for YA). Once a file is saved, traverse the AST looking for expressions (the pattern matching rules could be abstracted into a module) that use identifiers like Backbone, $, jQuery, _, etc and npm or bower install those dependencies. These vendor dependencies would then get picked up in the bundling phase and usable without you having to do more than write code that states that you want to use those libraries.
Running test suites could involve traversing the AST looking for testing-specific constructs like “describe” or “it” for setting up Mocha and Grunt (to run the suite on save of a JS file).
Let’s build smarter tools
We don’t have to download the kitchen sink and we don’t have to adhere to the same rules/tools for every project. We should have a tools that detect what we’re doing and set up what we need. Thanks to static analysis, it’s totally feasible to automate a bunch of the processes we currently write code to configure.
There will definitely be custom needs as your app grows. It will be interesting to see the limits of static analysis task generation tools like YA. Even if a single, almighty tool isn’t the answer, perhaps some of the ideas could get integrated into our day to day tools like Browserify, Grunt, and Gulp.
If you’re interested in exploring this space, here are some libraries that help:
- https://github.com/ariya/esprima
- https://github.com/defunctzombie/node-required
- https://github.com/substack/node-detective
- https://github.com/mrjoelkemp/node-detective-amd
- https://github.com/Constellation/escodegen
- https://github.com/mrjoelkemp/node-app-root
- https://github.com/mrjoelkemp/module-definition
- https://github.com/fshost/node-dir
- https://github.com/substack/node-falafel
- https://github.com/Constellation/estraverse
- https://github.com/mrjoelkemp/node-source-walk
Thoughts? Feel free to chat with me on Twitter: https://twitter.com/mrjoelkemp