Buy

on('end'): Async and Listeners

Run Gulp! Spoiler alert: Gulp is lying to you. It looks like everything runs in order: clean starts, clean finishes, then styles starts. But that's wrong. The truth is that everything is happening all at once, asynchronously. And to be fair, Gulp isn't really lying - it actually has no idea when each task actually finishes. Well, at least not yet.

Let's find out what's really going on.

Gulp Streams are like a Promise

Each line in a Gulp stream is asynchronous - like an AJAX call. This means that before gulp.src() finishes, the next pipe() is already being called. In fact, the whole function might finish before gulp.src() is done.

But we really need each line to run in order. So when you call pipe(), it doesn't run what's inside immediately: it schedules it to be called once the previous line finishes. The effect is like making an AJAX call, adding a success listener, then making another AJAX call from inside it.

I wonder then, does the main.css file finish compiling before dinosaur.css starts? Does the scripts wait for the styles task to finish? Let's find out.

Adding on('end') Listeners

Like with AJAX, each line returns something that acts like a Promise. That means, for any line, we can write on to add a listener for when this specific line actually finishes. When that happens, let's console.log('start '+filename).

Copy this and add another listener to the last line. Change the text to "end":

93 lines gulpfile.js
... lines 1 - 14
app.addStyle = function(paths, outputFilename) {
gulp.src(paths).on('end', function() { console.log('start '+outputFilename)})
... lines 17 - 28
.pipe(gulp.dest('.')).on('end', function() { console.log('end '+outputFilename)})
};
... lines 31 - 93

Ok, run gulp!

gulp

Woh! When it said it finished "styles", it really means it was done executing the styles task. But things really finish way later. In fact, they don't even start the process until later. And what's really crazy is that dinosaur.css starts before main.css, even though main is the first style we add.

So, you can't depend on anything happening in order. But, what if you need to?

Race Condition in the Manifest

There's a bug with our manifest file - a race condition! Ah gross! Because of the merge option, it opens up the manifest, reads the existing keys, updates one of them, then re-dumps the whole file.

For styles, the manifest file is opened twice for main.css and dinosaur.css. If one opens the file before the other finishes writing, when it writes, it'll run over the changes from the first.

How can we make the first addStyle finish before starting the second?

Using on to Control Order

It turns out the answer is easy. We can attach an end listener to any part of the Gulp stream. Return the stream from addStyle. Then in styles, attach an on('end') and only process dinosaur.css once the previous call is finished:

93 lines gulpfile.js
... lines 1 - 14
app.addStyle = function(paths, outputFilename) {
return gulp.src(paths).on('end', function() { console.log('start '+outputFilename)})
... lines 17 - 29
};
... lines 31 - 52
gulp.task('styles', function() {
app.addStyle([
config.bowerDir+'/bootstrap/dist/css/bootstrap.css',
config.bowerDir+'/font-awesome/css/font-awesome.css',
config.assetsDir+'/sass/layout.scss',
config.assetsDir+'/sass/styles.scss'
], 'main.css').on('end', function() {
app.addStyle([
config.assetsDir+'/sass/dinosaur.scss'
], 'dinosaur.css');
});
});
... lines 65 - 93

I know, it's ugly - we'll fix it, I promise! But let's see if it works:

gulp

Perfect! main.css starts and ends. Then dinosaur.css starts.

Using the Pipeline

This is the key idea. But the syntax here is terrible. If we have 10 CSS files, we'll need 10 levels of nested listeners. That's not good enough.

To help fix this, I'll paste in some code I wrote:

129 lines gulpfile.js
... lines 1 - 53
var Pipeline = function() {
this.entries = [];
};
Pipeline.prototype.add = function() {
this.entries.push(arguments);
};
Pipeline.prototype.run = function(callable) {
var deferred = Q.defer();
var i = 0;
var entries = this.entries;
var runNextEntry = function() {
// see if we're all done looping
if (typeof entries[i] === 'undefined') {
deferred.resolve();
return;
}
// pass app as this, though we should avoid using "this"
// in those functions anyways
callable.apply(app, entries[i]).on('end', function() {
i++;
runNextEntry();
});
};
runNextEntry();
return deferred.promise;
};
... lines 84 - 129

It's an object called Pipeline - and it'll help us execute Gulp streams one at a time. It has a dependency on an object called q, so let's go install that:

npm install q --save-dev

On top, add var Q = require('q')

129 lines gulpfile.js
... lines 1 - 3
var Q = require('q');
... lines 5 - 129

To use it, create a pipeline variable and set it to new Pipeline(). Now, instead of calling app.addStyle() directly, call pipeline.add() with the same arguments. Now we can move dinosaur.css out of the nested callback and use pipeline.add() again. Woops, typo on pipeline!

129 lines gulpfile.js
... lines 1 - 84
gulp.task('styles', function() {
var pipeline = new Pipeline();
pipeline.add([
config.bowerDir+'/bootstrap/dist/css/bootstrap.css',
config.bowerDir+'/font-awesome/css/font-awesome.css',
config.assetsDir+'/sass/layout.scss',
config.assetsDir+'/sass/styles.scss'
], 'main.css');
pipeline.add([
config.assetsDir+'/sass/dinosaur.scss'
], 'dinosaur.css');
... lines 98 - 99
});
... lines 101 - 129

pipeline.add is basically queuing those to be run. So at the end, call pipeline.run() and pass it the actual function it should call:

129 lines gulpfile.js
... lines 1 - 84
gulp.task('styles', function() {
... lines 86 - 94
pipeline.add([
config.assetsDir+'/sass/dinosaur.scss'
], 'dinosaur.css');
pipeline.run(app.addStyle);
});
... lines 101 - 129

Behind the scenes, the Pipeline is doing what we did before: calling addStyle, waiting until it finishes, then calling addStyle again.

Try it!

gulp

Cool - we've got the same ordering.

Pipelining scripts

Ok! Let's add this pipeline stuff to scripts. First, clean up my ugly debug code. Make sure you actually return from addScript - we need that stream so the Pipeline can add an end listener.

133 lines gulpfile.js
... lines 1 - 32
app.addScript = function(paths, outputFilename) {
return gulp.src(paths)
... lines 35 - 133

Down in scripts work your magic! Create the pipeline variable, then pipeline.add(). And, pipeline.run() to finish:

133 lines gulpfile.js
... lines 1 - 101
gulp.task('scripts', function() {
var pipeline = new Pipeline();
pipeline.add([
config.bowerDir+'/jquery/dist/jquery.js',
config.assetsDir+'/js/main.js'
], 'site.js');
pipeline.run(app.addScript);
});
... lines 112 - 133

Ok, try it!

gulp

Good, no errors! Use the Pipeline if you like it. But either way, remember that Gulp runs everything all at once. You can make one entire task wait for another to finish, but we'll talk about that later.

Leave a comment!

  • 2016-08-19 Kevin

    Hey again weaverryan !

    I found a bug with the manifest file. It seems to work with 2 tasks (styles & scripts) in the tutorial.

    Then, I wanted to do 4 tasks (styles, styles_libraries, scripts and scripts libraries), in order to avoid generate the libraries files when I watch the custom styles, because it takes so long to generate the file when using some libraries. (~3s, too long for some CSS modifications)

    After I had the 4 tasks, the manifest has gone crazy, it was rewriting when another task was still running even with the Pipeline.
    I wonder if the problem comes from the fact that we initialize multiple Pipeline object and they can't communicate between them...? I don't have enough knowledge about Promise to be sure. Any ideas ?

    Some threads say the manifest need a base path to have a good merge :


    var config = {
    baseRevManifestPath: 'app/Resources/assets',
    revManifestPath: 'app/Resources/assets/rev-manifest.json'
    };

    .pipe(plugins.rev.manifest(config.revManifestPath, {
    base: config.baseRevManifestPath,
    merge: true
    }))
    .pipe(gulp.dest(config.baseRevManifestPath));

    However, this doesn't fix the issue, I don't know if this is necessary...

    To fix the issue, I had to finally use the deps array of gulp tasks and make styles_libraries dependant from styles, scripts dependant from styles_libraries and scripts_libraries dependant of scripts.

    I don't know if it's a good solution, but it seems to work so far.

    What do you think about this ?

  • 2016-08-19 Kevin

    Hey again weaverryan !

    I found a bug with the manifest file. It seems to work with 2 tasks (styles & scripts).

    Then, I wanted to do 4 tasks (styles, styles_libraries, scripts and scripts libraries), in order to avoid generate the libraries files when I watch the custom styles, because it takes so long to generate the file when using some libraries. (~3s, couldn't wait so long to reload the browser)

    After I had the 4 tasks, the manifest has gone crazy, it was rewriting when another task was still running even with the Pipeline.

    I wonder if the problem comes from the fact that we initialize multiple Pipeline object and they can't communicate between them...? I don't have enough knowledge about Promise to be sure. Any ideas ?

    Some threads say the manifest need a base path to have a good merge :


    var config = {
    baseRevManifestPath: 'app/Resources/assets',
    revManifestPath: 'app/Resources/assets/rev-manifest.json'
    };

    .pipe(plugins.rev.manifest(config.revManifestPath, {
    base: config.baseRevManifestPath,
    merge: true
    }))
    .pipe(gulp.dest(config.baseRevManifestPath));

    However, this doesn't fix the issue, I don't know if this is necessary...

    To fix the issue, I had to finally use the deps array of gulp tasks and make styles_libraries dependant from styles, scripts dependant from styles_libraries and scripts_libraries dependant of scripts.

    I don't know if it's a good solution, but it seems to work so far.

    What do you think about this ?

  • 2016-08-19 Kevin

    It works perfectly, thank you :)

  • 2016-08-18 weaverryan

    Hi Kevin!

    Ah, I think you're right! There is a bug here :). Fortunately, it looks like a well-known bug with a well-known workaround. Try updating both of your plugin.plumber calls (for CSS and JS) to look like this


    .pipe(plugins.plumber(function(error) {
    console.log(error.toString());
    this.emit('end');
    }))

    Specifically, instead of just calling plumber(), you need to provide a callback. This callback should print the error (which plumber did automatically before adding this function) and then call this.emit('end'). That is the key: this tells the watch task that plumber is finished... which for some reason is the key to the whole thing.

    Let me know if it works! I'm going to make an update to the tutorial.

    Thanks for the question!

  • 2016-08-18 Kevin

    Hi weaverryan,

    Thanks for the tutorial, it's really good !

    I've got an issue here. Your Pipeline method is working great for me but :

    The problem is that when I make a syntax error in the SCSS file, the plumber catch the error and gulp is still running, but nothing happen when I fix the syntax, no more CSS files are generated, it's like the watch styles is down.

    Did I make something wrong ? Did you test the plumber with the Pipeline ?

  • 2016-08-18 Kevin

    Hi weaverryan,

    Thanks for the tutorial, it's really good !
    I've got an issue here. Your Pipeline method is working great for me but :

    The problem is that when I make a syntax error in the SCSS file, the plumber catch the error and gulp is still running, but nothing happen when I fix the syntax, no more CSS files are generated, it's like the watch styles is down.

    Did I make something wrong ? Did you test the plumber with the Pipeline ?

  • 2015-11-17 alsbury

    I'm with you, it seems like an obvious problem, so it would seem that something would exist already.

    What about a collection in memory of resources? When a resource get's updated in memory, the whole collection gets written? Maybe that idea wouldn't play well with gulp-rev.

  • 2015-11-17 weaverryan

    Hey alsbury!

    Ah, really glad this was useful! So, I don't know of any such library, but I got the *exact* same impression as you: there was a logical problem... so I was expecting a library to already exist. To this day, the fact that there wasn't/isn't a library, leaves me thinking that there *might* be some other direction to take in solving this. But so far, I haven't seen anything. I don't have plans to release it as a library since (A) it's pretty small and (B) I still think that *eventually* a different method or library will be uncovered. But until then, it's working great for me and others.

    Cheers!

  • 2015-11-17 alsbury

    Hey Ryan,
    Really helpful tutorial. We've already started to take what we've learned here and started to apply it to a project. After watching the last video, it would seem like the Pipeline class would already exist somewhere as a library in the NPM database. Do you know of a library? Do you have plans to release it as a library? Or should someone like me create it?