Buy

Versioning to Bust Cache

We've got a real nice setup here! With gulp running, if we update any Sass files, this main.css gets regenerated. But there's just one problem: when we deploy an updated main.css, how can we bust browser cache so our visitors see the new stuff?

To solve this easily, we could go into the template, add a ?v= to the end, then manually update this on each deploy. Of course, I'll definitely forget to do this, so let's find a better way with Gulp.

Introducing gulp-rev

Search for the plugin gulp-rev, as in "revision". Open up its docs. This plugin does one thing: you point it at a file - like unicorn.css - and it changes the name, adding a hash on the end. That hash is based on the contents, so it'll change whenever the file changes.

So let's think about this: if we can somehow make our template automatically point to whatever the latest hashed filename is, we've got instant cache-busting. Every time we deploy with an update, your CSS file will have a new name.

I want to do that, so copy the install line and get that downloading:

npm install --save-dev gulp-rev

Now, head to gulpfile.js. Remember, we're using gulp-load-plugins, so we don't need the require line. In addStyle, add the new pipe() right before the sourcemaps are dumped so that both the CSS file and its map are renamed. Inside, use plugins.rev():

73 lines gulpfile.js
... lines 1 - 13
gulp.src(paths)
... lines 15 - 19
.pipe(plugins.rev())
... lines 21 - 22
};
... lines 24 - 73

Ok, let's see what this does. I'll clean out web/css:

rm -rf web/css/*

Now run gulp:

gulp

Go check out that directory. Instead of main.css and dinosaur.css, we have main-50d83f6c.css and dinosaur-32046959.css And the maps also got renamed - so our browser will still find them.

But you probably also see the problem: the site is broken! We're still including the old main.css file in our layout.

Dumping the rev-manifest.json File

We can't just update base.html.twig to use the new hashed name because it would re-break every time we changed the file. What we need is a map that says: "Hey, main.css is actually called main-50d83f6c.css." If we had that, we could use it inside our PHP code to rewrite the main.css in the base template to hashed version automatically. When the hashed name updates, the map would update, and so would our code.

And of course, the gulp-rev people thought of this! They call that map a "manifest". To get gulp-rev to create that for us, we need to ask it really nicely. At the end, add another pipe() to plugins.rev.manifest() and tell that where we want the manifest. Let's put it next to our assets at app/Resources/assets/rev-manifest.json:

76 lines gulpfile.js
... lines 1 - 12
app.addStyle = function(paths, outputFilename) {
gulp.src(paths)
... lines 15 - 22
// write the rev-manifest.json file for gulp-rev
.pipe(plugins.rev.manifest('app/Resources/assets/rev-manifest.json'))
... line 25
};
... lines 27 - 76

As you'll see, this file doesn't need to be publicly accessible - our PHP code just needs to be able to read it.

There's one more interesting step: pipe() this into gulp.dest('.'):

76 lines gulpfile.js
... lines 1 - 13
gulp.src(paths)
... lines 15 - 22
// write the rev-manifest.json file for gulp-rev
.pipe(plugins.rev.manifest('app/Resources/assets/rev-manifest.json'))
.pipe(gulp.dest('.'));
... lines 26 - 76

What?

What do Multiple dest()'s mean?

So far, we've always had one gulp.src() at the top and one gulp.dest() at the bottom, but you can have more. Our first gulp.dest() writes the CSS file. But once we pipe to plugins.rev.manifest(), the Gulp stream changes. Instead of being the CSS file, the manifest is now being passed through the pipes. So the last gulp.dest() just writes that file relative to the root directory.

Let me show you. Stop gulp and restart:

gulp

And there's our rev-manifest.json file:

{
    "main.css": "main-50d83f6c.css"
}

It holds the map from main.css to its actual filename right now. It is missing dinosaur.css, but we'll fix that in a second.

Fixing the manifest base directory

But there's another problem I want to tackle first. In a second, we're going to put JavaScript paths into the manifest too. So I really need this to have the full public path - css/main.css - instead of just the filename.

So why does it just say main.css? Because when we call addStyle(), we pass in only main.css. This is passed to concat() and that becomes the path that's used by gulp-rev.

The fix is easy! Inside concat(), update it to css/ then the filename. That changes the filename that's inside the Gulp stream. To keep the file in the same spot, just take the css/ out of the gulp.dest() call:

76 lines gulpfile.js
... lines 1 - 12
app.addStyle = function(paths, outputFilename) {
gulp.src(paths)
... lines 15 - 17
.pipe(plugins.concat('css/'+outputFilename))
... lines 19 - 21
.pipe(gulp.dest('web'))
... lines 23 - 25
};
... lines 27 - 76

So nice: those two pipes work together to put the file in the same spot. Run gulp again:

gulp

Now, rev-manifest.json has the css/ prefix we need:

{
    "css/main.css": "css/main-50d83f6c.css"
}

Merging Manifests

So why the heck doesn't my dinosaur.css show up here? The addStyle() function is called twice: once for main.css and once for dinosaur.css. But the second time, since the manifest file is already there, it does nothing. Unless, you pass an option called merge and set it to true:

78 lines gulpfile.js
... lines 1 - 13
gulp.src(paths)
... lines 15 - 22
// write the rev-manifest.json file for gulp-rev
.pipe(plugins.rev.manifest('app/Resources/assets/rev-manifest.json', {
merge: true
}))
... lines 27 - 78

Let's see if this fixed it! Re-run gulp:

gulp

Yes! The hard part is done - this is a perfect manifest file:

{
    "css/main.css": "css/main-50d83f6c.css",
    "css/dinosaur.css": "css/dinosaur-32046959.css"
}

Phew! We're in the homestretch - the Gulp stuff is done. The only thing left is to make our PHP use the manifest file.

Since I'm in Twig, I'm going to invent a new filter called asset_version:

48 lines app/Resources/views/base.html.twig
... lines 1 - 9
<link rel="stylesheet" href="{{ asset('css/main.css'|asset_version) }}"/>
... lines 11 - 48

Let's make it do something! I already created an empty Twig extension file to get us started:

26 lines src/AppBundle/Twig/AssetVersionExtension.php
<?php
namespace AppBundle\Twig;
class AssetVersionExtension extends \Twig_Extension
{
private $appDir;
public function __construct($appDir)
{
$this->appDir = $appDir;
}
public function getFilters()
{
return array(
);
}
public function getName()
{
return 'asset_version';
}
}

And I told Twig about this in my app/config/services.yml file:

12 lines app/config/services.yml
... lines 1 - 5
services:
twig_asset_version_extension:
class: AppBundle\Twig\AssetVersionExtension
arguments: ["%kernel.root_dir%"]
tags:
- { name: twig.extension }

So, this Twig extension is ready to go! All we need to do is register this asset_version filter, which I'll do inside getFilters() with new \Twig_SimpleFilter('asset_version', ...) and we'll have it call a method in this class called getAssetVersion:

41 lines src/AppBundle/Twig/AssetVersionExtension.php
... lines 1 - 13
public function getFilters()
{
return array(
new \Twig_SimpleFilter('asset_version', array($this, 'getAssetVersion')),
);
}
... lines 20 - 41

Below, I'll add that function. It'll be passed the $filename that we're trying to version. So for us, css/main.css.

Ok, our job is simple: open up rev-manifest.json, find the path, then return its versioned filename value. The path to that file is $this->appDir - I've already setup that property to point to the app/ directory - then /Resources/assets/rev-manifest.json:

41 lines src/AppBundle/Twig/AssetVersionExtension.php
... lines 1 - 20
public function getAssetVersion($filename)
{
$manifestPath = $this->appDir.'/Resources/assets/rev-manifest.json';
... lines 24 - 33
}
... lines 35 - 41

With the power of TV, I'll magically add the next few lines. First, throw a clear exception if the file is missing. Next, open it up, decode the JSON, and set the map to an $assets variable. Since the manifest file has the original filename as the key, let's throw one more exception if the file isn't in the map. I want to know when I mess up. And finally, return that mapped value!

41 lines src/AppBundle/Twig/AssetVersionExtension.php
... lines 1 - 20
public function getAssetVersion($filename)
{
$manifestPath = $this->appDir.'/Resources/assets/rev-manifest.json';
if (!file_exists($manifestPath)) {
throw new \Exception(sprintf('Cannot find manifest file: "%s"', $manifestPath));
}
$paths = json_decode(file_get_contents($manifestPath), true);
if (!isset($paths[$filename])) {
throw new \Exception(sprintf('There is no file "%s" in the version manifest!', $filename));
}
return $paths[$filename];
}
... lines 35 - 41

So we give it css/main.css and it gives us the hashed filename.

Let's give it a shot! Take a deep breath and refresh. Victory! Our beautiful site is back - the hashed filename shows up in the source.

Ok ok, let's play with it. Open layout.scss and give everything a red background. The Gulp watch robots are working, so I immediately see a brand new hashed main.css file in web/css. But will our layout automatically update to the new filename? Refresh to find out. Yes! The new CSS filename pops up and the site has this subtle red background.

Go back and undo that change. Things go right back to green. Oh, and we do have one other CSS file on the dino show page. It should be giving us a little more space below the T-Rex, but it's 404'ing. We need to make it use the versioned filename.

So, open up show.html.twig and give it the asset_version filter:

23 lines app/Resources/views/dinosaurs/show.html.twig
... lines 1 - 5
<link rel="stylesheet" href="{{ asset('css/dinosaur.css'|asset_version) }}"/>
... lines 7 - 23

Refresh - perfect! No 404 error, and our button can get a little breathing room from the T-Rex. It took a little setup, but congrats - you've got automatic cache-busting.

Tip

You can make the getAssetVersion() function more efficient by following the advice in this comment.

Tip

In Symfony, it's also possible to avoid needing to use the filter by leveraging a cool thing called "version strategies". Check out the details posted by a helpful user here.

Don't commit the manifest

But should we commit the rev-manifest.json file to source control? I'd say no: it's generated automatically by Gulp. So, finish things off by adding it to your .gitignore file:

18 lines .gitignore
... lines 1 - 16
/app/Resources/assets/rev-manifest.json

Leave a comment!

  • 2016-09-13 irozgar

    I made a PR

  • 2016-09-12 weaverryan

    Ahhh, this is awesome! What a cool way to handle this - you're absolutely right that the VersionStrategyInterface is *meant* for this kind of stuff :). Thanks for sharing! If you want, you could also add a tip in the script about this here https://github.com/knpuniversi... (there's already an example of a tip). I think this would be a cool thing to point out.

    Cheers!

  • 2016-09-11 irozgar

    Hi Ryan,

    Thanks for your great tutorials. When I ended this video I asked myself if there was a better way than adding the asset_version filter in every versioned asset (I know that I'd forget writting it more than once) so I started looking of other ways of doing it and I learned
    about the VersionStrategyInterface from the Asset Component. After a few hours I've made a simple VersionStrategy to use with assets
    versioned with gulp-rev. With it the filter is no needed anymore.

    Here is a gist with the code
    https://gist.github.com/irozga...

    And a Symfony Bundle for Symfony >= 2.7 based on the gist
    https://github.com/irozgar/gul...

  • 2016-09-09 ElHornair

    Alright, PR is ready: https://github.com/knpuniversi...

  • 2016-09-09 weaverryan

    My pleasure - it ends up looking nice in Disqus, but it's got a weird, hidden syntax!

  • 2016-09-09 ElHornair

    Alright, will try to do that.
    Thanks for wrapping my code btw, didn't know you can do that in Disqus ;)

  • 2016-09-09 weaverryan

    Yo ElHornair!

    Yea, you're totally right - my algorithm could easily be made more efficient :). This *is* on GitHub (https://github.com/knpuniversi..., but to actually update it is a little complex - we use a proprietary "diff" system to manage changes that happen *during* the tutorial. BUT, actually, I think we shouldn't change the file anyways - I like to have the code blocks and code download match the video. But, what we *could* do is add a note to the script in *just* the right spot that points over here to your comment. Best of both worlds :). If you agree and want to get your proper props on the GitHub repo, you could edit the script here: https://github.com/knpuniversi.... We have a special ***TIP syntax you can see - an example is in this script: https://raw.githubusercontent....

    Thanks for the comment!

  • 2016-09-09 ElHornair

    Hi Ryan

    If you have a couple of files in the rev-manifest, you will be doing unnecessary file system reads. Because the Twig extension is a singleton, you can load the paths once and save them in a class variable. Here is the adapted code:


    private $paths = [];

    public function getAssetVersion($filename)
    {
    if (count($this->paths) === 0) {
    $manifestPath = $this->appDir.'/Resources/assets/rev-manifest.json';

    if (!file_exists($manifestPath)) {
    throw new \Exception(sprintf('Cannot find manifest file: "%s"', $manifestPath));
    }

    $this->paths = json_decode(file_get_contents($manifestPath), true);
    }

    if (!isset($this->paths[$filename])) {
    throw new \Exception(sprintf('There is no file "%s" in the version manifest!', $filename));
    }

    return $this->paths[$filename];
    }

    PS: I'd be happy to add this as a PR to the above tutorial (which is great btw, thanks a lot!) but I couldn't find this on Github. Any hints?

  • 2015-07-20 weaverryan

    Hey Marcus!

    Hmm - could you throw up a simple example repository on GitHub? Perhaps something changed with some version of these libraries that's causing the problems. This was indeed the trickiest part of this whole tutorial :).

    Cheers!

  • 2015-07-20 Marcus Stöhr

    Hi Ryan.

    I just checked, double-checked, and in fact triple-checked the chapter you mentioned and my gulpfile.js contains all of this. However, it doesn't work and this is driving me nuts.

    Any hint what else I can do?

  • 2015-07-20 weaverryan

    Hey Marcus!

    Yes, I know the problem well! Have you gone through the last 2 chapters in this tutorial (especially https://knpuniversity.com/scre.... We talk about this exact issue. When recording this chapter - the manifest issue hadn't reared it's head - so we had to fix it in a later chapter.

    Let me know if that helps - and happy you're enjoying things (you'll enjoy them more when cache busting works)!

  • 2015-07-19 Marcus Stöhr

    Hey Ryan,

    thank you for this great tutorial. The version busting strategy is really cool but I have a really hard time to get it working for all my styles and scripts.
    I have set it up like you showed in the upcoming chapters using the pipeline. However, the manifest-file doesn't contain all my files but either only one or two of them. It also mixes styles and scripts.

    Whats going on here?

  • 2015-04-08 s.molinari

    I am playing with a JS framework called Aurelia (the one I mentioned in my other post)

    http://aurelia.io/

    and I noticed it wouldn't "correct" when changing files, because of the cache in my browser. Obviously I could have the browser load every file again on every call to develop, but cache busting as described here is the better answer. We'll need it anyway to fix bugs in production later, right? So I actually built in the steps you explained in this tutorial into the Aurelia demo app under "Get Started". However, because Aurelia (similar to Angular?) uses data binding with custom tag attributes as built in directives like <body aurelia-app="">, which also formulates the name of the file, cache busting busts the application too. :-(

    I found another cache busting module, which partially does something like what is needed I believe.

    https://github.com/hollandben/...

    However, in Aurelia, it doesn't use complete file names in the router.

    { route: ['','welcome'], moduleId: './welcome', nav: true, title:'Welcome' },
    { route: 'flickr', moduleId: './flickr', nav: true },
    { route: 'child-router', moduleId: './child-router', nav: true, title:'Child Router' }

    I highly doubt this would work.

    { route: ['','welcome'], moduleId: './welcome#grunt-cache-bust', nav: true, title:'Welcome' },
    { route: 'flickr', moduleId: './flickr#grunt-cache-bust', nav: true },
    { route: 'child-router', moduleId: './child-router#grunt-cache-bust', nav: true, title:'Child Router' }

    However, as I said "partially does what is needed", the initial call for app.js through the custom tag attribute is still not resolved.

    Scott

  • 2015-04-08 weaverryan

    Hey again Scott!

    Hmm, what's your use-case? There is a plugin I *almost* talked about, which is useful if you're moving files from one directory to another: https://www.npmjs.com/package/.... And one exists for this and gulp-rev (https://www.npmjs.com/package/... but I haven't tried it yet! Let me know if it fits for you (and what you're use-case in - I'm curious).

    Cheers!

  • 2015-04-07 s.molinari

    Hey Ryan,

    What if the names of files are important to the actual execution of the JS app? Cache busting would break the app. Is there a way to "insert" the new versions smartly within the code too?

    Scott