Buy

Faster Deploy with Shared Files

So far, each release is independent. And sometimes, that sucks! Each release has its own log files. There's nothing in logs/ right now, but usually you'll find a prod.log file. The problem is that if you need to go look inside to debug an issue, you might have to look through 10 separate prod.log files across 10 separate deploys!

Sharing a Path between Deploys

This is a perfect example of a file that we want to share between deployments. Fortunately - as I mentioned earlier - Ansistrano has us covered. Check out the Ansistrano documentation. Ah yes, we need this ansistrano_shared_paths variable! Copy it! Then, in deploy.yml, add it near the top:

53 lines ansible/deploy.yml
---
- hosts: aws
... lines 3 - 14
vars:
... lines 16 - 18
# Ansistrano vars
... lines 20 - 21
# Arrays of directories and files to be shared.
# The following arrays of directories and files will be symlinked to the current release directory after the 'update-code' step and its callbacks
# Notes:
# * Paths are relative to the /shared directory (no starting /)
# * If your items are in a subdirectory, write the entire path to each shared directory
#
# Example:
# ansistrano_shared_paths:
# - path/to/first-dir
# - path/next-dir
# ansistrano_shared_files:
# - my-file.txt
# - path/to/file.txt
ansistrano_shared_paths:
... lines 36 - 53

It's simple, we want to share var/logs: that entire directory:

53 lines ansible/deploy.yml
---
- hosts: aws
... lines 3 - 14
vars:
... lines 16 - 18
# Ansistrano vars
... lines 20 - 21
# Arrays of directories and files to be shared.
# The following arrays of directories and files will be symlinked to the current release directory after the 'update-code' step and its callbacks
# Notes:
# * Paths are relative to the /shared directory (no starting /)
# * If your items are in a subdirectory, write the entire path to each shared directory
#
# Example:
# ansistrano_shared_paths:
# - path/to/first-dir
# - path/next-dir
# ansistrano_shared_files:
# - my-file.txt
# - path/to/file.txt
ansistrano_shared_paths:
- var/logs
... lines 37 - 53

Oh, and now that var/logs will be a symlink, in after-symlink-shared.yml, under the permissions task, we need to add follow: true so that the permissions change follows the symlink:

49 lines ansible/deploy/after-symlink-shared.yml
... lines 1 - 40
- name: Setup directory permissions for var/
file:
... lines 43 - 46
# We need it because logs are symlinks now
follow: yes

And back in deploy.yml... yea...my variable didn't paste well. Make sure your indentation is correct!

53 lines ansible/deploy.yml
---
- hosts: aws
... lines 3 - 14
vars:
... lines 16 - 18
# Ansistrano vars
... lines 20 - 21
# Arrays of directories and files to be shared.
# The following arrays of directories and files will be symlinked to the current release directory after the 'update-code' step and its callbacks
# Notes:
# * Paths are relative to the /shared directory (no starting /)
# * If your items are in a subdirectory, write the entire path to each shared directory
#
# Example:
# ansistrano_shared_paths:
# - path/to/first-dir
# - path/next-dir
# ansistrano_shared_files:
# - my-file.txt
# - path/to/file.txt
ansistrano_shared_paths:
- var/logs
... lines 37 - 53

Now! Find your local terminal and deploy!

ansible-playbook ansible/deploy.yml -i ansible/hosts.ini --ask-vault-pass

Shared Paths & Avoiding Server Storage

Use beefpass and deploy to master. Make sure you think about what other directories or files you might need to share between deploys, like web/uploads if you store uploaded files on your server. Or, web/imagine if you use LiipImagineBundle. Otherwise, all the thumbnails will need to be re-created after each deploy. That's lame!

But also keep in mind that there are some big advantages to not storing files like these on your server. Instead of putting uploaded files in web/uploads, you could store them in a cloud store, like S3. If you put nothing extra on your server, it's really easy to destroy your server and launch a new one... without needing to worry about copying over a bunch of random, uploaded files. It also makes using multiple servers possible.

Ding! Move over to the terminal that's SSH'ed onto the server. Go out of current/ and then back in:

cd ..
cd current/

Check out var/logs:

ls -la var/

it's now a symlink to shared/var/logs. That directory is empty, but as soon as we log something, a prod.log file will show up there.

Sharing Files for Performance

There's one other reason you might want to share some paths: speed! Right now, our deploy is slow! You may not have noticed because of the magic of television: we've been fast-forwarding through the deploys! But, in real life, the Install Node dependencies task takes almost 2 minutes to run! Woh!

Why is it so slow? Because it needs to download all of the dependencies on every single deploy. But if we shared the node_modules/ directory, then yarn install would start with a full directory. It would only need to download any changes since the last deploy! This is an easy win!

Add node_modules to the shared paths:

54 lines ansible/deploy.yml
---
- hosts: aws
... lines 3 - 14
vars:
... lines 16 - 34
ansistrano_shared_paths:
- var/logs
- node_modules
... lines 38 - 54

Cleaning up old Releases

Oh, and before we deploy, it's time to fix one other thing. Back on the server, go into the releases/ directory:

cd releases/
ls -la

Ok! It's getting crowded in here! Each deploy creates a new directory... and this will gone on forever and ever until we run out of disk space. Fun! Go back to the Ansistrano docs and find the ansistrano_keep_releases variable. This is the key. In deploy.yml, paste that and set it to 3:

55 lines ansible/deploy.yml
---
- hosts: aws
... lines 3 - 14
vars:
... lines 16 - 18
# Ansistrano vars
... line 20
ansistrano_keep_releases: 3 # Releases to keep after a new deployment. See "Pruning old releases".
... lines 22 - 55

Ok, let's try it! Find your local terminal and deploy:

ansible-playbook ansible/deploy.yml -i ansible/hosts.ini --ask-vault-pass

Use beefpass and deploy to master. I'll fast-forward... but I'll tell you how long the deploy really took. This first deploy will still be slow: the node_modules/ directory will start empty. By the way, the composer install command is also a little bit slow... but not nearly as slow as yarn install. Why? Because Composer caches the packages behind the scenes. So even though the vendor/ directory starts empty, composer install runs pretty quickly. We could make it faster by sharing vendor/... but that's a bad idea! If we did that, when a future deploy updated the vendors, this would affect the live site during the deploy! Scary!

Ok, done! I'm deploying to a tiny, slow server, so that took 3 and a half minutes. Almost half of that was for yarn install!

Let's deploy again:

ansible-playbook ansible/deploy.yml -i ansible/hosts.ini --ask-vault-pass

While we're waiting, go back into the server:

cd releases/
ls -la

Yes! There are only 3 releases. In shared/, node_modules/ is populated, thanks to the last deploy:

cd ..
cd shared/
ls -la node_modules/

When the deploy finishes... awesome! The yarn install task was almost instant, and the deploy was nearly two minutes faster! Zoom!

Next, it's time to demystify and fix our cache directory permissions.

Leave a comment!

  • 2017-11-04 Cesar

    Thank you very much Victor. It works!

  • 2017-11-02 Victor Bocharsky

    Hey Cesar,

    Yeah, I faced this error once. It's because Ansistrano removes old releases without sudo, so if you have wrong permissions - you'll have this issue. So, first of all, you need to figure out what exactly permissions you have for those files. Run:

    ls -la /var/www/project/releases/20171101171422Z/var/sessions/prod/sess_339q0lh6gbmn8oj2dvnc9g3q4e

    Notice its permissions and its owner/group. Most probably you need to fix them, or maybe you just need to add the current "ubuntu" user to "www-data" group. Anyway, it depends on what permissions those files have.

    P.S. As a workaround, if it's not a big deal for you, you can just comment out this ansistrano_keep_releases var and return to it later, when we fix the permissions properly. Actually, we're going to fix permissions properly in the next chapter :)

    Cheers!

  • 2017-11-02 Cesar

    Hi. I am on the screencast 14 and when I am trying to use the ansistrano_keep_releases variable, I get the error: cannot remove ā€˜/var/www/project/releases/20171101171422Z/var/sessions/prod/sess_339q0lh6gbmn8oj2dvnc9g3q4eā€™: Permission denied.

    I have tried to use a file command in the after_symlink_shared.yml (like the one you use for var) to set permissions in releases but is not working. Have you ever have this issue? I will appreciate any idea you can give me.