Automated Deployment Messages

Friday, October 5, 2012

Make deployment visible with Capistrano, Autotagger, Git and Sendgrid

There comes a time in every project when the deployment process comes of age, and that development arrives with its own set of Capistrano recipes and Rake tasks. The project I’m on hit that point recently, and one of the neat outcomes of its nascent puberty was a simple Capistrano recipe to send a git changelog to our project mailing list.

Here’s what this looks like:

$ cap staging deploy
... stuff happens here ...
  * executing `sendgrid:notify'
Changelog:
04fc6dd adding capistrano deployment messages

To use this in your Rails project, the first thing you need is a sendgrid account. If you’re budget-minded, you can always use the credentials your Heroku app is using.

$ heroku create
Creating heroku-wackiness-90210... done, stack is cedar
http://heroku-wackiness-90210.herokuapp.com/ | git@heroku.com:heroku-wackiness-90210.git
Git remote heroku added
$ heroku addons:add sendgrid:starter
Adding sendgrid:starter on heroku-wackiness-90210... done, v2 (free)
Use `heroku addons:docs sendgrid:starter` to view documentation.
$ heroku config -s
SENDGRID_PASSWORD=s3kr17
SENDGRID_USERNAME=yodawg@heroku.com

This process uses Capistrano and Autotagger. For information on setting up Capistrano, their wiki is an excellent starting point. For Autotagger setup with Capistrano, Jeff Dean’s auto_tagger repository is the canonical source of information.

After you’re up and running with Capistrano and Autotagger, you need to add the following file under lib/recipes/sendgrid_notifier.rb:

require 'mail'

set :sendgrid_user, "whatever"
set :sendgrid_password, "secret"
set :sendgrid_domain, "pivotallabs.com"

set :sender, "Now Hiring <jobs@pivotallabs.com>"
set :recipient, "Steve Squivot <you@square.com>"

namespace :sendgrid do
  task :notify do
    sendgrid = {
      :address   => "smtp.sendgrid.net",
      :port      => 587,
      :domain    => sendgrid_domain,
      :user_name => sendgrid_user,
      :password  => sendgrid_password,
      :authentication => 'plain',
      :enable_starttls_auto => true
    }

    auto_tagger = AutoTagger::CapistranoHelper.new(
      :stage => rails_env,
      :stages => auto_tagger_stages).auto_tagger
    previous_sha = auto_tagger.refs_for_stage(stage).last.sha
    current_sha = auto_tagger.repo.latest_commit_sha

    mail = Mail.new(from: recipient, to: sender)
    mail.delivery_method :smtp, sendgrid
    mail.subject = "[#{stage}] New Deployment!"
    mail.body = `git log --oneline #{previous_sha}..#{current_sha}`
    mail.deliver!
  end
end

Then, let’s add a line to include this recipe in our Capfile:

require File.expand_path("../lib/recipes/sendgrid_notifier.rb", __FILE__)

Finally, let’s try it out:

$ cap ci sendgrid:notify
  * executing `sendgrid:notify'
Changelog:
04fc6dd add a recruiter message to send off to Square

Alright! The email’s on its way. If you need to call this in your custom deployment step, it’s as easy as sticking sendgrid.notify into your Capfile.

Happy deploying!

Testing capistrano recipes with cucumber

Pivotal Labs

Sunday, April 5, 2009

In this post, I’ll show you how to set up end-to-end Capistrano testing using Cucumber. I’ve extracted this from the cucumber features I wrote for a gem I’m building named auto_tagger. To fully test capistrano recipes, your tests will have to:

Create a local git repository
Create a local app with a config/deploy.rb file
Push the app to the local repository
Run cap deploy:setup from the app (which will setup a directory inside your local test directory)
Run a cap deploy from the app (which will deploy to your test directory)
Assert against the content of the deployed app in the test directory

Background – Capistrano recipes are almost never tested

Looking around online, I couldn’t find a single list of capistrano packages that has an automated test suite, even ones from some big hosts. It’s no surprise that Capistrano tasks are seldom tested – testing capistrano recipes is hard, and even when you do test them, there are still so many variables in real-life deploys that you can’t account for everything.

It’s like Rummy said:

There are known knowns. There are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don’t know. But there are also unknown unknowns. There are things we do not know we don’t know.
from wikipedia

However, there are some things you can do to stave off the “known unknowns”. For example, you know that someone might forget to set an important variable in their cap task and you know they might be using cap-ext-multistage. For these kinds of examples, Capistrano testing can give you much more assurance that a bug in your recipe is less likely to rm -rf /* on your remote machine.

Getting started: setup your keys

To make life easy, you’ll want to be able to ssh to your own machine. To do this, you’ll need to create a key, then add that key to your authorized keys. If you don’t already have a key setup locally, check out the excellent RailsMachine guide. Once you have a key, you can copy it to authorized keys like so:

cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys

Now you should be able to ssh to your own box without entering a password. To log into your own box, you can use the IP address or the computer’s name. Depending on your /etc/hosts file entries, you may also be able to log in using localhost.

If you are a Mac user you’ll have to enable “Remote Access” from System Preferences to be able to ssh in to your own box. For security, only allow yourself to log in via ssh. The system preferences pane will show you the IP address you can use to ssh into.

Mac Remote Login Preference Pane

NOTE: this only will not work on Windows

Setup your cucumber file system

The file system I’ll use for this demo will look like this:

|-- features
|   |-- capistrano.feature
|   |-- step_definitions
|   |   `-- capistrano_steps.rb
|   |-- support
|   |   `-- env.rb
|   `-- templates
|       `-- deploy.erb
|-- recipes
|   `-- my_recipe.rb
`-- test_files

Add the feature

Let’s say you have a simple cap task that writes a file to shared after you deploy. The feature file might look something like this:

features/capistrano.feature

Feature: Deployment
  In order to know feel better about myself
  As a person who needs lots of reinforcement
  I want leave files named PEOPLE_LIKE_YOU all around my remote machine

  Scenario: User deploys
    Given a an app
    When I deploy
    Then the PEOPLE_LIKE_YOU file should be written to shared

Now you can run cucumber features/ and you’ll see that you have several pending steps.

Get your setup correct

For these features to work, we’ll need a test directory (that’s outside of the features directory), and we’ll need to delete everything from it before running every scenario:

features/support/env.rb

require 'spec'
require 'erb'
require 'etc'

Before do
  @test_files_dir = File.join(Dir.pwd, "test_files")
  @app_dir  = File.join(@test_files_dir, "app")
  @repo_dir = File.join(@test_files_dir, "repo")

  FileUtils.rm_r(@test_files_dir) if File.exists?(@test_files_dir)
  FileUtils.mkdir_p(@test_files_dir)
end

Fill in the steps

features/step_definitions/capistrano_steps.rb

Given /^a an app$/ do

  # Create the git repo
  FileUtils.mkdir_p @repo_dir
  Dir.chdir(@repo_dir) do
    system "git --bare init"
  end

  # Create and capify the dummy app, and push it to the local repo
  FileUtils.mkdir_p @app_dir
  Dir.chdir(@app_dir) do
    [
      %Q{git init},
      %Q{mkdir config},
      %Q{capify .},
      %Q{git add .},
      %Q{git commit -m "first commit"},
      %Q{git remote add origin file://#{@repo_dir}},
      %Q{git push origin master}
    ].each do |command|
      system command
    end
  end

  # Write a custom deploy file to the app, using an ERB template
  deploy_variables = {
    :deploy_to => File.join(@test_files_dir, "deployed"),
    :repository => @repo_dir,
    :git_executable => `which git`.strip,
    :logged_in_user => Etc.getlogin
  }

  template_path     = File.expand_path(File.join(__FILE__, "..", "..", "templates", "deploy.erb"))
  compiled_template = ERB.new(File.read(template_path)).result(binding)

  File.open(File.join(@app_dir, "config", "deploy.rb"), 'w') {|f|
    f.write compiled_template
  }
end

When /^I deploy$/ do
  Dir.chdir(@app_dir) do
    system "cap deploy:setup"
    system "cap deploy"
  end

end

Then /^the PEOPLE_LIKE_YOU file should be written to shared$/ do
  File.exists?(File.join(@test_files_dir, "deployed", "shared", "PEOPLE_LIKE_YOU")).should be_true
end

Now when you run cucumber features/ and you’ll see that you a failure because you don’t have the correct cap file.

Make it pass

To make this pass, add a recipe like this:

recipes/my_recipe.rb

Capistrano::Configuration.instance(:must_exist).load do
  task "my_task" do
    run "echo PEOPLE_LIKE_YOU > #{shared_path}/PEOPLE_LIKE_YOU"
  end
end

Debugging

You’ll notice that when you run cucumber features you get all of the output from capistrano. This makes your output messy, but provides a lot of valuable debug information. If you want to silence it, you can use any number of tools, including piping the output to logs, or using methods like silence_stream.

You’ll also notice that the setup described above leaves the files in the test_files directory intact after each feature (it wipes it clean before each feature). This makes it easy to inspect the file system manually after each run. While developing, you can even cd into the test_files/app directory and re-run deployments, or tweak the config/deploy.rb file and re-deploy and then move your changes back to templates/deploy.erb.

Next Steps

This is just a quick sample to show you what you can do. It is not meant to be a good example of how to use cucumber (it has lots of instance variables, hard to read steps that are not reusable etc…), but rather a quick example of how to use cucumber to test cap recipes.

You’ll probably want to create a helper class of some sort to wrap up the file system calls, so your steps would look more like:

Given /^a an app$/ do
  MyFileHelper.create_repo
  MyFileHelper.create_app
  MyFileHelper.capify_app
end

Testing against non-local environments

You could in theory use this to test against any environment you have access to – just change the host in templates/deploy.erb. If you choose to test against a true remote machine, you’ll have to figure out how to shell out commands to it.

If you are on a mac, one thing that might help is to mount a remote machine over ssh.

Grab the source

The full source code for this app can be found at:

http://github.com/zilkey/testing-capistrano-demo/tree/master

Introducing AutoTagger – easy multi-stage deployment tagging with git

Pivotal Labs

Monday, March 30, 2009

AutoTagger is a gem that helps you automatically create a date-stamped tag for each stage of your deployment, and deploy from the last tag from the previous environment.

Let’s say you have the following workflow:

Run all test on a Continuous Integration (CI) server
Deploy to a staging server
Deploy to a production server

You can use the autotag command to tag releases on your CI box, then use the capistrano tasks to auto-tag each release.

Installation

gem sources -a http://gems.github.com
sudo gem install zilkey-auto_tagger

To contribute, you can fork the github repository.

You can also visit the tracker project

The autotag executable

Installing the gem creates an executable file named autotag, which takes two arguments: the stage, and optionally the path to the git repo:

$ autotag demo  # => creates a tag like demo/200804041234 in the current directory
$ autotag demo . # => same as above
$ autotag demo /Users/me/foo # => cd's to /Users/me/foo before creating the tag

Running autotag does the following:

$ git fetch origin --tags
$ git tag <stage>/<timestamp>
$ git push origin --tags

Capistrano Integration

AutoTagger comes with 2 capistrano tasks:

release_tagger:set_branch tries to set the branch to the last tag from the previous environment.
release_tagger:create_tag runs autotag for the current stage

Example config/deploy.rb file:

require 'release_tagger'

# The :stages variable is required
set :stages, [:ci, :staging, :production]

# The :working_directory variable is optional, and defaults to Dir.pwd
# :working_directory can be an absolute or relative path
set :working_directory, "../../"

task :production do
  # In each of your environments that need auto-branch setting, you need to set :current_stage
  set :current_stage, :production
end

task :staging do
  # If you do not set current_stage, it will not auto-set your branch
  # set :current_stage, :staging
end

# You need to add the before/ater callbacks yourself
before "deploy:update_code", "release_tagger:set_branch"
after  "deploy", "release_tagger:create_tag"

Assume you have the following tags in your git repository:

ci/01
staging/01
production/01

The deployments would look like this:

cap staging deploy    # => sets branch to ci/01
cap production deploy # => sets branch to staging/01

You can override with with the -Shead and -Stag options

cap staging deploy -Shead=true      # => sets branch to master
cap staging deploy -Stag=staging/01 # => sets branch to staging/01

Known Issues

DOES NOT work with capistrano ext multi-stage
It will accept invalid tag names (if you specify a tag name with a space, it will blow up when you try to create the tag)

I have this working on a single project, and there is a fairly complete spec suite, but use at your own risk. It’s probably worth setting up on a demo project before adding it to a production repo.

Over time it would be nice to add other scms like subversion as well, so that you have a single way of tagging, regardless of scm.

Acknowledgments

Special thanks to Brian Takita, who gave me an initial implementation. Another useful link is http://codeintensity.blogspot.com/2008/06/changelogs-and-deployment-notification.html.

Happy tagging!

Moving from Subversion to Git

Pivotal Labs

Thursday, October 16, 2008

Moving from Subversion to Git

We recently moved our project from subversion to git, and so far the move has gone very smoothly. The following post will detail what we did to make the move.

Our setup

For this project there 2 pairs working and we have 6 machines and one hosted service:

Two Mac OSX workstations with IDEA
One continuous integration server running Cruise Control
A staging server running a 2-year old version of Ubuntu
A subversion server
A production server hosted on Engine Yard
A Github account with the ability to create private repositories

The goal was to have one pair continue to work while the migration from svn to git was happening.

Getting the right software

Every machine must have git installed for this to work, and at least one workstation must have git svn installed. On Mac OSX we were able to install git reliably via macports with sudo port install git-core.

For the workstation that needed git svn it was a little more difficult. On a fresh install of OSX with macports, you should just be able to run sudo port install git-core +svn, but this requires perl bindings for svn and there are a number of dependencies that might be wrong if you’ve installed subversion from source or have conflicting macports.

On one machine, we were able to get past it by ininstalling and reinstalling subversion before running sudo port install git-core +svn

On a modern Ubuntu machine you should be able to run sudo apt-get install git-core, but since our Ubuntu machine is a few versions behind we had to install from source from http://git.or.cz/.

We planned on converting our svn externals to piston once we moved to git, so we also installed piston. We followed the instructions from http://technicalpickles.com/posts/piston-and-git-for-the-win to install it locally, since it’s not available as a standard gem.

Cloning the subversion repository

To clone the repository we used git svn. We only had a trunk on our project, so it was a bit easier for us – to start, all we had to do was:

git svn clone svn://our/repository/trunk

If you have branches and tags you may also want to add the --trunk, --branches and --tags flags as well – see the docs for more info.

For us the cloning took almost half an hour. When it was done we had a branch named git-svn (visible only with git branch -a), which we converted to master with:

git checkout -b master git-svn

We noticed when we looked at git log that we were missing several months from the commit history, even though the git svn command exited without error. To grab the rest of the revisions we ran git svn rebase and it finished the import. I’m not sure if this is a documented behavior or a bug, but it was surprising to us even though it was easily fixed.

Attributing checkins to pairs

We used to add the pair’s initials to the beginning of our svn commit comments, but with git there is a more elegant way to do it by changing the name of the pair.

In it’s simplest form, you can just change the git config user.name every time a new pair sits down. Brian Takita suggested listing all the pairs in the .git/config file, and just uncommenting the right one in the morning, which saves some time.

There are much more complex solutions, like [this][http://www.brynary.com/2008/9/1/setting-the-git-commit-author-to-pair-programmers-names] if you like as well (well worth the read even if you don’t end up using it)

Re-ignoring files

Any files that were ignored in svn are no longer ignored in git. We added several standard files to our ignore list, like the

log/*
IDEA project files
.DS_Store

Ignoring the log entries seemed to remove the log directory entirely, which caused us to have to manually create the log file on our different servers, but seemed like the right thing to do.

Grabbing externals

Our git svn clone did not pull in any of our externals. We added these back using piston, although it only worked for a few of the externals. The latest version of piston has support for adding svn repositories to git projects. The syntax is the same as it normally is:

cd your-project
piston import svn://some/svn/repo vendor/plugins/some_plugin

For us this worked about half the time, and the other half it hung endlessly and never finished the svn checkout. We still don’t know why that happened. For the plugins that we couldn’t piston we just svn exported those to our project, and once we figure out how to fix piston we’ll piston those as well.

Brian Takita mentioned that when there is a big pending changelist, piston becomes slower on git. In our case, we tried when there were no pending changes, and it still hung.

Adding git support to IDEA

Since we use IDEA for development, we decided to install a git plugin for IDEA to make it easier to view. We decided on git4idea because the project has been more active on github recently and seemed to have a decent feature set.

git clone git://github.com/markscott/git4idea.git
cd git4idea
cp Git4Idea.jar /Library/Applications/<your IDEA directory>/plugins

Then restart idea to make sure it takes effect. We added the .git directory to the list of ignored modules so that it wouldn’t appear in search history as well. A few things to note with this plugin:

when you commit, it does not push – the git workflow typically involves committing locally and then pushing in two separate steps
when you push it will ask you for a target – the default is default/origin or something like that – in our case typing “origin” works
the project explorer seems to get out of date easily. We run Version Control > Refresh File status often whenever it looks funky (or type “Alt + C, E” if you’ve got the alt keys working)
when you commit, there are annoying comments in the commit message textarea – it’s focused by default, so you can easily delete it, but it’s still annoying

We haven’t used the plugin extensively, but so far it looks decent.

When switching IDEA projects, you might find it helpful to copy your old project files into the new project to save some time re-setting your settings.

Pushing to github

Once you are setup, you can push to github. In our case we did the following:

added our local machine’s public ssh key to the account’s ssh keys (on githubs main account page)
created a new repository and marked it as private
followed the github instructions to add the remote repo and push to origin master

We decided to only push the master branch to github, since we wouldn’t be using git svn for more than a few hours as we migrated all of the machines. We didn’t test the continued use of git svn after cloning the repo, so if you need to continue with svn you may need to push the original branch as well, or change your .git/config file to make sure that dcommits still work etc…

Once we had it on github, we renamed our original project directory and did a fresh git clone of the github project, then:

updated our .git/config again with our pair names (alternately, you can set the config variables globally, like git config --global user.name 'foo & bar')
added the log directory (mkdir log)

Updating CI

To update the CI box, we:

Added a deploy key to github
Stopped all running cruise processes
Renamed our current project to -SVN
Updated to the latest version of CCRB
Added our new project with the standard cruise command
Updated our “scratch pad” checkout from the svn repo to the git repo
Rebooted CI to make sure everything worked (you probably don’t need to reboot)

Github provides the ability to create read-only accounts for specific repositories. These users are identified by public keys and they must be unique across github. For us, we had to:

Log into our CI box
Create a public/private key pair with ssh-keygen
Copy the public key to github’s deploy key area in the Project Name > Admin section

Then we updated to the latest version of Cruise Control by going to CCRB directory and running git pull. If you’ve added Cruise Control by some other method, you’ll have to update to the latest source from http://github.com/thoughtworks/cruisecontrol.rb/tree/master.

In our setup, our projects are in ~/.cruise/projects. Cruise Control loops through every directory in /projects and loads it’s cruise_config.rb file, so you can have multiple builds running at the same time. After renaming our original project, we added the new project by going to ~/.cruise and typing:

~/.cruise./cruise add your-projectname --repository git@github.com:projectname.git --source-control git

This checked out the repository and added the necessary config file. We then went in and manually created the log directory.

On our CI box we also have a scratch-pad checkout of our code, useful for debugging, located at ~/workspace/project-name. We blew that away and git cloned our repo, then added the log directory.

Once the CI build is green, you can delete the old project that was based on svn.

Git on capistrano

To deploy our app to our staging server, we:

Created a deploy key for the staging server and added it to github
Deleted all files from the remote cache directory
Updated deploy.rb to use git
Deployed twice (the second time to ensure that it worked from the remote cache)

We followed the excellent guide here to get our deploy settings correct.

The only stumbling block we found was that we use deploy_via :remote_cache, which stores a checkout of the repository on the server to make deployments faster. Since the remote cache had a subversion checkout, it was necessary to delete all files in the cache before deploying.

After the cache was cleared and deploy.rb was updated we deployed twice, both times without incident.

Deploying git on Engine Yard

We haven’t had the opportunity to deploy on git to EY, but given how easy it was to deploy on our staging server we anticipate that EY will be easy to deploy.

Josh Susser pointed out that because Github itself if hosted on Engine Yard, git deploys from git repos on Github to slices on Engine Yard servers are blazingly fast.

Managing the transition

While one pair was updating these servers, the other pair was checking into the existing subversion repository. Once all the work stations were set up, CI was up and deployments were working, we ran a final svn rebase and pushed to github. It looked something like this:

cd svn-project-dir
svn commit -m "made some well tested changes"`
cd git-project-dir
git pull        # => gets all of the changes that the other pair made, i.e. pistoned directories etc...
git svn rebase  # => gets latest changes from svn
git push        # => sends changes to github

So by the end of the day, there was almost no interruption for one pair.

Timing

All in all it took about a day for a single pair, but if we could do it again knowing what we know now we could probably do it in about half that time.

References

New York Standup 10/9/2008

Pivotal Labs

Thursday, October 9, 2008

Helps

What is capistrano multistage?

A plugin that allows you to store environment-specific variables in different files, and specify a default environment
It’s been on tracker for a while and seems to be stable
Other options are just specifying your environment-specific variables within separate tasks – keeping everything in one file