At Pivotal, our default choice for CI is Jenkins. I feel that Jenkins does a fine job running builds and reporting on a pass/fail. It also has nice plugins and plenty of features like build labeling and clustering.
JetBrains' TeamCity has all that, and more!
To be fair, TeamCity is missing some things that Jenkins has. However, I don't remember what they are because I've never used some of the deeper features of Jenkins. One thing TeamCity doesn't support well is running builds for multiple branches dynamically.
That said, I'm a huge fan of TeamCity. Here's why.
Interesting Things
A standup of pivots (Sam Coward, Sean Moon, Peter Jaros, and Brent Wheeldon) have recently run into a bug in TeamCity that causes trouble with symlinks when using git.
If you commit symlinks into your repo, TeamCity will not properly transfer these to the build agents. They end up being copied over to the agents as plain text files.
The workaround for this issue is to use Agent-side Checkouts instead of Server-side Checkouts.
Helps
Is there a way to get MySql indexing to speed up queries involving greater and less than operators on date columns?
Postgres handles these operators a little bit better than MySql, but may not actually solve this problem.
Using millis instead of dates would give the DB the best chance of handling this scenario.
We are using Git's subtree merge facility instead of submodules to stay synced to a different repository for part of our project. How do we push changes back to that repo?
See Tim Connor's blog post "Git sub-tree merging back to the subtree for pushing to an upstream". Early in that post is a pointer to an article describing the the subtree merge operation. Tim goes on to explain how to push your changes back through the chain.
Interesting
Some cloud environments leave the names of temp files visible even when their contents are not accessible. Be sure to use obfuscated names for your temporary files!
The "Headless" gem allows you to easily set up an alternate "display" that allows programs to execute in a headless environment. See this blog post about how to use Headless to run Selenium tests on a CI box: http://www.aentos.com/blog/easy-setup-your-cucumber-scenarios-using-headless-gem-run-selenium-your-ci-server
Ccrb will bog down to painfully slow levels if more than a couple of CC Tray clients are pinging it repeatedly during a build.
Cron will not honor your .rvmrc file unless you do some work to set up the environment. If you set up your cron job like this:
0 6 * * * /bin/bash -l -c 'echo /home/someuser/.rvm/bin/rvm rvmrc trust ... && cd ...
the -l & -c parameters cause bash to load your environment as if your were logging in before running the specified commands. Someone also mentioned that rvm-shell can be used as a solution to this problem.
Apple's inclusion of the UI Automation component in Instruments with iOS 4 is a definite step in the right direction. It's the first reasonable way to write tests that externally exercise your actual app, rather than weirdly injecting test code into it. It's also the only way to programmatically test lifecycle issues, such as how your app behaves when put in the background, when rotated, when the device locks, etc. Good stuff. Unfortunately, the current implementation of UI Automation also has some significant problems:
- There's no way to run tests from the command line. The subtitle of the WWDC talk for UI Automation was "find bugs while you sleep;" unfortunately, you can't find bugs while you sleep if you have to wake up to click the "Run" button.
- There's no way to set up or reset state. The lack of fixtures which set up a known state at the beginning of iPhone tests has been a problem for unit testing (with OCUnit, Cedar, or what have you), particularly for apps that use CoreData. Now it's worse than ever, because UI Automation manipulates the actual state of the app on the device, much like Selenium does in a browser. Sadly, UI Automation provides no method for reseting the device's state, making it nigh impossible to prevent tests from affecting one another.
- Part of the previous problem is that UI Automation has no concept of discrete tests; it provides no form of organization for your test scripts. No test methods, no set up or tear down methods, just one big stream of consciousness line of execution. Obviously you can break this up into functions as you see fit, but why reinvent the wheel? Since the test script is JavaScript, I like the idea of using Jasmine for this.
- There's no way to programmatically retrieve the results of the test run. You could debate the value of solving this issue at the moment, considering there's no way to programmatically start the tests either. However, even if you were to write some clever AppleScript to kick off the tests automatically the only indication of the pass/fail status is in the Instruments UI, so you still have to wake up to check the results. I searched around a bit for information on deconstructing the protocol that UI Automation uses to talk to the device, but I came up empty.
I'll definitely use UI Automation, particularly for app lifecycle testing. But, not being able to add those tests to a CI build definitely stings. I very much hope Apple keeps their momentum for automated testing and makes it more developer-friendly.
One of the most common complaints I've read about OCUnit, the unit testing framework built into Xcode, is that the tests you write with it won't run on the device. In addition, I personally have found the process of setting up a target for tests that depend on UIKit confusing and onerous. So, one of our goals for Cedar was to make testing UI elements easy (or easier), by making it easy to run specs in the simulator or on the device.
Probably the second most common complaint I've read about OCUnit is that the tests run as part of the build. This makes the test output difficult to separate from the build output, and makes it impossible to use the debugger when running tests. So, in addition to making it easy to run specs on the device, we wanted to be able to run them as a separate, debuggable executable.
Finally, we consider it important that our specs run in our CI system. That means we wanted to be able to run Cedar specs from the command line, and get an exit code signifying success or failure. At the same time, some of us appreciate the value of the green/red feedback for specs passing and failing, so sometimes we like a nice UI. As of today, Cedar will accommodate all of these various requirements.
Responding to my last post, Josh pointed out what should probably be obvious: It's likely a bad idea to bundle bundler. There's a potential for version conflicts.
For our second attempt, we're
As a second attempt, we're now cribbing from the continuous integration setup from the Rails project. So far, so good:
Our RAILS_ROOT/cruise_config.rb...
require 'fileutils'
Project.configure do |project|
project.build_command = 'sudo gem update --system && ruby lib/cruise/build.rb'
end
And, the referenced lib/cruise/build.rb (the important parts)...
#!/usr/bin/env ruby
require 'fileutils'
include FileUtils
def root_dir
@root_dir ||= File.expand_path(File.dirname(__FILE__) + '/../..')
end
def rake(*tasks)
tasks.each { |task| return false unless system("#{root_dir}/bin/rake", task, 'RAILS_ENV=test')}
end
build_results = {}
cd root_dir do
build_results[:bundle] = system 'gem bundle' # bundling here, rather than in a task (not in Rails context)
build_results[:spec] = rake 'cruise:spec'
end
failures = build_results.select { |key, value| value == false }
if failures.empty?
exit(0)
else
exit(-1)
end
Thanks go to
- Josh Susser for help via email
- John Pignata for suggesting we look at the Rails project
- Rails team for the reference scripts
More comments and suggestions are encouraged.
On a current project, we've just switched from GemInstaller to Bundler for managing our application's gems.
All in all, the transition was painless... in our development environments. Of course, in order to keep things on running smoothly on the continuous integration box, we amended our rake cruise:spec task to start by running sh "gem bundle".
So, wonderful! Any changes to our gem dependency list will picked up when cruise starts and made available for that "build"; there's no need to log in and make any manual updates. Right?
Not quite. We're using the disable_system_gems option and, in that case, Bundler (very intentionally) modifies your GEM_PATH such that only "vendor'd" gems are available to the application. Which of course means that Bundler itself, being "a gem to bundle gems", is unavailable when that sh "gem bundle" command is run.
Our solution: bundle Bundler, obviously! That's right, in our Bundler Gemfile we've included gem "bundler". Now, after a single manual execution of gem bundle to pick up the bundled Gem Bundler gem bundle (heh), any subsequent Gemfile changes (to gems other than Bundler) get picked up at the start of the build... and we're cruisin'
Have another solution? Please let us know in the comments.
Interesting Things
- Branches + JsUnit + CI + IE = :-( : Apparently it is difficult to manage IE's cache in CI. One project apparently has a bat file on CI that clears the cache every 30 minutes. Another team solved this by making the cache directory read only. Often browser/OS combinations have some technique for disabling caching.
- Test Swarm Alpha: this is a crowd sourced javascript testing solution (think seti@home for javascript testing) being developed by John Resig.
Ask for Help
"AR attribute appears to be skipped by text field helper?!" Apparently the model method is bypassed by the text field helper if a column of the same name is present in the underlying table. This was experienced in Rails 2.2.
Others have apparently experienced this in the past but a clear answer did not surface.