2010 Comparing indexed date columns, git subtrees, Headless Selenium for CI

Friday, December 17, 2010

Helps

Is there a way to get MySql indexing to speed up queries involving greater and less than operators on date columns?

Postgres handles these operators a little bit better than MySql, but may not actually solve this problem.

Using millis instead of dates would give the DB the best chance of handling this scenario.

We are using Git’s subtree merge facility instead of submodules to stay synced to a different repository for part of our project. How do we push changes back to that repo?

See Tim Connor’s blog post “Git sub-tree merging back to the subtree for pushing to an upstream“. Early in that post is a pointer to an article describing the the subtree merge operation. Tim goes on to explain how to push your changes back through the chain.

Interesting

Some cloud environments leave the names of temp files visible even when their contents are not accessible. Be sure to use obfuscated names for your temporary files!
The “Headless” gem allows you to easily set up an alternate “display” that allows programs to execute in a headless environment. See this blog post about how to use Headless to run Selenium tests on a CI box: http://www.aentos.com/blog/easy-setup-your-cucumber-scenarios-using-headless-gem-run-selenium-your-ci-server
Ccrb will bog down to painfully slow levels if more than a couple of CC Tray clients are pinging it repeatedly during a build.
Cron will not honor your .rvmrc file unless you do some work to set up the environment. If you set up your cron job like this:

0 6 * * * /bin/bash -l -c 'echo /home/someuser/.rvm/bin/rvm rvmrc trust ... && cd ...

the -l & -c parameters cause bash to load your environment as if your were logging in before running the specified commands. Someone also mentioned that rvm-shell can be used as a solution to this problem.

iPhone UI Automation tests: a decent start

Adam Milligan

Tuesday, July 20, 2010

Apple’s inclusion of the UI Automation component in Instruments with iOS 4 is a definite step in the right direction. It’s the first reasonable way to write tests that externally exercise your actual app, rather than weirdly injecting test code into it. It’s also the only way to programmatically test lifecycle issues, such as how your app behaves when put in the background, when rotated, when the device locks, etc. Good stuff. Unfortunately, the current implementation of UI Automation also has some significant problems:

There’s no way to run tests from the command line. The subtitle of the WWDC talk for UI Automation was “find bugs while you sleep;” unfortunately, you can’t find bugs while you sleep if you have to wake up to click the “Run” button.
There’s no way to set up or reset state. The lack of fixtures which set up a known state at the beginning of iPhone tests has been a problem for unit testing (with OCUnit, Cedar, or what have you), particularly for apps that use CoreData. Now it’s worse than ever, because UI Automation manipulates the actual state of the app on the device, much like Selenium does in a browser. Sadly, UI Automation provides no method for reseting the device’s state, making it nigh impossible to prevent tests from affecting one another.
Part of the previous problem is that UI Automation has no concept of discrete tests; it provides no form of organization for your test scripts. No test methods, no set up or tear down methods, just one big stream of consciousness line of execution. Obviously you can break this up into functions as you see fit, but why reinvent the wheel? Since the test script is JavaScript, I like the idea of using Jasmine for this.
There’s no way to programmatically retrieve the results of the test run. You could debate the value of solving this issue at the moment, considering there’s no way to programmatically start the tests either. However, even if you were to write some clever AppleScript to kick off the tests automatically the only indication of the pass/fail status is in the Instruments UI, so you still have to wake up to check the results. I searched around a bit for information on deconstructing the protocol that UI Automation uses to talk to the device, but I came up empty.

I’ll definitely use UI Automation, particularly for app lifecycle testing. But, not being able to add those tests to a CI build definitely stings. I very much hope Apple keeps their momentum for automated testing and makes it more developer-friendly.

Cedar device specs and CI

Adam Milligan

Tuesday, July 6, 2010

One of the most common complaints I’ve read about OCUnit, the unit testing framework built into Xcode, is that the tests you write with it won’t run on the device. In addition, I personally have found the process of setting up a target for tests that depend on UIKit confusing and onerous. So, one of our goals for Cedar was to make testing UI elements easy (or easier), by making it easy to run specs in the simulator or on the device.

Probably the second most common complaint I’ve read about OCUnit is that the tests run as part of the build. This makes the test output difficult to separate from the build output, and makes it impossible to use the debugger when running tests. So, in addition to making it easy to run specs on the device, we wanted to be able to run them as a separate, debuggable executable.

Finally, we consider it important that our specs run in our CI system. That means we wanted to be able to run Cedar specs from the command line, and get an exit code signifying success or failure. At the same time, some of us appreciate the value of the green/red feedback for specs passing and failing, so sometimes we like a nice UI. As of today, Cedar will accommodate all of these various requirements.

If you have simple tests that don’t depend on iOS, you can simply link the Cedar dynamic framework into an OS X command line executable build target, write your specs, and run. Running from inside Xcode should give you output that looks something like this:
Xcode spec run pass

For specs that do depend on iOS frameworks, such as custom views or view controllers, you can link the Cedar static framework into an iOS executable, use the Cedar-specific application delegate (CedarAppDelegate), and you get something that looks like this:
Xcode UI spec run pass

The elements in the table represent the top-level specs that you have written. You can click on each one to drill into the state of each of its children, and its childrens’ children, etc.

Of course, those nice green bars aren’t terribly useful for the CI build, so if you’d prefer to skip the UI you can set the CEDAR_HEADLESS_SPECS environment variable, and then running the specs on the simulator (or device) reverts back to the dots:
Xcode headless UI spec run pass

Notice that we’ve created rake tasks for running the specs on the command line, for the sake of simplicity and ease of integration with CCRB. The Rakefile is available in the Cedar project, and does little more than set environment variables and execute bash shell commands. If you’d prefer to use your own bash script, or something similar, feel free to copy and paste your way to happiness.

I’ll spare you another screenshot, but you can see the results and build logs for the Cedar CI build here, and its current status is always displayed on our public CI Monitor page.

The last piece of the testing puzzle for iOS is full-stack integration and lifecycle testing (à la Selenium), which should be possible with Apple’s new UI Automation tool. Unfortunately, Apple hasn’t yet provided a way to run UI Automation tests, or get test results, from the command line. Hopefully we’ll find a way soon; their JavaScript DSL for iOS testing combined with Jasmine could be a potent combination.

Bundler + Cruise… Take Two

Pivotal Labs

Wednesday, November 25, 2009

Responding to my last post, Josh pointed out what should probably be obvious: It’s likely a bad idea to bundle bundler. There’s a potential for version conflicts.

For our second attempt, we’re

As a second attempt, we’re now cribbing from the continuous integration setup from the Rails project. So far, so good:

Our RAILS_ROOT/cruise_config.rb…

require 'fileutils'

Project.configure do |project|
  project.build_command = 'sudo gem update --system && ruby lib/cruise/build.rb'
end

And, the referenced lib/cruise/build.rb (the important parts)…

#!/usr/bin/env ruby
require 'fileutils'
include FileUtils

def root_dir
    @root_dir ||= File.expand_path(File.dirname(__FILE__) + '/../..')
end

def rake(*tasks)
  tasks.each { |task| return false unless system("#{root_dir}/bin/rake", task, 'RAILS_ENV=test')}
end

build_results = {}

cd root_dir do
  build_results[:bundle] = system 'gem bundle'  # bundling here, rather than in a task (not in Rails context)
  build_results[:spec] = rake 'cruise:spec'
end

failures = build_results.select { |key, value| value == false }

if failures.empty?
  exit(0)
else
  exit(-1)
end

Thanks go to

Josh Susser for help via email
John Pignata for suggesting we look at the Rails project
Rails team for the reference scripts

More comments and suggestions are encouraged.

Bundle(r) Yourself… and Get Ready to Cruise!

Pivotal Labs

Tuesday, November 17, 2009

On a current project, we’ve just switched from GemInstaller to Bundler for managing our application’s gems.

All in all, the transition was painless… in our development environments. Of course, in order to keep things on running smoothly on the continuous integration box, we amended our rake cruise:spec task to start by running sh "gem bundle".

So, wonderful! Any changes to our gem dependency list will picked up when cruise starts and made available for that “build”; there’s no need to log in and make any manual updates. Right?

Not quite. We’re using the disable_system_gems option and, in that case, Bundler (very intentionally) modifies your GEM_PATH such that only “vendor’d” gems are available to the application. Which of course means that Bundler itself, being “a gem to bundle gems”, is unavailable when that sh "gem bundle" command is run.

Our solution: bundle Bundler, obviously! That’s right, in our Bundler Gemfile we’ve included gem "bundler". Now, after a single manual execution of gem bundle to pick up the bundled Gem Bundler gem bundle (heh), any subsequent Gemfile changes (to gems other than Bundler) get picked up at the start of the build… and we’re cruisin’

Have another solution? Please let us know in the comments.

Standup 3/25/2009: Branches + JSUnit + CI + IE = :-(

Pivotal Labs

Wednesday, March 25, 2009

Interesting Things

Branches + JsUnit + CI + IE = :-( : Apparently it is difficult to manage IE’s cache in CI. One project apparently has a bat file on CI that clears the cache every 30 minutes. Another team solved this by making the cache directory read only. Often browser/OS combinations have some technique for disabling caching.
Test Swarm Alpha: this is a crowd sourced javascript testing solution (think seti@home for javascript testing) being developed by John Resig.

Ask for Help

“AR attribute appears to be skipped by text field helper?!” Apparently the model method is bypassed by the text field helper if a column of the same name is present in the underlying table. This was experienced in Rails 2.2.

Others have apparently experienced this in the past but a clear answer did not surface.