Helps
- Capybara WebKit + Twitter Bootstrap icon asplodes
One of my integration tests started to fail (Broke Pipe). Something to do with font-awesome & Qt?
Pivot Pong Grand Finals
Wednesday, October 10, 2012
Monthly Archives: October 2012
Using DeployStudio Across Subnets—a Path not Taken
Tuesday, October 9, 2012
At Pivotal Labs we use DeployStudio to rapidly image machines over the network. It was an excellent solution when the DeployStudio server and the client were on the same subnet. It did not work when they were on different subnets.
We found that, with a combination of clever use of tcpdump, a carefully-crafted dhcpd configuration file, and a judicious set of firewall exceptions, we were able to extend DeployStudio so that it worked across subnets.
Unfortunately, it was an epic fail: every third install would cause our firewall (m0n0wall 1.8.0b512) to lock up. We have put the project on ice until we get a new firewall.
Audience
This blog post is intended for IT organizations with the following characteristics
- use DeployStudio to deploy OS X workstations
- have multiple subnets
- are uncomfortable having a DeployStudio server span multiple networks (most often these are security concerns; by compromising the DeployStudio server, a hacker would gain access to all the networks) (a DeployStudio server must run several services, at least one of which, NFS, requires discipline to implement in a secure manner)
- use an ISC DHCP server
- are willing to put their firewall to the test
The easy way
See Ryan’s comments below. With a few lines of Cisco configuration (assuming you have a Cisco router), you can easily configure DeployStudio boots across subnets.
The rest of this blog post is the much more difficult path that I took, and I don’t recommend it unless you really enjoy doing things the hard way.
The Hard Way: Start with tcpdump
To make DeployStudio work across subnets, you first need to use tcpdump to capture how it works within a subnet. In this case, we used a laptop (kate-enet), and our DeployStudio server (deploystudio).
First, we started the capture. We captured to a file so that we could examine the output at our leisure. We ran the following command on our deploystudio server:
sudo tcpdump -w /tmp/kate.tcp -s 1536 host kate-enet
Next, we started a network install:
- we turned on kate-enet (a 13″ MacBook Air laptop with a thunderbolt ethernet adapter)
- we held down the option-key so that we were presented with a choice of boot options
- we chose the network install
- when DeployStudio runtime screen came up, we ctrl-c’d the tcpdump—we had what we needed.
Then we examined the tcpdump file using the following command:
sudo tcpdump -r /tmp/kate.tcp -vvv | less
There were two packets we were particularly interested in:
deploystudio.sf.pivotallabs.com.bootps > kate-enet.sf.pivotallabs.com.bootpc: [bad udp cksum 2b5a!] BOOTP/DHCP, Reply, length 319, Flags [none] (0x0000)
Client-IP kate-enet.sf.pivotallabs.com
Client-Ethernet-Address 40:6c:8f:3d:e6:b4 (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: ACK
Server-ID Option 54, length 4: deploystudio.sf.pivotallabs.com
Vendor-Class Option 60, length 9: "AAPLBSDPC"
Vendor-Option Option 43, length 56: 1.1.1.4.2.127.209.7.4.130.0.4.56.8.4.130.0.4.56.9.35.130.0.4.56.30.49.48.46.56.95.109.97.99.95.109.105.110.105.95.115.101.114.118.101.114.45.50.48.49.50.45.48.56.48.54
END Option 255, length 0
And
deploystudio.sf.pivotallabs.com.bootps > kate-enet.sf.pivotallabs.com.bootpc: [bad udp cksum 254b!] BOOTP/DHCP, Reply, length 379, Flags [none] (0x0000)
Client-IP kate-enet.sf.pivotallabs.com
Server-IP deploystudio.sf.pivotallabs.com
Client-Ethernet-Address 40:6c:8f:3d:e6:b4 (oui Unknown)
sname "deploystudio.sf.pivotallabs.com"
file "/private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter"
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: ACK
Server-ID Option 54, length 4: deploystudio.sf.pivotallabs.com
Vendor-Class Option 60, length 9: "AAPLBSDPC"
RP Option 17, length 93: "nfs:10.80.28.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg"
Vendor-Option Option 43, length 21: 1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48
END Option 255, length 0
Note:
- You can ignore any ‘bad cksum’ messages: those messages are an artifact of the checksums being calculated by the ethernet hardware (TCP checksum offloading) instead of by the kernel.
- deploystudio responds to DHCP queries even though it is not a DHCP server. It is not dishing out IP addresses; it is merely providing additional data for netbooting to work.
There are 4 crucial pieces of data that you must capture.
- The file directive
- The RP Option 17
- The two Vendor-Option Option 43
We then added the information we had culled from the tcpdump to our dhcpd.conf file (special thanks to Pepijn Oomen and Bennett Perkins; see bibliography):
class "netboot" {
match if substring (option vendor-class-identifier, 0, 9) = "AAPLBSDPC";
option dhcp-parameter-request-list 1,3,17,43,60;
if (option dhcp-message-type = 1) {
option vendor-class-identifier "AAPLBSDPC";
option vendor-encapsulated-options
08:04:81:00:00:89; # bsdp option 8 (length 04) -- selected image id;
} elsif (option dhcp-message-type = 8) {
option vendor-class-identifier "AAPLBSDPC";
if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
log(debug, "bsdp_msgtype_list");
# bsdp image list message:
# one image, plus one default image (both are the same)
option vendor-encapsulated-options
01:01:01:04:02:7f:d2:07:04:82:00:04:38:09:23:82:00:04:38:1e:31:30:2e:38:5f:6d:61:63:5f:6d:69:6e:69:5f:73:65:72:76:65:72:2d:32:30:31:32:2d:30:38:30:36;
} else {
log(debug, "bspd_msgtype_select");
# details about the selected image
#
option vendor-encapsulated-options
01:01:02:08:04:82:00:04:38:82:0a:4e:65:74:42:6f:6f:74:30:35:30;
next-server deploystudio.sf.pivotallabs.com;
filename "/private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter";
option root-path = "nfs:10.0.0.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg";
}
}
}
Resist the temptation to substitute a hostname for the NFS server’s IP address; (i.e. leave it “nfs:10.0.0.64″; do not put “nfs:deploystudio.sf.pivotallabs.com”). IP addresses will work; hostnames won’t.
We used ruby (irb) to convert the dotted-decimal strings in tcpdump to colon-hexadecimal in dhcpd.conf. In the following example, we convert “1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48″:
bc$ irb
1.9.3p194 :001 > string="1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48"
=> "1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48"
1.9.3p194 :002 > string.split(".").each { |n| printf("%02x:",n) }; p
01:01:02:08:04:82:00:04:38:82:0a:4e:65:74:42:6f:6f:74:30:35:30: => nil
Firewall Rules
If you have a firewall arbitrating traffic between the subnets, you’ll need to allow all inbound traffic to your DeployStudio server. Additionally, if your firewall can’t snoop TFTP traffic, you’ll need to allow outbound UDP traffic on unreserved ports (1024 – 65535).
Troubleshooting
If you’re having problems, you need to check that your TFTP and NFS are working, preferably from a machine that’s on the subnet of the client which your trying to image.
TFTP
In our example, we know that our tftp server is deploystudio.sf.pivotallabs.com, and the file we’re downloading is /private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter. Let’s try from the command line:
bc $ tftp deploystudio.sf.pivotallabs.com
tftp> get /private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter
Received 993680 bytes in 18.3 seconds
NFS
Testing NFS is a little tricky because the NFS path is slightly mangled. Specifically, a “:” is substituted for the second-to-last “/” in the pathname. For example, the dhcp root-path directive “nfs:10.80.28.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg”
is translated to a pathname of “/net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg” for testing purposes on a client machine. We take advantage of automount running on a typical OS X client. First do an ls to make sure we can see the file, then do a cp to make sure we can read the file:
ls /net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg
cp /net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg /dev/null
Performance
The time required to image a machine will more than double. A typical install will take 40 minutes or more.
Initial Boot-up
Certain operations are much slower. Specifically, the time between selecting netboot server and being presented with the DeployStudio runtime screen takes approximately 7 minutes. We have studied that lag, and over 4 minutes is due to abysmal (3.8kBps) TFTP throughput. We are unclear why there is such a gross lag; running the same tftp on the command line completes 20x faster (74.7kBps).
We have a firewall that negotiates traffic between our subnets, and we are aware that TFTP provides challenges for firewalls (it re-negotiates its destination port) (Cisco firewalls have special directives to handle TFTP traffic appropriately).
Bibliography
- Bennett Perkins’s blog post
- Pepijn Oomen’s blog post
- TFTP RFC
10/08/12: Movember Approaches
Monday, October 8, 2012
Helps
- Ajax login form not saving autocomplete values -
Our ajax login form never saves autocomplete values in Chrome. Has anyone else dealt with this before?
Possible solutions include submitting the form to an iframe or finding a way to intercept a submit event after Chrome saves the form data.
Interestings
- MOVEMBER is coming…
The website is back open for 2012. Please sign-in. Also, talk to your clients – we’d like to double our participation yet again this year!
Why not to use ARC
Sunday, October 7, 2012
If you have done any development for iOS in the past few years you have at least some familiarity with ARC. The overall response to ARC since Apple released it with iOS 5 has been little short of orgasmic. You can’t swing a dead internet cat without hitting a blog post from someone explaining how ARC saved his/her dying grandmother and if you’re not using it on every project you touch then you’re helping the Commies win.
I’ve seen some projects do perfectly well with ARC, but at the same time I feel it provides its own set of challenges which we should not overlook. Here are some reasons why you might want to consider not using ARC on your next Objective C project.
ARC probably isn’t solving the problem you need it to
Memory management. Programmers whisper these words in fearful tones, or brazenly avow the impossibility of doing it correctly. ARC takes care of memory management, thus solving one of the great problems of our generation, right?
As it turns out, memory management is actually quite simple; /relationship/ management presents the challenge. Memory problems are usually a result of poor relationship management. Object A leaks because objects B and C both have an ownership relationship to A, A has an ownership relationship to C, etc.
In simple cases ARC will clean up the unused memory for you, but you’re still left with a poorly designed object graph, and all its associated problems. In more complex cases you can have strong circular references — a common result of a messy object graph — and then even ARC can’t prevent the leaks.
ARC makes easy things easier, but difficult things more difficult
Here’s a simple example of a class that has a simple relationship to another class:
Class declaration without ARC:
@interface Person : NSObject @property (nonatomic, retain) Wallet *wallet; @end
Class declaration with ARC:
@interface Person : NSObject @property (strong, nonatomic) Wallet *wallet; @end
Not much difference here. Here’s a bit of the implementation:
Initializer without ARC:
- (id)init {
if (self = [super init];) {
self.wallet = [[[Wallet alloc] init] autorelease];
}
return self;
}
- (void)dealloc {
self.wallet = nil;
[super dealloc];
}
Initializer with ARC:
- (id)init {
if (self = [super init];) {
self.wallet = [[Wallet alloc] init];
}
return self;
}
In this simple case ARC makes life a little easier; you don’t have to type autorelease, and you don’t have to write a dealloc method to release owned objects. With a well designed object graph, these two things are the vast majority of memory management you need to do. They’re more tedious than difficult, but ARC makes them go away. That’s pretty handy, right?
Now let’s look at an example of retrieving a value from the iOS Keychain, which is a bit more complicated:
Without ARC:
NSMutableDictionary *query = [NSMutableDictionary dictionary];
[query setObject:kSecClassGenericPassword forKey:kSecClass];
[query setObject:[NSData dataWithBytes:SOME_ID length:ID_LENGTH] forKey:kSecAttrGeneric];
[query setObject:(id)kCFBooleanTrue forKey:kSecReturnData];
NSData *result = nil;
NSString *value = nil
if (errSecSuccess == SecItemCopyMatching((CFDictionaryRef)query, (CFTypeRef *)&result)) {
value = [[[NSString alloc] initWithData:[result autorelease] encoding:NSUTF8StringEncoding] autorelease];
}
With ARC:
NSMutableDictionary *query = [NSMutableDictionary dictionary];
[query setObject:(__bridge id)kSecClassGenericPassword forKey:(__bridge id)kSecClass];
[query setObject:[NSData dataWithBytes:SOME_ID length:ID_LENGTH] forKey:(__bridge id)kSecAttrGeneric];
[query setObject:(id)kCFBooleanTrue forKey:(__bridge id)kSecReturnData];
CFDataRef result = NULL;
NSString *value = nil;
if (errSecSuccess == SecItemCopyMatching((__bridge CFDictionaryRef)query, (CFTypeRef *)&result)) {
value = [[NSString alloc] initWithData:(__bridge_transfer NSData *)result encoding:NSUTF8StringEncoding];
}
ARC saves us from two autorelease calls, but at the cost of five __bridge casts and a __bridge_transfer cast. ARC won’t let you declare the result variable as an NSData, since ARC disallows casting indirect pointers entirely. Thus, when you receive the result in a CFDataRef you’re still responsible for releasing that memory. You could use CFRelease() on it, or, as shown here, __bridge_transfer the CFDataRef into an NSData * temporary that ARC cleans up at the end of the execution of the statement.
Not exactly simpler, is it?
Sometimes ARC doesn’t help at all
Do you see the memory leak in this code?
self.thing = [Thing thing];
self.thing.completeNotification = ^{
[self spreadTheWord];
};
[self.thing startTask];
By default, Objective C blocks retain everything they refer to from the local scope, including the self pointer. In this case, self retains the Thing, the Thing retains the block, and the block retains self. This creates a circular reference and a memory leak, with or without ARC.
Here’s how you fix it without ARC:
__block ThisClass *this = self;
self.thing.completeNotification = ^{
[this spreadTheWord];
};
and with ARC:
__weak ThisClass *this = self;
self.thing.completeNotification = ^{
[this spreadTheWord];
};
Even with ARC you still have to think about managing memory in some cases. If you rely on ARC to handle all of your memory this type of leak is probably more likely to happen.
The ARC compiler is broken
So ARC may not be helping as much as you thought, but at least it’s not really hurting anything, right? Unfortunately, turning on ARC injects bugs into the compiler. Until LLVM 4.1 (in XCode 4.5) code containing C++ templates wouldn’t even compile with ARC enabled. While LLVM 4.1 is a significant improvement over its predecessors, now the code compiles but fails to work correctly. Here’s an example of a simple function template with a specialization:
class Thing {
public:
template<typename T>
void do_something(const T &);
};
template<typename T>
void Thing::do_something(const T &) {
NSLog(@"Do something with generic type");
}
template<>
void Thing::do_something(UIView * const &) {
NSLog(@"Do something with UIView *");
}
</typename></typename>When invoked:
Thing thing; thing.do_something(someViewController.view);
here is the output without ARC:
2012-10-07 20:17:57.387 Project[65248:c07] Do something with UIView *
and with ARC:
2012-10-07 20:18:54.400 Project[65984:c07] Do something with generic type
What does C++ template type resolution have to do with Objective C reference counting? Nothing, as far as I can tell. It seems the ARC compiler is working off a different branch than the non-ARC compiler; a branch with significant unrelated defects. Who knows what other problems lie in wait?
Sometimes ARC is really, really broken
If you try digging around in the more esoteric capabilities of Objective C, ARC will sometimes get confused and do the wrong thing without warning you. For instance, if you’d like to change the class of an object at runtime, for example to a proxy class, with object_setClass, ARC won’t make a peep. However, it can get confused and release the object when its class changes, leading to overrelease and EXC_BAD_ACCESS. Worst of all, since ARC is completely out of your control, there’s nothing you can do about it!
Weirdly, for something that is meant to operate 100% at compile time, this behavior changes depending on what platform you run on. It happens in some instances on the simulator for older iOS versions, in different instances on the iOS 6 simulator, and (mercifully) never on a device.
ARC doesn’t really improve performance
This isn’t really a problem with ARC, but I’ve seen a number of blog posts about how great ARC is because it improves performance. These blog posts are usually filled with breathless tales of tail call optimization and assembly language listings. Now, there’s nothing wrong with Objective C from a performance perspective, but if you want to get down to questions of how many processor cycles your code takes then Objective C is pretty slow. Removing a few processor cycles from your iOS app is not going to have a noticeable effect.
How to write Well-Formed User Stories
Sunday, October 7, 2012
Writing Well-Formed User Stories
Convention Over Configuration is one of core principles of the Rails approach to software development, and delivers enormous value.
Convention Over Configuration – means that Rails makes assumptions about what you want to do and how you’re going to do it, rather than requiring you to specify every little thing…
Oddly, we tend not to apply the same perspective to project planning: on almost every project, the team re-invents the wheel of “how should we write and format our stories?”. I’ve worked closely with the Product team on about a dozen projects in the past few years, and rigorous story-writing is one of the most common areas for low-cost, high-gain improvement. I encourage every team to adopt (or at least consider) these techniques.
- Write every story in Gherkin. I don’t care whether or not you use cucumber: use Gherkin. Which is to say, every story should be in “Given / When / Then” form. This is the cheapest and easiest way to apply Convention Over Configuration to your user stories, and can have a HUGE benefit for your team.
Scenario: User adds item to cart Given I'm a logged-in User When I go to the Item page And I click "Add item to cart" Then the quantity of items in my cart should go up And my subtotal should increment And the warehouse inventory should decrement - Every feature story should include an “As a / I want to / Because…” block, which illustrates the motivation behind a story. Compelling the product team to specify the motivation behind a story help illuminate what exactly the requirement is, as well as providing guidance to the developers. Some people prefer “So That…” instead of “Because“, but in most cases “Because” helps drive out motivation—the Final Cause—whereas “So that” may only drive out the Effective Cause, which is less useful for understanding the story. (Thanks to Sam Coward for this insight.)
Feature: Shopping Cart As a Shopper I want to put items in my shopping cart Because I want to manage items before I check out - Every story title should include the word “should”. NEVER use the word “can”, which camouflages desired behavior. E.g. It’s unclear whether the story “User can delete comment” is a feature or a bug. “User should be able to delete comment” or “User should not be able to delete comment” are much clearer: the former is a feature, the latter a bug. Don’t make me guess.
When a story feels a little fishy, check that these bases are covered. If any are missing, fix then before you do anything else. The answer will often be driven out in the process of working the story into Well Formed shape.
Benefits
Well Formed stories truly drive out the feature from the user’s perspective; this catches 80% of weird edge cases while the whole team is together, in context, and in planning mode, instead of having to interrupt-drive the PM. Well Formed stories make it impossible to camouflage large stories as small stories by elision. Because the story has to be written out step-by-step, all the complexity might otherwise be hidden is forced out into the open. And when you find yourself with conditionals or switches? That’s a new scenario! Now all stories are forced into roughly the same size. Another side-effect is that once one story ~= one scenario, the amount of work to be done can be roughly gauged spatially, by looking at how much of your wall is covered by index cards. For bonus points, use the story title as your git commit, e.g. the story “User should be able to recommend a product” becomes the git commit “User is able to recommend a product”, and your git log tells the narrative of your project.
How did this start?
Once apon a time, J (the anchor) made N (a very bright, technical Product Manager) write stories in Gherkin. Most stories weren’t 100% ready to be pasted into cucumber, but it usually didn’t take too much work to get them there. The team would discuss in IPM, and then devs could copy-and-paste stories right into Cucumber. This doesn’t work for every PM, but even in the worst case, teams with less than tech-savvy PMs see real benefits from writing their stories at the right level of granularity. Once I was exposed to a team where we wrote Gherkin all the time, anything else felt like broken process.
UPDATE: To be clear, the opinions in this article are my own, and do not reflect anything close to consensus or standard practice on the part of Pivotal. Some Pivots will agree with this position, while many others will not.
UPDATE 3/17: Added a brief introduction elaborating on how Well-Formed Stories help bring principles of Convention Over Configuration to story-writing.
Debugging Travis builds
Sunday, October 7, 2012
(Due credit to Trung Lê‘s article on which all this is based)
We recently moved our project’s CI from a TeamCity server onto Travis CI’s new private CI-as-a-service program. We compared several other hosted CI services and found Travis to be the easiest to use, and with the help of Trung’s article also the easiest to debug.
Travis provides all their build worker images as Vagrant boxes available for download. I’d never worked with Vagrant before, but getting it set up is pretty simple. Follow the guide on Vagrant’s page and download the latest version of Vagrant. I’ve found that the boxes that Travis provide right now don’t work with VirtualBox 4.2, and so I’d recommend installing VirtualBox with 4.1 from their ‘older builds’ page.
Once you’re done installing both, you’ll want to install the worker box. The Ruby box is installable via
vagrant box add travis-ruby http://files.travis-ci.org/boxes/provisioned/travis-ruby.box
This will download and install the VM, which will probably take a few minutes. After that’s complete, initialize the box with
vagrant init travis-ruby
which will create a Vagrantfile for you where you can configure various settings for how Vagrant hosts the VM on your machine. I’ve found it necessary to add
config.ssh.username = "travis"
to get SSH to work properly. After that,
vagrant up
vagrant ssh
will connect you to the box, and verify that things are working properly. If you are prompted for a password upon sshing, it should be travis.
Now you’ll want to either scp your Github SSH key or create a new one and clone your project down to the box. After you have your project, all that remains is to get Travis running.
I haven’t yet figured out how Travis starts from .travis.yml files, so for now we have just created a shell script in which we specify all our Travis setup tasks, put it as the ‘script’ key in our .travis.yml, and then just run it directly on our local box. You can find more info about doing this on the Travis docs page. Aside from working around the .travis.yml, we haven’t seen any other gotchas in simulating the Travis worker process.
Once you’ve successfully gotten your build to run inside the Vagrant box, you can follow the instructions on Vagrant’s site to repackage your customized Travis worker and share it with the rest of your team.
Automated Deployment Messages
Friday, October 5, 2012
Make deployment visible with Capistrano, Autotagger, Git and Sendgrid
There comes a time in every project when the deployment process comes of age, and that development arrives with its own set of Capistrano recipes and Rake tasks. The project I’m on hit that point recently, and one of the neat outcomes of its nascent puberty was a simple Capistrano recipe to send a git changelog to our project mailing list.
Here’s what this looks like:
$ cap staging deploy
... stuff happens here ...
* executing `sendgrid:notify'
Changelog:
04fc6dd adding capistrano deployment messages
To use this in your Rails project, the first thing you need is a sendgrid account. If you’re budget-minded, you can always use the credentials your Heroku app is using.
$ heroku create
Creating heroku-wackiness-90210... done, stack is cedar
http://heroku-wackiness-90210.herokuapp.com/ | git@heroku.com:heroku-wackiness-90210.git
Git remote heroku added
$ heroku addons:add sendgrid:starter
Adding sendgrid:starter on heroku-wackiness-90210... done, v2 (free)
Use `heroku addons:docs sendgrid:starter` to view documentation.
$ heroku config -s
SENDGRID_PASSWORD=s3kr17
SENDGRID_USERNAME=yodawg@heroku.com
This process uses Capistrano and Autotagger. For information on setting up Capistrano, their wiki is an excellent starting point. For Autotagger setup with Capistrano, Jeff Dean’s auto_tagger repository is the canonical source of information.
After you’re up and running with Capistrano and Autotagger, you need to add the following file under lib/recipes/sendgrid_notifier.rb:
require 'mail'
set :sendgrid_user, "whatever"
set :sendgrid_password, "secret"
set :sendgrid_domain, "pivotallabs.com"
set :sender, "Now Hiring <jobs@pivotallabs.com>"
set :recipient, "Steve Squivot <you@square.com>"
namespace :sendgrid do
task :notify do
sendgrid = {
:address => "smtp.sendgrid.net",
:port => 587,
:domain => sendgrid_domain,
:user_name => sendgrid_user,
:password => sendgrid_password,
:authentication => 'plain',
:enable_starttls_auto => true
}
auto_tagger = AutoTagger::CapistranoHelper.new(
:stage => rails_env,
:stages => auto_tagger_stages).auto_tagger
previous_sha = auto_tagger.refs_for_stage(stage).last.sha
current_sha = auto_tagger.repo.latest_commit_sha
mail = Mail.new(from: recipient, to: sender)
mail.delivery_method :smtp, sendgrid
mail.subject = "[#{stage}] New Deployment!"
mail.body = `git log --oneline #{previous_sha}..#{current_sha}`
mail.deliver!
end
end
Then, let’s add a line to include this recipe in our Capfile:
require File.expand_path("../lib/recipes/sendgrid_notifier.rb", __FILE__)
Finally, let’s try it out:
$ cap ci sendgrid:notify
* executing `sendgrid:notify'
Changelog:
04fc6dd add a recruiter message to send off to Square
Alright! The email’s on its way. If you need to call this in your custom deployment step, it’s as easy as sticking sendgrid.notify into your Capfile.
Happy deploying!
Field Trip Chronicles: Pivotal NYC Hardware Hacking Club
Thursday, October 4, 2012
Pivotal NYC’s resident solder-happy Arduino-slingers (Pivotal Labs Labs, if you will) took a field trip today to Hack Manhattan, a hacker space, garden, and science laboratory on 14th street near our office. Hack Manhattan has, among other useful tools, a CNC mill, lathe, plotter, and several 3D printers in various stages of assembly. Members and friendly visitors are free to use the tools provided they know how, and may attend classes on various topics.
A joint venture between followers of the collective hacker ethos espoused by Brooklyn space NYC.Resistor, and a team of vigilante biologists, Hack Manhattan also boasts a rooftop garden complete with a solar-powered hydrolysis device and a chest of lively bees. Click through to a brief photo tour of our trip.
Testing accessibility with RSpec and Capybara
Tuesday, October 2, 2012
Edit 04/07/13: See the followup article for an alternative to using skip navigation links.
An exploration in automated accessibility testing
Today Grant Hutchins and I took on several stories to enhance the accessibility of a site. One of them was to add a skip-navigation link to the application.
To understand why skip-nav links are important, visit Jim Thatcher’s explanation.
Our immediate inclination was to write a request spec with Capybara to drive out the solution. We came up with the following.
The Test
require "spec_helper"
feature "Keyboard Navigation" do
scenario "hidden skip navigation link shows when focused and jumps to content", js: true do
login_as(users(:user))
visit root_path
skip_link = page.find("#skip-navigation a")
skip_link.should have_content "Skip navigation"
skip_link.native.location.y.should be < 0
body_element = page.find("body")
body_element.native.send_keys(:tab)
skip_link.native.location.y.should == 0
skip_link.native.send_keys(:return)
skip_link.native.location.y.should be < 0
current_url.should match(/#content$/)
end
end
The Markup
%body
#skip-navigation
%p= link_to "Skip navigation", "#content", tabindex: 0
...
#content
...
The Styles
body {
#skip-navigation {
a, a:hover, a:visited {
position:absolute;
left:0px;
top:-500px;
overflow:hidden;
}
a:active, a:focus {
position:absolute;
left:0;
top:0;
}
}
}
What’s happening here?
We are asserting that the “hidden skip navigation link shows when focused and jumps to content” when clicked.
The most important aspect of what we did was emulating keyboard navigation. We’re using js: true so we have access to Selenium’s native methods and thus the send_keys method. This allows us to send keypress messages to the browser.
Since we’ve used positioning to hide the element, we also have an assertion around that property.
Problems in CI!
The application behaved as expected and the tests passed locally.
When we ran the tests in CI however, the tests failed. The reason is that _the browser must retain foremost focus in the OS in order for the :focus css pseudo-selector to fire on the skip-nav element. Without the :focus style rules applied, the skip-nav element remained invisible, and the tests failed.
We tried a workaround using within_window and forcing browser focus, but couldn’t get it to work. We’ve got a few more tricks up our sleeve that we’re going to try, and will report back here.
Edit:
We added within_window to the test to force focus on the browser. This makes the test less brittle locally (because it won’t fail if you click out of the window).
window = page.driver.browser.window_handles.last
page.within_window(window) do
skip_link.native.location.y.should == 0
end
Uncle Bob @ Agile Testing and BDD
Monday, October 1, 2012
Today I’m in the Ace Hotel at a one day conference titled “Agile Testing and BDD Exchange” #bddxny, and I’ll be taking and sharing notes throughout the day.
Skills Matter is a UK based technical education company who are starting out in New York. This is their second conference. I’ve been to a few one day conferences, training days and evening meetups at their London base. I’m giving them bonus points for bring PG Tips.
Please forgive the inevitable spelling mistakes and typos, but I hope there’s some interest in this.
Uncle Bob starts with some 2001 Space Oddesy music to the video of a code file scanning down. It takes at least 3 minutes. He then starts, “I am your new CTO”, and continues to give his talk as if we, the audience, are employees at the same company. He outlines not how he wants us to work, but what he expects the outcomes of that work are.
Here follows the notes from his talk.
“I am your new CTO”
- “I will not tell you what to do, but I’ll tell you what to expect”
- “we will not ship shit”
- “there are times when you don’t ship all the features”
- “we will always be ready”
- “you will be ready all the time”
- “I want to say ‘ship it now’ and it goes”
- “I expect stable productivity”
- “I don’t expect you to start fast and slow down as the application gets bigger/older”
- “I expect inexpensive adaptability”
- “requirements will change, the platform may change – you will adapt and be able to adapt”
- “you will move quickly and inexpensively”
- “if you’re doing Rails/Spring (insert other framework, library or dependency here), I want to be able to rip that out – adapt inexpensively”
- “Continuous Improvement”
- “I don’t expect any one part of the system needs to be refactored”
- “keep the system as clean as it can be”
- “always be improving continuously”
- “if you note it starts getting messy, take steps”
- “professionals don’t let things get out of control”
- “Fearless competence”
- “I want you do say: ‘I can change that if I feel like it’”
- “Should not be afraid of the code”
- “No freezing of the code but constantly able to change the code”
- “Extreme quality”
- “you will focus on the highest quality system you can produce”
- “tradeoff in features, not quality”
- “I don’t expect bugs”
- “We will not dump on QA”
- “I expect QA will find nothing”
- “QA ought to wonder why they exist”
- “I expect QA to move to the front of the process, setting the tests and requirements to define how the system should operate”
- “Automation”
- “tests, builds and anything that be automated, will be automated”
- “manually testing is immoral – humans should only do things machines cannot”
- “Nothing fragile”
- “I expect things to be robust”
- “we know how to make non-fragile software”
- “there is no part of the system that deserves bad design; production code, tests, deployment – everything”
- “We cover for each other”
- “Sportsman and soldiers can do someone’s job”
- “Someone knows how to take over if they get sick, go on vacation or leave”
- “I don’t care what you do, you can pair, you can do something else but you will cover for each other”
- “Honest estimates”
- “Accurate and precise to the best of your ability”
- I do not expect a date
- I expect a range of dates, something you’d be comfortable with
- “Done within a few weeks of December 15th”
- I want to know what the odds are of hitting a date
- I want those estimates to be constantly adjusted
- Tell those managers who demand a date, I expect you do say “No”
- A hallmark of a professional is the ability to say no
- You must defend the honesty of your estimate and the quality of your system
- I expect you to never use the words “Yes, I’ll try” after you say no.
- There is nothing you do will differently if you say the word “try”
- It does not made the impossible, possible.
- This means you’ve been holding back
- You say try to get rid of people, this is a lie
- Continuous Aggressive Learning
- Industry is changing by the week
- Systems, platforms and languages comes in waves
- Your job is to surf those waves
- Do not ride the wave to the bottom
- You expect your doctor or lawyers to be reading about new stuff, I expect ‘us’ to be that kind of people
- Mentoring
- I expect us to teach
- schools and universities aren’t teaching how to work in our profession
- Look at apprentice models
- (Uncle Bob always returns to being technical)
- “we will not ship shit”
Questions
“Setting expectations”
- There is nothing you can do in order to go faster
- work more hours? – not reliable
- overtime becomes the norm
- things become a mess and slow things down
- “Time pressure creates an environment that generates mess”
- We can reduce features
- We can add staff
- mythical man month
- have to be careful, can be done if you have enough time
- new people will be exposed to old code and will continue to make mess
- There is nothing you can do in order to go faster
“How do we deal with the people who say yes”
- You can’t, over time there will be a separation
- Medical profession separated about 200 years ago between professionals
- Let’s go the way of medicals rather than law
- get signatures of others professionals
- let’s get a residency, internship
- no exam, degree or government legislation (like law)
- “I don’t need to tell people what to do (TDD, agile, pairing), but what to expect”
- business people have implied expectations and are disappointed when things don’t work out
- Uncle Bob became programming in the 60s, there were very programmers
- Deadlines are not an excuse
- professionals learn to work with deadlines and still produce good product
- Jack Dansen, History of Computing article (?)
- iPhone today would be the economic output for the whole world 50 years ago
- “my laptop is 22 orders of magnitude the power of the machines I started coding on”
- while powers have increased, how have we improved as programmers? Certainly not 22 orders
- we had OO, functional, structural programming in the late 60s
- we haven’t had any programmer based innovations since the 60s
- I can transport a programmer from the 60s and they could get up to scratch in 24 hours
- What we do has changed very little
- isn’t it time we started doing it well
Where do we begin if we have lots of bad code
- avoid the temptation to start over
- a redesign is probably
- no guarantee that you will get anything from it
- “clean the mess one little bit at a time”
- “leave the camp ground cleaner than you found it”
- “add unit tests where you can, when you need them”
- “gradually refactor and redesign over time”
- “it will be painful”