We’re moving Pivotal Tracker to a new, faster database server with solid state drives tomorrow, and performing some other maintenance that requires downtime. To minimize disruption during work hours in as many parts of the world as possible, we’re planning this maintenance for tomorrow, Saturday, March 31, at 9am Pacific, and we anticipate it to take approximately one hour.
To get the latest updates, and real time status of this maintenance, please follow @pivotaltracker on Twitter.
On my current project, we needed to prove that an action cache was working as expected. Alas, the blogosphere had either out-of-date or unhelpful information. So, after many experiments, we came up with an RSpec test that does what we want. It seems ugly to me, and I hope there’s a better way. The names have been changed to protect the guilty. Any resemblances to actual classes and methods are purely coincidental.
We needed to confirm that a certain action was cached. This action is preview in the brands controller. Using the usual Rails url helpers, we construct some fixture data.
describe BrandsController do
describe "caching" do
let(:brand) { Factory.create(:brand) }
let(:preview_cache_path) {'views/test.host' + preview_brand_path(brand)}
end
end
Then we wrote our first test:
it "should action cache #preview" do
Rails.cache.clear
get :preview, :brand_id => brand.to_param
ActionController::Base.cache_store.exist?(preview_cache_path).should be_true
end
This won’t work at all, however; because, in the test environment, caching is turned off.
$ cat config/environments/test.rb
Activator::Application.configure do
...
config.action_controller.perform_caching = false
So, we need an around block to temporarily turn caching on:
around do |example|
caching, ActionController::Base.perform_caching = ActionController::Base.perform_caching, true
example.run
ActionController::Base.perform_caching = caching
end
That’s great, but the default cache store is the :null store, which, as its name implies, does nothing.
around do |example|
caching, ActionController::Base.perform_caching = ActionController::Base.perform_caching, true
store, ActionController::Base.cache_store = ActionController::Base.cache_store, :memory_store
example.run
ActionController::Base.cache_store = store
ActionController::Base.perform_caching = caching
end
Better. But our tests still won’t run because while ActionController uses the cache_store, Observers and Sweepers
use Rails.cache and that is only updated at boot time.
around do |example|
caching, ActionController::Base.perform_caching = ActionController::Base.perform_caching, true
store, ActionController::Base.cache_store = ActionController::Base.cache_store, :memory_store
silence_warnings { Object.const_set "RAILS_CACHE", ActionController::Base.cache_store }
example.run
silence_warnings { Object.const_set "RAILS_CACHE", store }
ActionController::Base.cache_store = store
ActionController::Base.perform_caching = caching
end
Did I mention that Rails.cache is an accessor for the global, constant, RAILS_CACHE. Ugh.
So, now, we can implement our method
class BrandsController < ApplicationController
caches_action :preview
def preview
end
end
But that is still not enough. caches_action
has an interesting performance enhancement; it doesn’t actually set up the action caching unless caching is enabled at class load time. Since we’re not turning caching on until test time, the caches_action method call in the controller class does nothing. We need to re-add it in our test spec.
it "should action cache #preview" do
Rails.cache.clear
BrandsController.caches_action :preview # must be recapitulated to get around load time weirdfullness
get :preview, :brand_id => brand.to_param
ActionController::Base.cache_store.exist?(preview_cache_path).should be_true
end
This is ugly; it doesn’t test very much (except the underlying caching module, and why bother testing the framework). At least it proves to ourselves that the action is cached and the cache key is what we expect.
Now that we’ve got caching under control, let’s check cache expiration (using a Sweeper).
it "should clear the cache on #update" do
ActionController::Base.cache_store.write(preview_cache_path, 'CACHED ACTION')
put :update, id: brand.to_param, brand: {one: 'attribute', after: 'another'}
ActionController::Base.cache_store.exist?(sign_up_cache_path).should be_false
end
First, I create a cached object, in this case, just the string ‘CACHED ACTION’ and then I invoke the action, and then, I hope, the cache will be expired.
It doesn’t really matter what happens in the #update method of the BrandsController as long as it updates a Brand object. A sweeper in Rails is a mix of Observer & controller filters, so all you need to do is “declare” it in the controller
class BrandsController < ApplicationController
caches_action :preview
cache_sweeper :brand_sweeper
def update
...
@brand.save
Awesome sauce! Now our tests are red and I’m ready to implement the sweeper
class BrandSweeper < ActionController::Caching::Sweeper
observe Brand # Observers will introspect on the class, but Sweepers don't
def after_update(brand)
expire_action :controller => "brand", :action => :preview, :brand_id => brand.to_param
end
...
And voilà! We have greenness.
So what have we learned from this? The Rails source is still your best friend when exploring a sticky problem. Caching is hard, and testing caching is even harder.
A project is moving from their initial layouts to Bootstrap. They have quite a few class name conflicts in their css, so in order to isolate the new Bootstrap styles to specific elements, they added the Bootstrap styles to their own (in SASS) under a .bootstrap namespace, and are nesting any new elements under divs with a class of .bootstrap, and now everything is pretty again.
Nesting in this way (rather than namespacing all of the preexisting css) ensures that no Bootstrap styling leaks into pre-existing elements, but means that occasionally a pre-Bootstrap style will leak into the new namespaced elements due to higher selector specificity.
When we moved to a new office, we faced a problem: how do we give printer access to everyone though we had segregated machines to different networks? And how do we make it transparent to the user?
The solution we found was to add a new VLAN (i.e. network segment) for the printers (and other common resources, e.g. license servers), use DNS Service Discovery (dns-sd), and add a handful of crafted records to our DNS server.
This blog post is directed at Operations staff at companies which have the following characteristics:
The short version: people couldn’t print from the WiFi network.
This is a synopsis of our network (note: the IP addresses and subnet masks are simplified for purposes of our discussion):
VLAN Name IP
1 PIVOT 10.0.1.0/24
2 SERVER 10.0.2.0/24
3 PAIRING_DMZ 10.0.3.0/24
4 VOIP 10.0.4.0/24
5 PIVOTAL_WIFI 10.0.5.0/24
6 PIVOTAL_GUEST 10.0.6.0/24
7 SECURITY 10.0.7.0/24
8 COMMON 10.0.8.0/24
Note that the last VLAN (“COMMON”) is the one where we have placed all the resources. We named it COMMON as in “Common Resources”.
We used a bonjour browser to discover which records we needed to add (note: you need to run the bonjour browser on a machine that is on the same network as the printers, otherwise the printers won’t show up). We navigated as follows: local. → _pdl-datastream._tcp. 1 We found our printers (goldfinger and blofeld, named after James Bond villains). Here are the relevant records for Goldfinger:
In our setup, we use tinydns to serve our DNS records. For those using BIND, the dns-sd.org website has an excellent how-to.
First, we need to create the basic dns-sd records. We only need to add these records once.
^b._dns-sd._udp.sf.pivotallabs.com.:sf.pivotallabs.com.:86400::
^lb._dns-sd._udp.sf.pivotallabs.com.:sf.pivotallabs.com.:86400::
Download this script to create the DNS records (courtesy Michael Sierchio). Save the script as “make_printer_dns”. Then use the bonjour browser to extract the printer information and save that information to a file with the same name as the printer (e.g. hp4020.sf.pivotallabs.com). In this example, we use a heredoc to save the information extracted from the HP 4700 Color Laserjet, and then we run the script to output the djbdns records:
$ cat > goldfinger.sf.pivotallabs.com. <<EOF
txtvers=1
qtotal=1
pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL
ty=HP Color LaserJet 4700
product=(HP Color LaserJet 4700)
priority=40
adminurl=http://goldfinger.sf.pivotallabs.com.
EOF
$ make_printer_dns goldfinger.sf.pivotallabs.com.
^_pdl-datastream._tcp.sf.pivotallabs.com.:goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.
:goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.:33:\000\000\000\000\043\214\012goldfinger\002sf\013pivotallabs\003com\000\000
:goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.:16:\011txtvers=1\010qtotal=1\112pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL\031ty=HP Color LaserJet 4700\040product=(HP Color LaserJet 4700)\013priority=40\056adminurl=http\072//goldfinger.sf.pivotallabs.com.Copy the above records into your djbdns source files and regenerate the database (i.e. tinydns-data). You have now created the records necessary to allow printer discovery across subnets.
[Editor's note: do not use this hard technique; use the easy way instead. This portion is meant for instruction rather than actual implementation]
We add a PTR record for Goldfinger printer. We’ll need to add a similar record every time we add a new printer:
^_pdl-datastream._tcp.sf.pivotallabs.com.:Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.
Then we need to create DNS SRV records for the Goldfinger printer. These can be crafted by hand, but I prefer to use Anders Brownworth’s tinydns record builder. We enter the following information for the SRV record builder:
Anders’s script gives us the following record, which we add to our tinydns records:
:Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.:33:\000\000\000\000\043\214\012goldfinger\002sf\013pivotallabs\003com\000:86400
But we’re not done: we still need to create the TXT record which has the important information we uncovered with our bonjour browser. A regular tinydns TXT record (one which begins with a “‘”) won’t do because we have several records. We need to use a generic record (a special TXT record). This one we’ll need to handcraft.
:Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.
:16:
txtvers=1
qtotal=1
pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL
ty=HP Color LaserJet 4700
product=(HP Color LaserJet 4700)
priority=40
adminurl=http://goldfinger.sf.pivotallabs.com.
ruby -e 'STDIN.read.split("\n").each { |t| printf("\\%03o%s",t.length,t.gsub(":","\\\\072")) }; puts'
:Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.:16:\011txtvers=1\010qtotal=1\112pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL\031ty=HP Color LaserJet 4700\040product=(HP Color LaserJet 4700)\013priority=40\056adminurl=http\072//goldfinger.sf.pivotallabs.com.
Once we’ve added the records and rebuilt our djbdns database, we test to make sure it really works:
nslookup -query=srv goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.
Server: 10.80.0.18
Address: 10.80.0.18#53
Non-authoritative answer:
goldfinger._pdl-datastream._tcp.sf.pivotallabs.com service = 0 0 9100 goldfinger.sf.pivotallabs.com.
nslookup -query=txt Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.
Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com text = "txtvers=1" "qtotal=1" "pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL" "ty=HP Color LaserJet 4700" "product=(HP Color LaserJet 4700)" "priority=40" "adminurl=http://goldfinger.sf.pivotallabs.com."Note the following:
After restarting our DNS server, our clients were able to add a printer easily through Mac OS X’s System Preferences → Printers → “+”.
Client machines need to have the domain (e.g. “sf.pivotallabs.com”) in their search-path for dns-sd to work. For ISC-dhcpd, the appropriate entry would be “option domain-name ‘sf.pivotallabs.com’;”
We noticed a propagation delay with older (Snow Leopard) Macs—they didn’t always see the printers right away. In these cases, we tried flushing the DNS cache (“dscacheutil -flushcache” or, in Lion, “sudo killall -HUP mDNSResponder”), but that seemed to have no effect. A few days later the printers became discoverable from those workstations.
Apple clients will use SNMP queries to determine some of the printers capabilities. We encourage you to all SNMP traffic to the network where the printers are located.
1 pdl-datastream is a Registered Port for printing. There are at least 2 other ports commonly used for printing: ipp (631) and printer (515). Our decision to use pdl-datastream over the other two was arbitrary. That being said, we have found that using the ipp port to be counter-productive and should be avoided. We have found that when we advertise ipp and the printer in question is Airprint-capable, then printing will not work (I suspect AirPrint does not work across subnets, but am not sure).
Special thanks to my co-authors Michael Sierchio and Matthew Kocher
tinydns, dns-sd, DNS Service Discovery
stub_model is not working correctly in rails 3.2, even after applying the rspec fix. The problem is that stubbed objects aren’t returning valid.
“Has anyone used cucumber for code coverage? We’re trying to find dead code.”
SimpleCov worked great for rspec, but when used with cucumber introduced a lot of timeouts.
its(:"items.count") { should == 5 }
git push heroku master and then forgotten to runheroku run rake db:migrate --app yellow-snow-3141 orheroku ps:restartEnter heroku_san, the 85% solution, which allows me to do this:
$ rake staging deployAnd this is what happens: Maintenance mode is turned on; the latest green build is deployed to my staging server; all pending migrations are run, the app restarted, and then maintenance mode disabled. All while I go get a fresh cup of joe.
The value for us is that we don’t repeat ourselves, that our app will deploy, and we are confident that extending the toolkit is easy with a clear API and good test coverage. That’s a big win when you have dozens of apps deploying all day long.
Installation is pretty simple; add gem 'heroku_san' to your Gemfile (you are using Bundler, aren’t you?). This gem is only used locally, so you can add it to your group :development, :test block. Then create and edit a configuration file; config/heroku.yml[1]. The README file has more details and there’s a wiki on github with handy tips on how to extend the basic functionality in common ways.
The gem has been around since 2010 and supported by a loyal core of maintainers. Version 2.1.1 brings changes large and small.
The biggest change since v1.3 is the creation of 2 new classes: Project and Stage, and a test suite to drive out the features. The task library is now almost entirely simple method calls, so there’s a lot of confidence that they will run correctly. Just to make sure, however, I’ve written an integration test, remote.feature which exercises the whole stack by creating an app on Heroku and run it through its paces. It’s worth a read.
There are some other new features worth highlighting:
heroku.yml file, heroku_san will deploy only the most recent revision with that tag to your stage. This is great for creating a multi-stage environment, and when combined with auto_tagger and a continuous integration server (CI), you get a lot of bang for a little bit. See the wiki for more complete instructions.Why write your own deploy scripts, again. Use heroku_san (or another gem like kumade)
[1]: I am working hard to remove the Rails dependencies.
I recently had to write a small capybara helper:
def without_page_refresh
page.execute_script("window._withoutPageRefresh = 'BAM'")
yield
page.evaluate_script("window._withoutPageRefresh").should == "BAM", "Page was navigated away"
end
Use case:
You are building a form that gracefully degrades
when javascript is not enabled. It goes something like this:
# uses rack-test so no javascript
scenario "User can lose money in an accessible way" do
visit "/your_account"
page.should have_content "You have $100"
click_button "Charge me"
page.should have_content "You have $50"
end
Next step is to make that form do the ajax thingy.
You copy test above and switch it to use javascript driver:
# uses selenium for javascript
scenario "User can lose money with style", :js => true do
visit "/your_account"
page.should have_content "You have $100"
click_button "Charge me"
page.should have_content "You have $50"
end
Newly written test is pretty good except that it passes without
writing a single line of javascript. So here’s when that helper
comes into play. Our javascript test becomes:
# uses selenium for javascript
scenario "User can lose money with style", :js => true do
visit "/your_account"
without_page_refresh do
page.should have_content "You have $100"
click_button "Charge me"
page.should have_content "You have $50"
end
end
Now javascript test fails with “Page was navigated away”
since nothing is preventing that form from submiting to another page.