Explore Blog posts about everything we are up to, Tech Talk videos covering a huge range of timely topics and Event listings to keep you current on happenings at the Labs.
I had the pleasure of appearing on the Ruby Rogues podcast this week talking about how Pivotal Labs does Extreme Programing (XP). Accompanying me was ex-pivot Josh Susser, James Edward Gray, Avdi Grimm, and Charles Max Wood. I had more fun recording it than I would have imagined and the pivots here have enjoyed listening to it, I hope you will too!
Apple OS X’s PlistBuddy is a tool for modifying p-list (i.e. Property List, .plist) files; however, it has difficulty populating null keys (though little difficulty creating them). This blog post describes a method of using PlistBuddy to populate null keys by first creating & populating a placeholder key and then using the copy directive to populate the null key.
Creating the null key is easy; use a double-colon (“::”):
/usr/libexec/PlistBuddy -c "add :: dict" /tmp/junk.plist; cat /tmp/junk.plist
File Doesn't Exist, Will Create: /tmp/junk.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key></key>
<dict/>
</dict>
</plist>
Populate the null key by first creating a placeholder key, populating that, and then copying it to the null key:
rm /tmp/junk.plist
/usr/libexec/PlistBuddy -c "add :placeholder dict" /tmp/junk.plist
/usr/libexec/PlistBuddy -c "add :placeholder:nullKeySubKey string 'hello'" /tmp/junk.plist
/usr/libexec/PlistBuddy -c "copy :placeholder ::" /tmp/junk.plist
/usr/libexec/PlistBuddy -c "delete :placeholder" /tmp/junk.plist
cat /tmp/junk.plist
....
<dict>
<key></key>
<dict>
<key>nullKeySubKey</key>
<string>hello</string>
</dict>
</dict>
...In general, PlistBuddy will concatenate a double-colon (“::”) into one (“:”). So the key ::nullKeySubKey is treated as :nullKeySubKey, i.e.
/usr/libexec/PlistBuddy -c "add /usr/libexec/PlistBuddy -c "add ::nullKeySubKey string 'hello'" /tmp/junk.plist will not create a null key.
Attempts to trick PlistBuddy with an empty string (e.g. “”) won’t work, either. /usr/libexec/PlistBuddy -c "add :'':nullKeySubKey string 'hello'" /tmp/junk.plist will not create a null key.
The ~/Library/Preferences/com.apple.desktop.plist is a typical p-list file that contains a null key. First, we’ll convert the p-list from binary to XML using plutil, and then we’ll examine the XML to confirm that there is a null key:
plutil -convert xml1 ~/Library/Preferences/com.apple.desktop.plist
less ~/Library/Preferences/com.apple.desktop.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Background</key>
<dict>
<key>spaces</key>
<dict>
<key></key>
...
There was a question about uploading to S3 on an iOS project backed by non-rails ruby.
Best practice (Rails or not) seems to be using the AWS gem to generate URLS, send those to the phone, and do all the manipulation and uploading from the device.
Ping pong tournament with other startups @ Pivotal Labs. We have 2 teams made up of: Palermo and Phan, and Michael and Danny. Come cheer us on!
The Bulletproof font-face syntax effectively gives you broad browser support for your fancy (or austere) web fonts. However, using the Bulletproof font-face syntax with SASS carries a couple of gotchas. Reference this article when you are starting a new project and setting up the fonts.
//
// mixin for bullet proof font declaration syntax
//
@mixin declare-font-face($font-family, $font-filename, $font-weight : normal, $font-style :normal, $font-stretch : normal) {
@font-face {
font-family: '#{$font-family}';
src: url(font-path('#{$font-filename}.eot'));
src: url(font-path('#{$font-filename}.eot?#iefix')) format('embedded-opentype'),
url(font-path('#{$font-filename}.woff')) format('woff'),
url(font-path('#{$font-filename}.ttf')) format('truetype'),
url(font-path('#{$font-filename}.svg##{$font-family}')) format('svg');
font-weight: $font-weight;
font-style: $font-style;
font-stretch: $font-stretch;
}
}
@include declare-font-face('Gill Sans', 'Gill-Sans-MT-Pro-Light', 200);
@include declare-font-face('Gill Sans', 'Gill-Sans-MT-Pro-Italic', 400, italic);
It uses SCSS syntax, because SCSS can do multi-line and SASS can’t. It’s fine if you use SASS syntax, just name this file .scss and @import it just the same.
The mixin has defaults for last three parameters (denoted by the colons in the method signature). This way, you don’t have to go out of your way to specify normal each time.
It uses the font-path inside of the url(…). That is because I have seen lots of bugs with complex file names for the font files, and the quotes guarantee the file will be found. Either rename all your fonts with no dashes or spaces or other weird characters, or use this trick.
This isn’t really a trick, but it is more convienent to put all font variants inside the same font-family. It just makes it easier to have everything as “Gill Sans” and change the font-weight: italic, rather than having “Gill Sans” and “Gill Sans Italic” and however other many variants you have.
The original trick: Originally developed by Paul Irish (Bulletproof font-face syntax) and more recently by Fontspring (The New Bulletproof syntax), the syntax gives you broad browser support for your web fonts. IE8 loads the first .eot file and is done, but it “tricks” IE9 by relying on a bug in the parser. It dies on the question mark in the second line, and doesn’t continue on to the other file formats (which don’t work in IE9, and would cause it to not display). The other formats are handled by Safari, Chrome and Firefox, and carry no such trickery.
And be sure to follow me on twitter for new fancy posts!
When you get into the rhythm of pushing new features through the product development lifecycle, it can be addictive. Theoretically every new push brings more customer value, so why not keep going? Restated, when should you stop writing code and put your efforts elsewhere?
If you assume that you’ll never truly know (except in a vacuum), I think there are two main factors to consider: First, every feature you implement requires justification to build and effort to maintain. Second, if you are following lean principles it’s important to create valid experiments by measuring the viability of your current product, not one that is constantly shifting.
Let’s break that down.
Internalizing the lifetime cost of everything you build is critical to your [and your customers’] sanity. A friend of mine recently endeavored to simplify her life by trying the 100 thing challenge. Take a look at the three driving principles and you’ll see the parallels to product development:
Reduce by reducing the number of our possessions for an extended period of time, we prove to ourselves that consumerism does not define us
Refuse by refusing to go along with in the misleading lifestyle of consumerism, we form new priorities in line with personal virtue and what is best for the world around us
Rejigger by rejiggering our lives through simplicity, we nurture better relationships with family and community and nature
If you ignore the preachiness, this can serve as a template for your product. Don’t add new features just for the sake of it. The best apps do something specific, and do it very well. Refuse to add something that serves as a barrier between your customers and derived value just because it’s an emerging trend. Lastly, by focusing on the relationship your customer has with your product, you’ll shift focus from business-centric to customer-centric interaction goals.
The purpose of shipping early and often is to err on the side of learning too much. If you’re launching a new product, chances are you’re only going after early adopters. These people LOVE using unfinished products, since the chance of finding something that exactly addresses their wants/needs is high. These initial customers probably try 10 new services a week. Instead of chasing their wish list, you should figure out their common thread and target more of their peers. Reach out to them and figure out which part of your value proposition they care about — this messaging should be clear and consistent throughout your marketing and product (i.e., product/market fit). If you can tune this acquisition engine, you should see clear improvement in your cohorts without additional feature work.
The caveat emptor to the above advice is that you can never have too many tools at your disposal. While your product may be “good enough,” the instrumentation you use to measure your product are probably not. Devote some dev cycles to email tracking, behavioral metrics and personalized drip campaigns. Some might argue these are core product enhancements, even if they’re invisible (when done well). If they are helping you attract and engage new customers, I think it’s a moot point — just make sure that you’re not spending your time building the perfect product at the expense of one that’s good enough.
On the Tracker team, I love hearing great stories about our users and the things they are able to accomplish using agile methodologies and collaboration. I’m also a woman in tech and have a keen interest in supporting other women as well as other diverse groups who work persistently to reach their goals in tech. One story I’ve been following lately is the story of Tracker user Martha Chelimo Njeri Chumo (“Njeri”) and her teammates.
After raising funds, Njeri was set to attend this summer’s Hacker school in New York. Unfortunately she was denied a visa by the U.S. consulate in Kenya, curtailing her original plans.
It’s very easy when facing something beyond your own control to give up or to become discouraged. Njeri, however, was able to completely turn the tables on this setback by looking towards her own community and reaching for collaboration. Today, Njeri and her teammates are making plans to set up Nairobi’s own hacker school: Nairobi Dev Camp.
While I’ve given the Nairobi Dev School some support on my own, I decided to take a lesson from them and rope in my own team to support her cause.
Unsurprisingly, the rest of the Tracker team was also inspired by Njeri’s determination and wanted to pass her story along. While she still has a lot of work ahead of her to make this camp a reality, she also recognizes that she can’t do it alone, which is why she launched a campaign for Nairobi Dev School on Indiegogo.
We hope that you will be as moved as we are and help the Nairobi Dev School anyway you see fit. We wish Njeri and her teammates success.
Join the Pivotal Labs product team for the daytime panel Q&A: Design Meets Code and Studio Tour to take a peek at how Lean UX design fits with agile development.
Design Meets Code - Wednesday, June 26 from 12:30-1:30pm
Let’s talk about designs coming to life in code. Designers at Pivotal Labs work side-by-side with developers and clients every day. Join the Pivotal Labs’ product team for a Q&A panel on how to ship software and fit a full-spectrum design process into an agile environment. You’ll leave our session with tips, tricks and questions you can take back to your team tomorrow. Whether you create static mocks or fully functional prototypes, you’ll have a fresh perspective on how to bring your products to life. Come at 12:30pm for snacks and drinks; the panel discussion begins at 12:45pm.
Tweet your questions before or during the event with the hashtag #pivotallabsdesign
RSVP http://www.eventbrite.com/event/6763109643
Pivotal Labs Studio Tour - Wednesday, June 26 from 7:00-8:00pm
What does an agile design team look like? Meet designers, developers and product managers who sit next to each other and their clients every day and build products together. Pivotal Labs offers agile design and development methodologies in our practice. Don’t go chasing waterfalls, swing by our office and see how we do. Tweet to us with #pivotallabsdesign
Will Read & ex-pivot Josh Susser talk with folks like Avdi Grimm about XP and what it looks like at Pivotal Labs. ~1hr.
http://rubyrogues.com/109-rr-extreme-programming-with-will-read/
Building a bookmarklet provides an interesting challenge. It involves interaction a website your application does not control where that site could be anything with any number of dependencies on CSS or Javascript libraries. The first choice to make is trying to work with that website and probably setting an !important on every CSS selector used and hope that there’s no namespace or versioning clashes with any Javascript included; or use an iframe.
Iframes seem to have fallen out of favour in recent times but the sandboxed nature of the content inside an iframe mean the worries of CSS and Javascript clashes are gone. However this is replaced with a communications overhead of communicating between the host document and the iframe. I wanted to touch on some of the things our team did on a recent project to try and make this fairly seamless.
A bookmarklet is a small piece of Javascript that a user can drag onto the bookmark bar and upon pressing the link the Javascript will run. Because there is no guarentee what site will be loaded and if that will have any number of Javascript libraries included it’s best to use plain old Javascript to create an iframe element and append it to the body of the document. Our bookmarklet also needed to have some javascript if for nothing else but to be able to dismiss and remove the newly created elements. This can be done through the creation of a script element and appending to the body just like the iframe itself.
Once the iframe and script tags are appended they are treated the same as any other element. The content is loaded and the script is executed. The next step is getting the window to talk to the iframe. As a convienence the domain with protocol and port of the iframe is stored in a variable for later use.
element = document.createElement('iframe');
element.id = 'example_iframe';
element.src = 'example.com?referrer=' + window.location;
document.body.appendChild(element);
script = document.createElement('script');
script.src = 'example.com/bookmark.js';
document.body.appendChild(script);
The location is also sent to the remote server to load the iframe so that it can also store that location for passing messages to the host.
At this point this Javascript could also append a script tag to a version of libraries that may be required for it’s own application to run. It could also test for the existance of that library before hand so it doesn’t bring down an incompatible version. For example, bringing in jquery:
if ($ === undefined) {
var jq = document.createElement('script');
jq.src = "//ajax.googleapis.com/ajax/libs/jquery/1.10.1/jquery.min.js";
document.body.appendChild(jq);
}
The postmessage method is available to communicate bewtween the window and the iframe. The host window with a reference to the iframe can call postmessage with a string as a message and as a security measure the target location. We had stored this during the loading of the elements as described above.
iframe.postmessage('hello', 'www.example.com');
That message won’t get anywhere unless the iframe is listening for the message event on the other end.
// native Javascript
window.addEventListener('message', function(event){ … });
// jQuery
$(window).on('message', function(event){ … });
I’ve used the native Javascript above but really, once in the iframe itself the application has full control and could use JQuery or any other library at this point. Our application needs to listen to messages on both sides though so we needed the above to run in the host anyway.
Sending a message is all well and good but any non-trival application is going to have more than one function to run. We took influence from Remote Procedure Calls (RPC) to call functions within the host and remote sites. The message sent was stringified JSON with a very light ‘schema’ of the function to run and parameters to send to the function.
{
f: 'theFunction',
params: {
...
}
}
The recipient could then parse the string it knew to be JSON, extract the function to call and call it with any optional parameters also sent. This does create a binding between the host and the iframe but as the appliction controlled both sides we deemed it an acceptable risk. The function can be run from the window like so:
window['theFunction']();
// or from the event listener
var fn = JSON.parse(event.data).['f'];
window[fn]();
It was mentioned earlier that one of the goals of using an iframe was not to clobber any Javascript namespaces but we did end up including Javascript in the host and to avoid this we used an application namespace. However calling that as a property on the window would no longer work.
// doesn't work
window['my.app.function']();
// works
window['my']['app']['function']();
We looked to ElementalJS as an example of dealing with namespaced functions to parse the function.
window.addEventListener('message', function(){
var fn = window;
var data = JSON.parse(event.data);
var namespaced = data['f'].split('.');
for (var i in namespaced) {
fn = fn[namespaced[i]];
}
fn(data.params);
});
This is natually fairly crude, some defensive code could be added but this demonstrates the intent of the processing. Defense like making sure that function existed, or that ‘f’ existed in the data for the window could receive a message from another iframe.
The bookmarket inserts two elements, an iframe and a script. The iframe has the source example.com?referer=bar.com. bar.com is inserted as a variable in the iframe Javascript code. The iframe inserted has an id of example_iframe
The host Javascript listens to the message event
window.addEventListener('message', function(event){
var fn = window;
var data = JSON.parse(event.data);
var namespaced = (data['f'] || "").split('.');
for (var i in namespaced) {
fn = fn[namespaced[i]];
}
if (typoe(fn) === 'function') {
fn(data.params);
}
});
The host also sets up a namespace and a function to be called.
window.example = window.example || {};
example.hello = function(){ … }
When the iframe has loaded, it sends a ready message to the host
$(document).ready(function(){
var data = JSON.strinify({f: 'example.hello'});
parent.postMessage(data, referer); // referer set by server from the request param for the iframe
});
The host from the script loaded in the bookmarket has the example.hello function and it’s run. This in turn replies to the iframe.
var example.hello = function(){
var iframe = document.getElementById('example_iframe');
var data = JSON.stringify({f: 'example.world'})
iframe.contentWindow.postMessage(data, 'example.com');
};
The iframe has an event listener which is the same code as the host, and runs the function example.world
var example.world = function(){
// hello, world
};
This has shown some of the techniques for a ‘hello, world’ bookmark with two way communication between host and iframe that uses Javascript namespaces. This was enough to get our application off the ground as the two way communication acted as a solid base to build upon.
Programmers are constantly implementing time-related features, and accidentally including time-related bugs. I’m one of those programmers, and I would like to reduce the number of time-related bugs that I write. Some of them are small mistakes: time zone issues arise when running a test suite on a machine in a different time zone. These bugs are often fixed with consistent use of the time zone feature of a given time library. Others are more sinister, and lurk deep within the design of a system. They manifest in places where it’s tricky to get the system into a certain state because it is so heavily dependent on the current time. I propose that designing for the ability to set the current time from the outside of the system reduces the prevalence of timing related bugs, and has the happy accident of making code more reusable and testable.
Testing has many forms in the world of software. There are automated acceptance tests, unit tests, functional tests, enemy tests and so on. There are also tests carried out by humans. Sometimes referred to as ‘click testing’ or Quality Assurance, this kind of testing is an essential part of the process of delivering working software. At Pivotal Labs, it’s usually the Product Manager who has the final say about whether a feature is complete, or a bug is fixed. In order to evaluate whether a feature is ready, the PM exercises the area of the application in question, using the interface that a customer or stakeholder has been provided with.
When testing software with a mouse and keyboard becomes difficult, however, it often doesn’t get done. When it doesn’t get done, bugs introduced by diligent programmers, who test-drive their code, are missed and end up making their way into production.
After a few cycles of missing bugs like this, a team will look for ways to ease the pain of click testing their app. The programmers on the team might come up with clever, easy-to-implement solutions to this problem:
When these sorts of techniques dig in, new problems arise. In the former case, you can end up with code that works when the fake Time library is used, but not when the real one is used. In the latter, you are committed to a maintenance chore: when a timestamp field is added, it needs to change along with the rest, and when the code that changes those fields gets out of sync with how things really change over time, more bugs arise. It becomes difficult to tell which bugs are genuine and which are a consequence of artificially shifting time data.
It’s usually programmers who propose the above solutions. We often think in low-level terms like libraries and direct data manipulation. From a Product Manager’s perspective, however, what’s really needed is a design change. One could imagine the user story presented to the team like this:
As a Product Manager
I want to travel in time
So that I can test, for example, that an account gets billed each month
Time travel sounds like science fiction. How could a user of your system possibly travel in time? It turns out that there are low-level solutions to this. For example, Timecop, which many of us at Pivotal have contributed to. If depending on the passing of time is your addiction, then Timecop is enabling you. It lets you easily manipulate time, usually for automated testing purposes. For example:
Timecop.freeze(1.month.from_now) do
future_time = Time.now
sleep 10
future_time == Time.now # this is true
end
Here we’ve frozen time to pretend that it’s one month in the future. I can imagine some cases where this would be useful (not least in existing systems that are infected with code coupled to the current time), but in a lot of cases this is just wrong. Under what circumstance do you actually expect your code to be frozen in time? What are the consequences of testing code under these conditions?
Perhaps most importantly to the topic of this post, Timecop lets you forget about managing time at the unit level, and doesn’t encourage you to build time controls into your application.
I think there are better ways to get a grip on time that we should all consider before reaching for the magic wand. Let’s look at some real-world problems that can occur and then look at ways of building time control into an app’s design.
Let’s think about the properties of the current time:
So, the current time is a global, constantly mutating singleton. We know that the presence of global singletons is undesirable, because they are polluting. We also know that mutation ought to be contained, because mutable state makes our programs less predictable and harder to reason about. If a function deals with mutable state, then it might have different results each time it is called, even when it apparently has the same inputs.
Let’s look at a timing bug that can result from the fact that the current time has these undesirable properties:
policy = Policy.find(1)
if policy.current_state == :active
notify_customer("You are still insured!")
end
# more code goes here
if policy.current_state == :inactive
notify_customer("You are not insured. Hope you weren't planning on driving anywhere today.")
end
Imagine that the above code is within a web request. The request comes in at a certain time, and the customer wants to know whether their insurance policy is current. The code above is deep in the guts of a model somewhere, and gets called after the customer has been authenticated, their request has been authorized, and their account record has been pulled out of the database. Now it’s time to see what the state of the policy is, so we use a method someone wrote (current_state) that fetches the current time and returns a state based on whether the policy’s end date was before or after that time.
The customer sees this on their screen:
You are still insured! You are not insured. Hope you weren't planning on driving anywhere today.
The policy could potentially be active on one line and inactive the next. This kind of bug gets worse when one line makes an external call if the object were in one state, and the next makes a conflicting call if it were in another.
I recently ran into a real bug similar to this on my current project, which was caught when an acceptance test I was writing would fail on one run and pass on the next. The temporary workaround was to memoize the method that checked the current state (current_state above). Unfortunately, this introduced even more mutable state, because memoization requires changing the state of an instance variable. The next programmer might wonder why fetching the current state works the first time he asks, but stays the same with consecutive calls.
The current_state method is guilty here. But what of? It has a hidden input, which is the current time. It’s not explicit, and that’s where the confusion lies. It wouldn’t make sense to have a method called current_something and have it take the time as an argument, because the prefix “current_” implies that it’s supposed to know what the current state is.
The internal functions of a program shouldn’t know this stuff. In most web apps, a request is made at a certain point in time, but it’s not important that the request takes some time. With most scheduled jobs, the job is run at a certain time, but it’s not important that the job takes some time (or if it is, it’s stored as metadata).
A name less prone to attracting this kind of bug might be state_as_of(time). If we force ourselves to pass the time as a parameter to all of our low-level functions, then we can:
Scheduled jobs are often concerned with when they think they’re being run, but a PM doesn’t want to wait for a month to see if, for example, the billing system is working. It’s important to give control over when a job thinks it’s being run to the PM or other person evaluating whether a system works. This might mean dropping your out-of-the-box scheduling interface for the purposes of feature acceptance.
Resque-scheduler has become very popular amongst developers as it’s easy to install and provides a cron-like syntax for declaring when jobs are run. It also provides a GUI for triggering scheduled jobs immediately. Unfortunately, there’s no way to set parameters for the jobs, so the time can’t be set. If you choose to heed the advice in this post and parameterize time, you’ll need to provide your own interface for passing the current time into the system. This is a good idea anyway. See GOOS for a good treatment of externalizing event sources, which goes even further than the suggestions in this post in many ways.
Making time a parameter to your jobs can often make the jobs more reusable. For example, I might want to invalidate all sessions in a particular time range because there was a system fault at those times.
What happens if your external dependencies are dependent on the current time? Well, you’re going to have problems with that no matter how much control you build into your system. I would argue that the services should be wrapped, and the wrappers should allow the time to be passed in to fake responses if necessary. The acceptance (as in story acceptance in Pivotal Tracker) of your system doesn’t have to depend on the state of the other systems.
What if your customer needs to be shown what’s happening right now?
I think the trick here is to depend on the order of time, and not the passing of time. To move into the future, we should be reacting to events by adding data, not mutating it. A database can easily figure out what the latest order is, or what the latest billing cycle is. It doesn’t have to be a function of the current time.
If, however, it’s just too difficult to implement the functionality without checking the current time, it could be argued that the current time be a parameter passed in through the browser, available only in certain testing environments. I would resist this as far as possible, but think that it would be a preferable solution to mutating the state of the database.