Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Tools
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker
Robbie Clutton

API Versioning

Robbie Clutton
Friday, May 31, 2013

How to version an API has been a thoroughly discussed topic in the last several years regardless of protocol or approach, be that SOAP, REST or Hypermedia. Why contribute another post to the topic? My last post on avoiding breaking changes through better design led to conversations in the office and online about versioning.

@moonmaster9000 @robb1e @jaderubick you should version using a vendor string in the Accept header.

— David Celis (@davidcelis) May 16, 2013

I wanted to dig a little deeper into versioning and look at some of the different ways people are using versioning for their resources on the web.

Compatibility as version control

Mark Nottingham has written a lot of the subject of API versioning and it’s difficult to disagree with the high level arguments outlined in his API evolution post.

  • keep compatible changes out of names
  • avoid new major versions
  • makes changes backwards-compatible
  • think about forwards-compatibility

In another post Nottingham adds that we shouldn’t use a custom header for versioning and touches on versioning through the Accept header but there is a fundamental thread here: try as hard as possible to not introduce breaking changes so that versioning isn’t a big issue.

One of the issues to content with when considering versioning is often building APIs requires more thought up front that an agile developer might be used to. URIs, Data structure, meta-data and extensibility are important and would be best considered up front. Once those decisions have been made changes to the structure often result in breaking older versions of the API. Anyone at an early stage of building an API would do well to put some thought into the API design and setting some rules for future consistency.

Accept header

The Twitter comment above pointed me towards using the Accept header as part of content negotiation and I found a number of blog posts covering the subject as well as some companies using this approach.

RFC4288 section 3.2 outlines how a vendor, i.e. an application, can make use of customisable MIME types in the Accept header. Steve Klabnik shows how an application can make use of this to include a version number as part of the Accept header used for content negotiation.

Looking at some concrete examples through the well documented Github API shows how the Accept header can be used with their services:

Accept: application/vnd.github[.version].param[+json]

curl -v -H 'Accept:application/vnd.example.v1+json' localhost:3000

The vnd part is the vendor definition as outlined in RFC4288. Let’s take a look at a number of options to make use of this header.

Parsing the Accept header

Within an application, headers can be inspected from a HTTP request. Extracting the version number would require some regular expression matching, something along the lines of the following in an application_controller perhaps:

def api_version
request.headers["Accept"][/^application\/vnd\.github\.v(\d)/, 1].to_i
end

Once the application has this version number it can decide how to behave for the response.

Registering the MIME type

An application could alternatively register a MIME type for each and use the respond_to blocks to decide how to render a response.

To register a MIME type in config/initializers/mime_types.rb

Mime::Type.register "application/vnd.github.com.v1+json", :json_v1
Mime::Type.register "application/vnd.github.com.v2+json", :json_v2

MIME types also allows parameters but would require registration for each one also:

Mime::Type.register "application/vnd.github.com+json; version=1", :json_v1
Mime::Type.register "application/vnd.github.com+json; version=2", :json_v2

So a controller could look something like the following and deal with the request appropriately.

posts = Api::Posts.all

respond_to do |format|
    format.json_v1 { posts.v1.as_json }
    format.json_v1 { posts.v2.as_json }
end

Version via a request parameter

The thing with the version in an Accept header is the URI is difficult to share. If I wanted to share the URI with version information with a college I would have to send instructions on what curl arguments to send to the server to get the right response. It’s can be as frustrating as trying to share a holiday on a website that renders pages based on what’s in the session for the individual. I would want to share a URI that someone can paste into a web browser address bar and see an appropriate response. Let’s compare the two approaches:

curl -v -H 'Accept:application/vnd.example.v1+json' localhost:3000

vs

curl -v localhost:3000/?version=v1

or

curl -v localhost:3000/?version=20130603

Version number as a date

The Foursquare API allows clients to send a version as a date in the format yyyymmdd which conveniently is an always increasing number. When a client starts to use the API they can use that days date and the response will always be in that format. I like this concept as it removes the burden of having to know what endpoint to use or what version to send in a header. The client uses a known point in time and the response will always match if that is sent. If nothing is sent then the latest version of the resource is returned.

With this in mind, I wrote a very small gem to put this into practice to demonstrate how I thought this could be achieved which boils down to the following. Given a number (i.e. the version sent in the request) and a list of numbers (i.e. known versions with some change in an application), find the previous closest number in the list from the version number given. For example, if the date 20130101 is passed and the list contains two dates of 20130101 and 20130601, then 20130101 is returned.

def find_version_for version, list
    return list.last if version.nil?
    list.select { |i| i <= version }.last || list.first
end

Using the returned number, the server can decide how to render the response to the client as defined in my previous post.

Robustness principle

Jon Postel says "Be conservative in what you do, be liberal in what you accept from others", so perhaps we could allow our clients to do both?

Perhaps using the Accept header makes you a better denizen of the Internet, adhering to HATEAOS principles but I think using a version request parameter makes for a better Web experience. As a developer, I want to be able to put a URI into a browser and see a response rendered and for that I'd lean more towards the version parameter.

This approach can be generalised across other information sent from a client to a server. Clients like web browsers send information in every request, and these should honored as the defaults. The information sent includes what language, data and encoding formats the client would like. Using request parameters can offer overrides and support compatibility between different types of clients who want to use an API and cool bookmarkable URIs.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Luan Santos

Playing with Ember.js and Devise

Luan Santos
Sunday, May 26, 2013

I have been playing with Ember in and out since the beginning of 2012, you may have seen my two (obsolete?) libraries ember-facebook and ember-formbuilder. I wrote them out of real needs I had on projects back then.

I was away from ember since version 0.9.8 and ember-data I don’t even remember if it had a version number yet, and I started playing around with it again this last few weeks.

Right now there are not a lot decent resources of getting started with ember and unfortunately some of them are outdated. I am sure that this will get better and better now that the framework is stabilizing.

Anyway, I am working in a little project with a friend and we decided to use ember on it. First thing we implemented was authentication. This is a very simple problem with several solutions. We want to use devise. A very simple one is to rely on the devise’s engine and just make it a separate part of the app, after that you render the user’s session JSON on a metatag and parse it in ember. There is a very nice tutorial on how to do that by Alexander Zaytsev here.

Alternatively you can implement all the interface yourself and have a truly single page application, and it is not that hard. First thing you have to do is turn devise into an API by creating a few controllers. I am just going to show sign-up and login here.

Turning Devise into and API

For sign up let’s create our own users controller to handle registrations:

class UsersController < ApplicationController
  def create
    user = User.new(params[:user])

    if user.save
      render json: user, status: :created
    else
      respond_with user
    end
  end
end

It’s very simple, an API endpoint to create users using the devise’s user model and respond with the JSON for that user (you may or may not be using active_model_serializers here, but you should be).

Similarly we create a sessions controller to handle, well, sessions:

class SessionsController < ApplicationController
  def create
    user = User.find_for_database_authentication(email: params[:session][:email])

    if user && user.valid_password?(params[:session][:password])
    	sign_in user
    	render json: {
    	  session: { id: user.id, email: user.email }
    	}, status: :created
    else
      render json: {
        errors: {
          email: "invalid email or password"
        }
      }, status: :unprocessable_entity
    end
  end

  def destroy
    sign_out :user
    render json: {}, status: :accepted
  end
end

This one is longer, but it is also very simple: we find the user, we validate whether the password is correct and we then respond with either the basic user data and a successful status or an error message to be shown by ember with an errors status.
We are also adding the destroy action for loggout.

Don’t forget that you need to add theses to your routes.rb:

resources :users, only: [:create]
resources :sessions, only: [:create, :destroy]

Models

We are going to need two Ember models for this, one for our registration and another for the sessions, models in ember-data are extremely simple and there is not a whole lot to say about them.

App.User = DS.Model.extend({
  email: DS.attr('string'),
  password: DS.attr('string'),
  passwordConfirmation: DS.attr('string')
});

App.Session = DS.Model.extend({
  email: DS.attr('string'),
  password: DS.attr('string')
});

That is already enough for us to play around in the console and create users/sessions:

var user = App.User.createRecord({email: 'foo@bar.baz', password: 'password', password_confirmation: 'password'});
user.save();
var session = App.Session.createRecord({email: 'foo@bar.baz', password: 'password'});
session.save();

Routes

Let’s define our routes:

App.Router.map(function() {
  this.resource('users', function() {
    this.route('new');
  });

  this.resource('sessions', function() {
    this.route('new');
    this.route('destroy');
  });
});

The first two routes we have to add are simple and are just setting the content properties of their controllers. The destroy route is slightly more complicated:

App.UsersNewRoute = Ember.Route.extend({
  model: function() {
    return App.User.createRecord();
  },

  setupController: function(controller, model) {
    controller.set('content', model);
  }
});

App.SessionsNewRoute = Ember.Route.extend({
  model: function() {
    return App.Session.createRecord();
  },

  setupController: function(controller, model) {
    controller.set('content', model);
  }
});

App.SessionsDestroyRoute = Ember.Route.extend({
  enter: function() {
    var controller = this.controllerFor('currentUser');
    controller.set('content', undefined);

    App.Session.find('current').then(function(session) {
      session.deleteRecord();
      controller.store.commit();
    });

    this.transitionTo('index');
  }
});

This destroy route does not have a controller or view (at least not an explicit one), it basically unsets the content of the currentUser controller (check next section) then issue a delete request on the sessions controller of our API and redirect the user back to the root. Note that we are “finding” the session using the id of “current”, ember-data does not support singletons explicitly, so we have to work around that fact by doing that. You can read more about this here.

Current User

This part is based on the post I mentioned earlier by Alexander Zaytsev. It’s an object controller that will hold the existence (or not) of the current user session.

App.CurrentUserController = Ember.ObjectController.extend({
  isSignedIn: function() {
    return this.get('content') && this.get('content').get('isLoaded');
  }.property('content.isLoaded')
});

We’ll also need this initializer to populate it once the app is loaded.

Ember.Application.initializer({
  name: 'currentUser',

  initialize: function(container) {
    var store = container.lookup('store:main');
    var user = App.User.find('current');

    container.lookup('controller:currentUser').set('content', user);
    container.typeInjection('controller', 'currentUser', 'controller:currentUser');
  }
});

Here we’re asking or API for the current user with the same singleton strategy App.User.find('current'), the app is supposed to return either a successful message with the user JSON or an error. We now need a show endpoint in our users controller, as simple as respond_with current_user, don’t forget to add the show route in your resource on routes.rb.

Templates

We need two things here, we need a sign up form and a login form. Another thing we may want is navigation links. Let’s start with the forms. There is no point in logging-in if you don’t have an account right? So let us sign up!

<h1>Create an Account</h1>
<form class="form-horizontal user-form">
  <fieldset>
    <div class="control-group" {{bindAttr class="errors.email:error"}}>
      {{view Ember.TextField valueBinding='email' name='email' placeholder='Email'}}
      <span class="help-inline">
        {{errors.email}}
      </span>
    </div>
    <div class="control-group" {{bindAttr class="errors.password:error"}}>
      {{view Ember.TextField type="password" valueBinding='password' placeholder='Password'}}
      <span class="help-inline">
        {{errors.password}}
      </span>
    </div>
    <div class="control-group" {{bindAttr class="errors.passwordConfirmation:error"}}>
      {{view Ember.TextField type="password" valueBinding='passwordConfirmation' placeholder='Password Confirmation'}}
      <span class="help-inline">
        {{errors.passwordConfirmation}}
      </span>
    </div>
    <a href='#' {{action cancel}} class='btn'>Cancel</a>
    <button type="submit" {{action save}} class='btn btn-large btn-primary'>Sign Up</button>
  </fieldset>
</form>

 

Assuming you’re on rails and using the standard ember gems you should put this file in app/assets/javascripts/templates/users/new.hbs, ember will automatically pick it up if you follow it’s defaults.
I am using a bootstrap form structure here to show the inputs and error messages. Ember-data will automatically populate the errors property for you if you’re API is returning the right thing, which it is.

And our Login form:

<h1>Login</h1>
<form class="form-horizontal user-form">
  <fieldset>
    <div class="control-group" {{bindAttr class="errors.email:error"}}>
      {{view Ember.TextField valueBinding='email' name='email' placeholder='Email'}}
      <span class="help-inline">
        {{errors.email}}
      </span>
    </div>
    <div class="control-group" {{bindAttr class="errors.password:error"}}>
      {{view Ember.TextField type="password" valueBinding='password' placeholder='Password'}}
    </div>
    <a href='#' {{action cancel}} class='btn'>Cancel</a>
    <button type="submit" {{action save}} class='btn btn-large btn-primary'>Login</button>
  </fieldset>
</form>

This files goes into app/assets/javascripts/templates/sessions/new.hbs.
Very similar to the sign up form, in this case we’re only showing errors on the email field because we’re not specifying whether the email or password was wrong, sometimes you may want to do that, this is not our case here.

There you have it, a very simple sign up and login for that will just work. Right? No, we still have to define our actions, but we’re getting there. Note that we defined in both forms {{action save}} in our submit buttons, now we need our controllers to handle that.

Oh, and you may also want to add these navigation links somewhere:

{{#if currentUser.isSignedIn}}
  Logged in as {{currentUser.email}}
  {{#linkTo 'sessions.destroy'}}Logout{{/linkTo}}
{{else}}
  {{#linkTo 'users.new'}}Sign Up{{/linkTo}} |
  {{#linkTo 'sessions.new'}}Login{{/linkTo}}
{{/if}}

Controllers

Currently we have defined one controller that is the currentUser controller to control the active session, we now need to define controllers to react on actions on our forms, once again we’ll start with the signup form:

App.UsersNewController = Ember.ObjectController.extend({
  save: function() {
    var self = this;

    this.content.save().then(function() {
      self.transitionToRoute('index');
    });
  },

  cancel: function() {
    this.content.deleteRecord();
    this.transitionToRoute('index');
  }
});

When the user hits ‘save’ we tell the model to save itself this.content.save() and then we redirect the user to the root when we’re done. Everything else is being handled by ember and ember-data here behind the scenes, validation errors will automatically update the bindings in the template and will NOT call the ‘then’ callback.
When the user hits ‘cancel’ we just clean up the model this.content.deleteRecord() and redirect back to the root.

The sessions controller is a bit more complicated, let’s take a look:

App.SessionsNewController = Ember.ObjectController.extend({
  needs: ['currentUser'],

  save: function() {
    var self = this;

    this.content.save().then(function() {
      var userJSON = self.content.toJSON();
      userJSON.id = 'current';
      var object = self.store.load(App.User, userJSON);
      var user = App.User.find('current');

      self.get('controllers.currentUser').set('content', user);
      self.transitionToRoute('index');
    });
  },

  cancel: function() {
    this.content.deleteRecord();
    this.transitionToRoute('index');
  }
});

The ‘cancel’ action here is exactly the same as before, so let’s talk about saving. The first step is the same, we ask ember-data to ‘save’ our model this.content.save() and we hook into the ‘then’ callback. Since our create session endpoint is return a user’s JSON we’re gonna use it to populate the currentUser controller. At this point self.content.toJSON() will have the user’s email, but it may have more information as you application needs grow. We have to manually set the ‘id’ property on that JSON object since ember-data will not serialize it. We set it to ‘current’ to refer to our singleton currentUser.
And then we finally load that into a User object using the ember-data store. We then set the content of our currentUser controller to be that loaded user: self.get('controllers.currentUser').set('content', user).

And there we have it, if everything is wired correctly we should have a working signup/login/logout app. I hope it was as helpful for you as it was for me.

References

  1. Alexander Zaytsev – Using Rails & Devise with Ember.js
  2. Brian Cardarella – Building an Ember app with RailsAPI
  3. Jesse Wolgamott – API JSON authentication with Devise

Caveats

1. I had to add this line to my devise.rb file inorder to be able to get rid of the devise_for :users on the routes.rb file. That’s because devise won’t provide you with the dynamic helpers like current_user or user_signed_in? if you don’t call add_mapping, and devise_for calls that method underneath the hood.

config.add_mapping :users, {}

2. This strategy relies on cookies and sessions, if you’re going with this instead of using a auth token you should make sure your app is secure by having protect_from_forgery on. Then you will need to hack into ember-data’s ajax adapter to send the csrf-token on every request. You can achieve that with something like this:

$(function() {
  var token = $('meta[name="csrf-token"]').attr('content');
  $.ajaxPrefilter(function(options, originalOptions, xhr) {
    xhr.setRequestHeader('X-CSRF-Token', token);
  });
});
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Andrew Bruce

Designing an API in Hell

Andrew Bruce
Sunday, May 19, 2013

Minitest, Ruby’s built-in testing library, has some great out-of-the-box features. One of these is test parallelization. Parallel testing is often added after a suite gets slow enough to hurt. That can be achieved using the parallel_tests gem, which takes advantage of today’s multi-core processors, or using custom solutions for dividing chunks of a suite across several machines. Arguably, test speed should be dealt with by making code design changes, but that’s another story: what interests me most about minitest’s parallelization is the constraints it places upon the design of stateful systems when TDDing from scratch.

You can turn on parallelization for a particular test case:

describe Server do
  parallelize_me!
end

or for all tests:

require 'minitest/hell'

As the name implies, the latter approach turns up the test pain level to 11, but it’s the kind of pain that can have positive effects. For ‘fun’, I started to use minitest’s parallelization on a side project, which has a stateful API backed by a relational database. Here are some of the decisions that were forced out by using parallel test examples.

Commitment to fast tests

I thought that running tests in parallel from the start of a project would make me lazy, causing me to neglect slow tests because they’d be running at the same time as others. Surprisingly, the opposite happened: the need to constantly rerun the whole suite to iron out nondeterministic conflicts encouraged me to fix slow tests early. I ended up being able to run the entire suite several times within a matter of seconds in order to check the tests’ ability to run in parallel.

Side note: this is an early-stage project, with a very low quantity of tests! It will be interesting to see how test speed increases as the volume of tests increases.

Avoiding test duplication

Since the unique constraints of my database could be hit by tests with the same fixture data running at the same time, I was encouraged to use more intention revealing test data for each example, avoiding foo and bar, which commonly litter a suite and make tests harder to read.

For IDs, I used Ruby’s SecureRandom library, which provides GUID and hex generation. I sometimes used hex generation when the user-supplied unique display name of something didn’t matter to the test.

Client-side ID generation

Although not strictly forced out from parallel tests, parallel testing got me thinking about how best to interact with the backend, which has a single database being served by multiple concurrent requests (just like any web server).

Using GUIDs instead of autoincrementing IDs can be a smart decision to make if you can (i.e. you don’t need human-friendly URLs), because it means your database server doesn’t need to worry as much about ensuring uniqueness, since the GUID algorithm effectively guarantees it.

TDDing my API from scratch, without external requirements, encouraged me to use GUIDs to simplify the design and to avoid bottlenecks at the database layer. POST requests canonically return the new URL of the resource you’re creating in the Location header of the response. So to test that a thing really got persisted I’d need to:

  1. POST to /items with a representation of the resource
  2. Grab the Location header of the response to get the new URL
  3. GET /items/:newid and ensure the response body matched the representation I sent

This seemed a very laborious process for storing some data. Much less work is:

  1. PUT to /items/:newid
  2. GET /items/:newid and ensure the response body matched the representation I sent

Since GUIDs can be treated as unique, it didn’t make much sense for the server to generate them.

Positive effect: the app would now cope with a distributed database system on the back-end, despite starting out on a technology that’s thought of as difficult to scale horizontally (SQLite).

Avoiding database resets

It’s common practice to wipe the whole database when starting a new example, or to run each example in a transaction and roll it back when each example finishes. I wanted real black-box tests, so I didn’t want to use transactions. Yet, deleting the whole database at the start of an example didn’t play nice with minitest’s parallelization, since data that one example required would be deleted by another.

The usual approach when using parallel_tests is to create a database for each process. However, since minitest doesn’t manage databases (nor should it) I chose to keep a single database and find a different solution.

I chose to sandbox all of the tests by creating new entities each time, and only checking for output that indicated that particular entity had been worked on. The product I’m working on is a Continuous Integration server, so I’d be creating CI projects (you might also know them as jobs) and expecting them to appear in an XML feed. The tests had to be OK with other data being present, since the other tests could be working too.

This approach precluded tests that checked that the number of records had increased by one, because in an otherwise acceptable “green” test situation they’d occasionally increase by more than one (another test added a record too), stay the same (another test deleted a record) or decrease (more than one had been deleted).

Constraints are fun

While I wouldn’t recommend going rogue like this on a client project, playing with constraints like truly parallel tests can get you thinking about your normal testing procedure. Some of the above decisions allowed for a much faster test execution time, and always having the assumption that other processes could be working on the database forced out some interesting techniques. Some of the techniques I had to avoid due to parallelization would normally necessitate different workarounds with their own drawbacks. For example, if you always assume the count of an ActiveRecord class will go up, you require exclusive use of the database. If instead you scope your queries to a parent entity, this restriction would be removed.

Hell isn’t so bad after all.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Robbie Clutton

Stop leaky APIs

Robbie Clutton
Wednesday, May 15, 2013

There are many blogs about how to expose an API for a Rails application and many times I look at this and am concerned about how these examples often leak the application design and the schema out through the API. When this leak occurs a change to the application internals can ripple out and break clients of an API, or force applications to namespace URI paths which I feel is unnecessary and ugly.

When the only consumer of application data models are the views within the same application then the object design can be fluid and malleable. Once an application exposes an API to more than one client, and especially if that client is on a different release cycle to the server, such as iPhone application, data models become rigid. Rails tends discouraged N-tier architecture to the benefit of development speed but APIs are contracts between a server and it’s client and can be difficult to change once they start being used.

Passing an object into the Rails JSON serialisation methods will work for a time, but relying on this will only get you so far. At some point a refactor will take place that will cause a breaking change. It could be something simple such as renaming a column, moving responsibilities from one class to another or adding extra meta-data to a response. Either way, adding this information into your model class starts to place more responsibilities into one place.

There are a few ways out of this potential issue. Let’s take a look at the classic blog application and its Post object. The Rails rendering engine will call as_json on an object if the request has sent the content-type of application\json to the server. Here we override the implementation from ActiveRecord to provide a stable, known version:

def as_json(options={})
    {
        author_id: author.id
        title: title
    }
end

A second option is to model the object explicitly and serialise the internal model into a public representation. We can duck-type the object to respond how ActiveRecord objects behave during a serialisation call. Although this can be seen as a step towards a N-tier architecture, it’s also a step towards service dependent abstraction:

class Api::Post
  attr_reader :post

  def initialize(post)
    @post = post
  end

  def as_json(options={})
    {
      author_id: post.author.id
      title: post.title
    }
  end
end

The benefit of doing this is a separation of concerns between your data model and the data presentation. An application model doesn’t need to know how it’ll be represented by an API, command line interface or any other outside communication mechanism. If an application were tending more towards HATEOAS for instance this separation could help resolve hyperlinks relevant to the interface. You may lose some of the Rails respond_with goodness with this:

respond_to :html, :json

def show
  post = Post.find(params[:id])
  respond_to |format| do
    format.html { @post = post }
    format.json { render json: Api::Post.new(post) }
  end
end

That can be regained with the help of a presenter:

respond_to :html, :json

def show
  post = Post.find(params[:id])
  @presenter = PostPresenter.new(post)
  respond_with @presenter
end

Where PostPresenter may look something like:

class PostPresenter < SimpleDelegator
  def as_json(options={})
    Api::Post.new(self).as_json(options)
  end
end

What’s the difference between this and putting the as_json method into Post directly? More control, separation of concerns with application modeling vs presentation and the big win is when breaking changes occur within the API. Now we can put version relevant information into new objects, or into the serialised class itself.

class Api::Post
  attr_reader :post, :version

  def initialize(post, version)
    @post = post
    @version = version
  end

  def as_json(options={})
    send("v#{:version}")
  end

  private
  def v20130505
    # version specific JSON
  end

  def v20121206
    # version specific JSON
  end
end

Through this we have versioning information in one place and through a request parameter of something like v=20130506 the application can handle multiple versions in one object. For me, this ultimately removes URIs like /v1/posts, but why is that important? The URI is an identifier which points to a resource and having v1 or v2 in the URI muddies the fact that the two identifiers are pointing to the same resource. Using a request parameter, much like pagination is handled, means we can ask for a representation of that resource rather than having to specify different resources. Then we can do away with needing controllers such as Api::V1::PostsController and just deal with Api::PostsController or even just PostsController and deal with the versioning within the object instead of the URI path.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Graham Siener

Feature hydra: how many heads does your product have?

Graham Siener
Monday, April 29, 2013

Should a web and iOS project have one or two tracks/teams/IPMs?

I posted that question to our internal Q&A forum a few months ago.  We were kicking off a client on a large project, building multiple applications for web (incl. mobile) and iOS. We expected some disparities in functionality between the two but assumed the iOS client would be a subset of the website. The client dedicated a PM for each track and there were many shared resources across both projects (designers, UX, backend, other in-house integrations).  One interesting wrinkle that would be a driving factor: the web would be a backbone app driven by an API that another team was in the process of developing.

I was fortunate to get a variety of viewpoints from Pivots that had been through similar projects (hat tip to Onsi for his great answer that influenced this post!).  My hypothesis was that we should stick to one IPM, one standup, etc. and effectively stay as one team as long as possible.  Three months in, I wanted to share an experience report on what has worked and where we’re still improving.

Clear Roles

Having a PM for each platform meant we could effectively manage the user experience specific to the device.  Mark Pincus pushes for everyone to be the “CEO of Something” and this structure ensured that the PM was not a bottleneck as our development team ramped up to six pairs.  Having one designer meant our UI and UX felt consistent across platforms.  Do not take for granted that a user transitioning from one device to another will be disoriented by an inconsistent experience.  Be careful to watch out for communication issues between PMs, and make sure they’re both describing mutually compatible products. It’s possible to specify a feature in a way that’s subtly incompatible between web and iOS — try to catch these issues early!

Group Discussion

We started off with one IPM but this grew unwieldy (talking through, and pointing out, six pairs worth of work just takes time).  We decided to split into two IPMs but the anchor from iOS joined the Web IPM and vice versa.  We also ensured that these anchors were part of a pre-IPM process that delivered the broad themes and saved the larger IPM audience from unimportant decision points.

Own the API

As I mentioned, we started off with the expectation that we’d develop our front ends while another team worked on an API to expose their backend.  It only took a few weeks to realize this would create a lot of churn and we absorbed a layer of API that let us iterate quickly and optimize endpoints for iOS.

The web is an agile-friendly platform where rapid deployment is possible. iOS, thanks to Apple’s review process, is not. Tying minor tweaks on the web client to a (potentially non-backward compatible) full deploy of the API will almost certainly land you into trouble. You have to think carefully about backward compatibility with the iOS app; imposing a clear separation between the web client and the Rails API really helps.  Have (API-level) integration tests in both your iOS suite and your web suite that hit the API app. These integration tests represent the contract that your API agrees to satisfy.

Parallelization is Hard

Understand that parallelizability will be very difficult at the beginning of the project and at major milestones of each sub-project (iOS and web). Breaking stories into tracks of work helps to highlight this.  One consequence of two backlogs is a tight coupling of stories across two Tracker projects.  iOS features that rely on additions to the API are necessarily blocked until the API team can deliver them, and the PMs must negotiate the priority of these stories in relation to web-centric features.  We opted to separate iOS and Web work, but perhaps a more fluid team could better navigate these dependencies.

Build Product One Screen At a Time

We’re actively trying to improve the story mapping/ideation process for new features.  It’s helpful to develop high-level features for all platforms at the same time, but it’s unusual that a web-focused feature can be copied verbatim to iOS.  Similarly, who breaks the tie when the web and iOS PMs have a different viewpoint on functionality or user experience?  Having one ultimate product owner could potentially address this point, yet it’s unclear how they wouldn’t be a giant bottleneck for both teams.

Closing Thoughts

Building this much (this fast) is fun!  We stood up complete experiences across three platforms while rapidly iterating on functionality.  With a team of this scale it’s critical that you’re re-Incepting your project and including the whole team in roadmap discussions to create a shared product vision.

What did I miss?  What are some best practices you’ve discovered managing products across web and mobile?

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Dan Podsedly

Pivotal Tracker API V2 To Be Removed on Jan 27

Dan Podsedly
Thursday, January 5, 2012

We’re working on a new version of the Pivotal Tracker developer API, with a long list of improvements. The release of this is a few months away, but to prepare for it, we’re removing the old version (V2) of the API. This change (removal of V2) is tentatively planned for January 27, 2012, just over three weeks from today.

If you’re using the API, please make sure that you’re using the current version (V3), as requests to V2 will no longer work.

To see which version you’re using, look at the URLs of the requests you’re sending. If you see “v2″ in the path, for example /services/v2/projects, you’re using the old version, and will need to upgrade to the current API version.

If you have any questions, or could use help figuring out how to make your code work with the V3 version of the API, please send an email to tracker@pivotallabs.com.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Dan Podsedly

Apigee Pivotal Tracker API Console

Dan Podsedly
Wednesday, December 28, 2011

Apigee, a company that helps you use and develop APIs, just announced a number of new API consoles, including for Pivotal Tracker. The Pivotal Tracker API Console allows you to explore the API from within your browser, and makes it easy to test and debug your code that uses the API. Check it out!

For the rest of the new consoles, see the Apigee announcement blog post.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Dan Podsedly

New in Tracker: Story assignment notification, API additions

Dan Podsedly
Tuesday, March 29, 2011

We’ve added a new email notification to Pivotal Tracker which will let you know when someone assigns a story to you. There are also a few new additions to the API, related to projects and iterations.

Story Assignment Notification

In the typical Tracker workflow, stories stay unassigned as they move their way up the backlog and into the current iteration. When a developer frees up, he or she starts the next unstarted story, and becomes that story’s owner.

There are times when it makes sense to assign a story to a particular person before it gets started by someone else, for example when the story calls for certain skills or specialization, like design or UX work.

With this new notification, when you make someone other than yourself the owner of a story, that person will receive an email, saying that the story has been assigned to them. As with all story notifications, it’s possible to reply to that story right from your email inbox, in order to initiate or continue the evolving conversation that the story represents.

This notification is enabled by default, but can be turned off on your Profile page. Look for the Assigned stories checkbox, in the Email Preferences section.

API Enhancements

We’ve added the following information to the projects API response: project start date (if specified in project settings), the exact date/time the first iteration started (based on the date of the first accepted story, or an explicit project start date), current iteration number, and whether tasks are enabled.

Also, iterations now include team strength, all list elements should now include the type=’array’ attribute (to make ActiveResource happy), and you can retrieve the current iteration and all iterations in the backlog in one call, like this:

 curl -H "X-TrackerToken: TOKEN" -X GET http://www.pivotaltracker.com/services/v3/projects/PROJECT_ID/iterations/current_backlog

See the API help page for details and examples.

Note: We’re planing on making more significant improvements to the API soon, as part of a new API version (V4). Look for more information on that in the next few weeks.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Presenters and Logical APIs

Pivotal Labs
Tuesday, September 21, 2010

MVC in Rails and Thick Views

The default way of using ActionController and ERB in Rails is to pass models into your views, and let the views figure out what to render and where. There’s been some pushback in the community on this, people are talking about getting rid of “thick views” with logicless templates like Mustache. Mustache is awesome, but if thick views are the problem, we don’t need to go that far if we just change the way we think about controllers a little bit.

Thick views are a controller problem

Consider this ERB, using a loop instead of partials for illustration:

<%@posts.each do |post|%>
  <%=post.author.name%>
<%end%>

If we write our controller like this, the controller will do two queries:

@posts = Post.all(:include => :author)

If we write our controller like this, it will do only one query in the controller:

@posts = Post.all

And the view will do as many queries as there are posts. I’m sure you’ve seen code like this in projects you have worked on. In the standard Rails model, we pass ActiveRecord objects to our views, they might or might not be pre-populated, and thin views can be hard to tell from thick views.

We can explicitly test this by mocking Post and counting queries, but that is a lot of extra work, and you have to remember to do it everywhere.

APIs are unit testable

In contrast to standard controller tests, the result of a JSON API call can be declaratively described. The difference between

{:posts => [:id => 4, :title => "Jerusalem", :author_id => 33]}

and

{:posts => [:id => 4, :title => "Jerusalem", :author => {:name => "Josephus, :id => 33}]}

Is pretty clear, and easy to assert on. If we TDD our controller methods this way it protects us from a lot of the n-query problems in views.

API work comes too late

I’ve seen serveral teams experience pain because Rails does not make it easy to develop a JSON API alongside your HTTP controllers. Rails 3 is better than Rails 2 on this, but you still have to handle the cases separately, particularly as things get more complex.

Twitter is a good example of this. The HTML endpoint recently started consuming its own JSON API over HTTP for generating pages:

One of the most important architectural changes is that Twitter.com is now a client of our own API. It fetches data from the same endpoints that the mobile site, our apps for iPhone, iPad, Android, and every third-party application use. This shift allowed us to allocate more resources to the API team, generating over 40 patches. In the initial page load and every call from the client, all data is now fetched from a highly optimized JSON fragment cache.

This is a very cool pattern in general for a web application. Consuming your own API keeps your business logic in one place, separate from your display logic. And caching on JSON API calls can be easier than caching after building HTML. Obviously you don’t want to run two processes when you’re getting started, but we can take inspiration from this as we organize our code.

The Presenter pattern

A Presenter is a controller that delivers data rather than objects. People like Martin Fowler have been talking about this for a while, and Twitter recently announced a major technical change along these lines.

From the Wikipedia article on “Presenter First”, which is mostly aimed at desktop design, but is applicable to the web, too:

When used in GUI applications, this approach allows the presentation logic and business logic of the application to be developed in a test first manner decoupled from on-screen widgets. Thus, the vast majority of the application programming can be tested via unit tests in an automated test suite. In so doing, the reliance on GUI testing tools to perform extensive system testing can be reduced to verifying basic GUI operation or eliminated entirely.

Unlike standard Rails TDD, if you construct your page data as an API call, your controller tests can now assert directly on that data, instead of the objects that your templates can query with. Testing on a data dictionary is more declarative, and it helps you write thinner views.

Make your view a logical client of your API

We can use the Presenter pattern in Rails by having our controllers call the methods that generate our JSON API at the moment right before serialization to a string. That way there is only one code path that does business logic, which solves the divergence problem.

Vanna

Vanna is an exploration of how MVP can work in Rails. It’s not fully functional (in particular it needs a way to do non-200 returns), but it is enough to illustrate the point. Mixing Vanna into ActionController::Metal restructures controller flow to mimic JSON API calls. Your controllers wind up looking like this:

class VillainsController < ApplicationController
  def index
    {:main => {"villains" => Villain.all}}
  end

  def show(opts = params)
    villain = Villain.named(opts["villain"]).first
    sidebar = catchphrases("villains" => villain["partners"])
    {:main => {:villain =>villain, :sidebar => sidebar}}
  end

  def catchphrases(opts=params)
    names = opts["villains"]
    Villain.named(names).map{|p| p["catchphrase"]}
  end
end

How does this differ from a standard Rails controller? For one thing there’s a silly “catchphrases” method, which is a placeholder for an API method that gives a set of fields for a set of records:

def catchphrases(opts=params)
  names = opts["villains"]
  Villain.named(names).map{|p| p["catchphrase"]}
end

This is structured like an API call. It expects to take in a set of keys, and return some subset of fields on the objects with those keys. Exactly the sort of thing an API does, but not something that you usually see in web apps in their early stages. There’s also an explicit (opts=params) in the signature. Let’s take a look at the show method to see why that is there:

def show(opts = params)
  villain = Villain.named(opts["villain"]).first
  sidebar = catchphrases("villains" => villain["partners"])
  {:villain =>villain, :sidebar => sidebar}
end

On the second line of the method we are explicitly calling one method in a controller from another, passing a params hash (suggested by Richard Crowley). We construct a data dictionary for these smaller calls and pass it through to our template.

Here’s what the view looks like:

<div id=main style="width:70%;float:left;border:black 5px solid;">
  <ul>
    <li><%=villain[:name]%></li>
    <li><%=villain[:catchphrase]%></li>
  </ul>
</div>
<div id=sidebar style="width:20%;float:right;border:black 5px solid;">
  <% sidebar.each do |catchphrase| %>
    <%=catchphrase%> <br/>
  <% end %>
</div>

This looks almost exactly like a standard Rails ERB template, except that the top level objects available to you are accessed as locals, not with an @.

API for free

Vanna makes all your controller methods available as API calls automatically. Because you are explicitly returning a data dictionary, there’s no need to have a different code path for your HTML and JSON. The only difference is whether you go to ERB, or call to_json on the dictionary. So now we can do our controller tests as API tests (the tests in Vanna itself are a little different):

 def test_catchphrases
   header "Accept", 'application/json'
   get "/villains/catchphrases?villains=luis"
   assert{ JSON(last_response.body) == ["Hungry like the volcano!"] }
 end

So OK, we can call our catchphrases method as a JSON call.

def test_show_has_catchphrases
  header "Accept", 'application/json'
  get "/villains/show?villain=luis"
  assert{ JSON(last_response.body)["main"]["sidebar"] == ["You're gonna get punted!"] }
end

And the same data is available inside the larger call which includes it.

def test_html_has_sidebar
  get "/villains/show?villain=luis"
  assert{last_response.body.match(/div id=sidebar/) != nil }
end

And the template actually renders out the sidebar.

Now we have a single data retrieval code path, that breaks off into HTML right before we actually render HTML.

The code for Vanna is on Github, but here’s the meat of it:

module Vanna
  def self.included(klass)
    raise "#{klass.name} does not inherit from  ActionController::Metal" unless  klass.ancestors.include?(ActionController::Metal)
    klass.send(:include,  AbstractController::Layouts)
    klass.send(:include,  AbstractController::Callbacks)
    klass.append_view_path "app/views"
    klass.class_eval("def logger; ActionController::Base.logger; end;")
  end

  def process_action(method_name, *args)
    run_callbacks(:process_action, method_name) do    dictionary = send_action(method_name, *args)    dictionary = @layout_pieces.merge(dictionary) if @layout_pieces  && dictionary.is_a?(Hash)    self.response_body = request.format.symbol == :json ?
 dictionary.to_json : html_render(dictionary)
    end
  end
  def html_render(dictionary)
    render(nil, :locals => dictionary)
  end
end

It’s an example more that a real tool – it’s a demonstration of a valuable way to think about building web applications.

Try it out

It’s extremely rough still, but it serves pages. Here’s the setup:

Gemfile

gem 'vanna', :git => 'git://github.com/MikeSofaer/vanna.git'

ApplicationController

Change ‘ActionController::Base’ to ‘Vanna::Base’

layouts/application.html.erb

Remove the javascript_include line. (and if you know what to do so this isn’t necessary, tell me.)

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Will Read

Write Once, Run Anywhere

Will Read
Monday, May 3, 2010

In response to some recent web browser related debates:
http://sachin.posterous.com/the-web-sucks
http://techcrunch.com/2010/04/30/joe-hewitt-web-development/
http://yehudakatz.com/2010/04/30/the-web-doesnt-suck-browsers-are-innovating/

The web has long since tried to help developers realize the [perhaps misguided] promise that you can write software once and run it anywhere without modification. This might have even seemed feasible when the only real consumer web-enabled device was a desktop or laptop PC with a fairly standard monitor resolution, keyboard, mouse – predictable. You could even bank on IE probably being the browser of choice at one point, like Netscape before it.

But then we started getting choices. Your screen might be 17″, it might be 27″. You might be using a keyboard and tabbing around, or you might only have a finger for input. Both hardware and software have been bringing their A games. Meanwhile web developers cried out for standards adherence in an attempt to create the One Ring to rule them all (but with less fiery eyeballs and dead hobbits).

Then phones made us wake up. The iPhone made the first strong compelling argument to target your web app for a specific resolution, input mechanism, and browser. Now we had two views for our website, the iPhone, and the ‘everything else bucket’.

The ‘everything else bucket’ is probably as good for the web as the No Child Left Behind program is for public education. Sure it has merits, like getting stuff out the door, but for tailored user experience, you’d need a lot more teachers and class rooms, or in our case developers.

Subsequently, yeah iPhone, iPad, Android, WebOS apps rock for UX when you do ‘em up right, but only because a business has placed value in that UX. Conversely, they’ve said that the UX between Firefox and Safari on a desktop is “pretty much the same” and there’s not a lot of value in differentiation.

What I would be curious to see is how much MVC the Rails Way has paid off in this time of many platforms. My guess is that it hasn’t been nearly as helpful as having an easily consumable API. Yes, there’s overlap in those two areas, but picture an API written without knowledge of the MVC pattern and a well structured MVC web site where you now want to add specific view code for PlatformX. I think nine times out of ten you’ll see the API gets more mileage because the views aren’t one-for-one on different platforms.

What might be a separate menu on a phone, could also be part of a side be on a desktop-like presentation. Now you find yourself writing new controllers, so all you’ve really saved on is the model code, which Rails writes most of that for you – you are only slightly better off than if you decided to write the thing from scratch.

It’s easy to argue for this implementation, or that, or ‘…if you rearrange things, then you can…’, but the problem is that there is a cost associated with catering to any specific platform. Developers have to know the nuances of those systems and be able to test in that specific environment. Then you have to balance that cost with the user base you can expect to gain. Until it becomes so cheap to know all the differences and simulate the all combinations of environments, targeted UX will continue to be a low priority for businesses.

Since I know we aren’t all holding our breath for that day, let me pose what may sound like an even more ridiculous solution: What if you could only get to certain parts of the web using a certain browser? Just like some software isn’t written for the Mac, or Windows, or Linux today, what if you could change the expectation that I can surf to any web site with any browser? What if we went to google.com and it said “Sorry, please use Chrome” would that be so awful? Or I went to cnn.com and it said “This site requires Firefox.” Would you do it if the user experience was amazing?

The two expectations, ‘I can go anywhere’, and ‘I can run my software anywhere’ are what cause us developers to beat our heads against the wall when a new browser comes out, or one deviates from the standard. I think if you can find a way to manage those expectations you can get down to brass tax with your users.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Topics

  • agile (783)
  • rails (117)
  • testing (90)
  • ruby (86)
  • ruby on rails (71)
  • jobs (62)
  • javascript (59)
  • techtalk (44)
  • ironblogger (42)
  • rspec (39)
  • bloggerdome (34)
  • productivity (34)
  • activerecord (30)
  • rubymine (30)
  • git (29)
  • gogaruco (29)
  • nyc (27)
  • design (24)
  • mobile (23)
  • pivotal tracker (22)
  • process (21)
  • cucumber (21)
  • jasmine (19)
  • ios (18)
  • tracker ecosystem (17)
  • webos (17)
  • objective-c (17)
  • fun (16)
  • android (16)
  • palm (16)
  • ci (16)
  • "soft" ware (16)
  • bdd (15)
  • tdd (15)
  • cedar (15)
  • rails3 (14)
  • performance (14)
  • css (14)
  • gem (13)
  • mouse-free development (12)
  • selenium (12)
  • goruco (12)
  • bundler (12)
  • api (12)
  • keyboard (11)
  • meetup (11)
  • railsconf (11)
  • nyc-standup (11)
  • capybara (10)
  • mac (10)
Subscribe to api Feed
  1. 1
  2. 2
  3. →
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Tools
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >