Designing an API in Hell

Sunday, May 19, 2013

Minitest, Ruby’s built-in testing library, has some great out-of-the-box features. One of these is test parallelization. Parallel testing is often added after a suite gets slow enough to hurt. That can be achieved using the parallel_tests gem, which takes advantage of today’s multi-core processors, or using custom solutions for dividing chunks of a suite across several machines. Arguably, test speed should be dealt with by making code design changes, but that’s another story: what interests me most about minitest’s parallelization is the constraints it places upon the design of stateful systems when TDDing from scratch.

You can turn on parallelization for a particular test case:

describe Server do
  parallelize_me!
end

or for all tests:

require 'minitest/hell'

As the name implies, the latter approach turns up the test pain level to 11, but it’s the kind of pain that can have positive effects. For ‘fun’, I started to use minitest’s parallelization on a side project, which has a stateful API backed by a relational database. Here are some of the decisions that were forced out by using parallel test examples.

Commitment to fast tests

I thought that running tests in parallel from the start of a project would make me lazy, causing me to neglect slow tests because they’d be running at the same time as others. Surprisingly, the opposite happened: the need to constantly rerun the whole suite to iron out nondeterministic conflicts encouraged me to fix slow tests early. I ended up being able to run the entire suite several times within a matter of seconds in order to check the tests’ ability to run in parallel.

Side note: this is an early-stage project, with a very low quantity of tests! It will be interesting to see how test speed increases as the volume of tests increases.

Avoiding test duplication

Since the unique constraints of my database could be hit by tests with the same fixture data running at the same time, I was encouraged to use more intention revealing test data for each example, avoiding foo and bar, which commonly litter a suite and make tests harder to read.

For IDs, I used Ruby’s SecureRandom library, which provides GUID and hex generation. I sometimes used hex generation when the user-supplied unique display name of something didn’t matter to the test.

Client-side ID generation

Although not strictly forced out from parallel tests, parallel testing got me thinking about how best to interact with the backend, which has a single database being served by multiple concurrent requests (just like any web server).

Using GUIDs instead of autoincrementing IDs can be a smart decision to make if you can (i.e. you don’t need human-friendly URLs), because it means your database server doesn’t need to worry as much about ensuring uniqueness, since the GUID algorithm effectively guarantees it.

TDDing my API from scratch, without external requirements, encouraged me to use GUIDs to simplify the design and to avoid bottlenecks at the database layer. POST requests canonically return the new URL of the resource you’re creating in the Location header of the response. So to test that a thing really got persisted I’d need to:

POST to /items with a representation of the resource
Grab the Location header of the response to get the new URL
GET /items/:newid and ensure the response body matched the representation I sent

This seemed a very laborious process for storing some data. Much less work is:

PUT to /items/:newid
GET /items/:newid and ensure the response body matched the representation I sent

Since GUIDs can be treated as unique, it didn’t make much sense for the server to generate them.

Positive effect: the app would now cope with a distributed database system on the back-end, despite starting out on a technology that’s thought of as difficult to scale horizontally (SQLite).

Avoiding database resets

It’s common practice to wipe the whole database when starting a new example, or to run each example in a transaction and roll it back when each example finishes. I wanted real black-box tests, so I didn’t want to use transactions. Yet, deleting the whole database at the start of an example didn’t play nice with minitest’s parallelization, since data that one example required would be deleted by another.

The usual approach when using parallel_tests is to create a database for each process. However, since minitest doesn’t manage databases (nor should it) I chose to keep a single database and find a different solution.

I chose to sandbox all of the tests by creating new entities each time, and only checking for output that indicated that particular entity had been worked on. The product I’m working on is a Continuous Integration server, so I’d be creating CI projects (you might also know them as jobs) and expecting them to appear in an XML feed. The tests had to be OK with other data being present, since the other tests could be working too.

This approach precluded tests that checked that the number of records had increased by one, because in an otherwise acceptable “green” test situation they’d occasionally increase by more than one (another test added a record too), stay the same (another test deleted a record) or decrease (more than one had been deleted).

Constraints are fun

While I wouldn’t recommend going rogue like this on a client project, playing with constraints like truly parallel tests can get you thinking about your normal testing procedure. Some of the above decisions allowed for a much faster test execution time, and always having the assumption that other processes could be working on the database forced out some interesting techniques. Some of the techniques I had to avoid due to parallelization would normally necessitate different workarounds with their own drawbacks. For example, if you always assume the count of an ActiveRecord class will go up, you require exclusive use of the database. If instead you scope your queries to a parent entity, this restriction would be removed.

Hell isn’t so bad after all.

Stop leaky APIs

Robbie Clutton

Wednesday, May 15, 2013

There are many blogs about how to expose an API for a Rails application and many times I look at this and am concerned about how these examples often leak the application design and the schema out through the API. When this leak occurs a change to the application internals can ripple out and break clients of an API, or force applications to namespace URI paths which I feel is unnecessary and ugly.

When the only consumer of application data models are the views within the same application then the object design can be fluid and malleable. Once an application exposes an API to more than one client, and especially if that client is on a different release cycle to the server, such as iPhone application, data models become rigid. Rails tends discouraged N-tier architecture to the benefit of development speed but APIs are contracts between a server and it’s client and can be difficult to change once they start being used.

Passing an object into the Rails JSON serialisation methods will work for a time, but relying on this will only get you so far. At some point a refactor will take place that will cause a breaking change. It could be something simple such as renaming a column, moving responsibilities from one class to another or adding extra meta-data to a response. Either way, adding this information into your model class starts to place more responsibilities into one place.

There are a few ways out of this potential issue. Let’s take a look at the classic blog application and its Post object. The Rails rendering engine will call as_json on an object if the request has sent the content-type of application\json to the server. Here we override the implementation from ActiveRecord to provide a stable, known version:

def as_json(options={})
    {
        author_id: author.id
        title: title
    }
end

A second option is to model the object explicitly and serialise the internal model into a public representation. We can duck-type the object to respond how ActiveRecord objects behave during a serialisation call. Although this can be seen as a step towards a N-tier architecture, it’s also a step towards service dependent abstraction:

class Api::Post
  attr_reader :post

  def initialize(post)
    @post = post
  end

  def as_json(options={})
    {
      author_id: post.author.id
      title: post.title
    }
  end
end

The benefit of doing this is a separation of concerns between your data model and the data presentation. An application model doesn’t need to know how it’ll be represented by an API, command line interface or any other outside communication mechanism. If an application were tending more towards HATEOAS for instance this separation could help resolve hyperlinks relevant to the interface. You may lose some of the Rails respond_with goodness with this:

respond_to :html, :json

def show
  post = Post.find(params[:id])
  respond_to |format| do
    format.html { @post = post }
    format.json { render json: Api::Post.new(post) }
  end
end

That can be regained with the help of a presenter:

respond_to :html, :json

def show
  post = Post.find(params[:id])
  @presenter = PostPresenter.new(post)
  respond_with @presenter
end

Where PostPresenter may look something like:

class PostPresenter < SimpleDelegator
  def as_json(options={})
    Api::Post.new(self).as_json(options)
  end
end

What’s the difference between this and putting the as_json method into Post directly? More control, separation of concerns with application modeling vs presentation and the big win is when breaking changes occur within the API. Now we can put version relevant information into new objects, or into the serialised class itself.

class Api::Post
  attr_reader :post, :version

  def initialize(post, version)
    @post = post
    @version = version
  end

  def as_json(options={})
    send("v#{:version}")
  end

  private
  def v20130505
    # version specific JSON
  end

  def v20121206
    # version specific JSON
  end
end

Through this we have versioning information in one place and through a request parameter of something like v=20130506 the application can handle multiple versions in one object. For me, this ultimately removes URIs like /v1/posts, but why is that important? The URI is an identifier which points to a resource and having v1 or v2 in the URI muddies the fact that the two identifiers are pointing to the same resource. Using a request parameter, much like pagination is handled, means we can ask for a representation of that resource rather than having to specify different resources. Then we can do away with needing controllers such as Api::V1::PostsController and just deal with Api::PostsController or even just PostsController and deal with the versioning within the object instead of the URI path.

Feature hydra: how many heads does your product have?

Graham Siener

Monday, April 29, 2013

Should a web and iOS project have one or two tracks/teams/IPMs?

I posted that question to our internal Q&A forum a few months ago. We were kicking off a client on a large project, building multiple applications for web (incl. mobile) and iOS. We expected some disparities in functionality between the two but assumed the iOS client would be a subset of the website. The client dedicated a PM for each track and there were many shared resources across both projects (designers, UX, backend, other in-house integrations). One interesting wrinkle that would be a driving factor: the web would be a backbone app driven by an API that another team was in the process of developing.

I was fortunate to get a variety of viewpoints from Pivots that had been through similar projects (hat tip to Onsi for his great answer that influenced this post!). My hypothesis was that we should stick to one IPM, one standup, etc. and effectively stay as one team as long as possible. Three months in, I wanted to share an experience report on what has worked and where we’re still improving.

Clear Roles

Having a PM for each platform meant we could effectively manage the user experience specific to the device. Mark Pincus pushes for everyone to be the “CEO of Something” and this structure ensured that the PM was not a bottleneck as our development team ramped up to six pairs. Having one designer meant our UI and UX felt consistent across platforms. Do not take for granted that a user transitioning from one device to another will be disoriented by an inconsistent experience. Be careful to watch out for communication issues between PMs, and make sure they’re both describing mutually compatible products. It’s possible to specify a feature in a way that’s subtly incompatible between web and iOS — try to catch these issues early!

Group Discussion

We started off with one IPM but this grew unwieldy (talking through, and pointing out, six pairs worth of work just takes time). We decided to split into two IPMs but the anchor from iOS joined the Web IPM and vice versa. We also ensured that these anchors were part of a pre-IPM process that delivered the broad themes and saved the larger IPM audience from unimportant decision points.

Own the API

As I mentioned, we started off with the expectation that we’d develop our front ends while another team worked on an API to expose their backend. It only took a few weeks to realize this would create a lot of churn and we absorbed a layer of API that let us iterate quickly and optimize endpoints for iOS.

The web is an agile-friendly platform where rapid deployment is possible. iOS, thanks to Apple’s review process, is not. Tying minor tweaks on the web client to a (potentially non-backward compatible) full deploy of the API will almost certainly land you into trouble. You have to think carefully about backward compatibility with the iOS app; imposing a clear separation between the web client and the Rails API really helps. Have (API-level) integration tests in both your iOS suite and your web suite that hit the API app. These integration tests represent the contract that your API agrees to satisfy.

Parallelization is Hard

Understand that parallelizability will be very difficult at the beginning of the project and at major milestones of each sub-project (iOS and web). Breaking stories into tracks of work helps to highlight this. One consequence of two backlogs is a tight coupling of stories across two Tracker projects. iOS features that rely on additions to the API are necessarily blocked until the API team can deliver them, and the PMs must negotiate the priority of these stories in relation to web-centric features. We opted to separate iOS and Web work, but perhaps a more fluid team could better navigate these dependencies.

Build Product One Screen At a Time

We’re actively trying to improve the story mapping/ideation process for new features. It’s helpful to develop high-level features for all platforms at the same time, but it’s unusual that a web-focused feature can be copied verbatim to iOS. Similarly, who breaks the tie when the web and iOS PMs have a different viewpoint on functionality or user experience? Having one ultimate product owner could potentially address this point, yet it’s unclear how they wouldn’t be a giant bottleneck for both teams.

Closing Thoughts

Building this much (this fast) is fun! We stood up complete experiences across three platforms while rapidly iterating on functionality. With a team of this scale it’s critical that you’re re-Incepting your project and including the whole team in roadmap discussions to create a shared product vision.

What did I miss? What are some best practices you’ve discovered managing products across web and mobile?

Pivotal Tracker API V2 To Be Removed on Jan 27

Dan Podsedly

Thursday, January 5, 2012

We’re working on a new version of the Pivotal Tracker developer API, with a long list of improvements. The release of this is a few months away, but to prepare for it, we’re removing the old version (V2) of the API. This change (removal of V2) is tentatively planned for January 27, 2012, just over three weeks from today.

If you’re using the API, please make sure that you’re using the current version (V3), as requests to V2 will no longer work.

To see which version you’re using, look at the URLs of the requests you’re sending. If you see “v2″ in the path, for example /services/v2/projects, you’re using the old version, and will need to upgrade to the current API version.

If you have any questions, or could use help figuring out how to make your code work with the V3 version of the API, please send an email to tracker@pivotallabs.com.

Apigee Pivotal Tracker API Console

Dan Podsedly

Wednesday, December 28, 2011

Apigee, a company that helps you use and develop APIs, just announced a number of new API consoles, including for Pivotal Tracker. The Pivotal Tracker API Console allows you to explore the API from within your browser, and makes it easy to test and debug your code that uses the API. Check it out!

For the rest of the new consoles, see the Apigee announcement blog post.

New in Tracker: Story assignment notification, API additions

Dan Podsedly

Tuesday, March 29, 2011

We’ve added a new email notification to Pivotal Tracker which will let you know when someone assigns a story to you. There are also a few new additions to the API, related to projects and iterations.

Story Assignment Notification

In the typical Tracker workflow, stories stay unassigned as they move their way up the backlog and into the current iteration. When a developer frees up, he or she starts the next unstarted story, and becomes that story’s owner.

There are times when it makes sense to assign a story to a particular person before it gets started by someone else, for example when the story calls for certain skills or specialization, like design or UX work.

With this new notification, when you make someone other than yourself the owner of a story, that person will receive an email, saying that the story has been assigned to them. As with all story notifications, it’s possible to reply to that story right from your email inbox, in order to initiate or continue the evolving conversation that the story represents.

This notification is enabled by default, but can be turned off on your Profile page. Look for the Assigned stories checkbox, in the Email Preferences section.

API Enhancements

We’ve added the following information to the projects API response: project start date (if specified in project settings), the exact date/time the first iteration started (based on the date of the first accepted story, or an explicit project start date), current iteration number, and whether tasks are enabled.

Also, iterations now include team strength, all list elements should now include the type=’array’ attribute (to make ActiveResource happy), and you can retrieve the current iteration and all iterations in the backlog in one call, like this:

 curl -H "X-TrackerToken: TOKEN" -X GET http://www.pivotaltracker.com/services/v3/projects/PROJECT_ID/iterations/current_backlog

See the API help page for details and examples.

Note: We’re planing on making more significant improvements to the API soon, as part of a new API version (V4). Look for more information on that in the next few weeks.

Presenters and Logical APIs

Pivotal Labs

Tuesday, September 21, 2010

MVC in Rails and Thick Views

The default way of using ActionController and ERB in Rails is to pass models into your views, and let the views figure out what to render and where. There’s been some pushback in the community on this, people are talking about getting rid of “thick views” with logicless templates like Mustache. Mustache is awesome, but if thick views are the problem, we don’t need to go that far if we just change the way we think about controllers a little bit.

Thick views are a controller problem

Consider this ERB, using a loop instead of partials for illustration:

<%@posts.each do |post|%>
  <%=post.author.name%>
<%end%>

If we write our controller like this, the controller will do two queries:

@posts = Post.all(:include => :author)

If we write our controller like this, it will do only one query in the controller:

@posts = Post.all

And the view will do as many queries as there are posts. I’m sure you’ve seen code like this in projects you have worked on. In the standard Rails model, we pass ActiveRecord objects to our views, they might or might not be pre-populated, and thin views can be hard to tell from thick views.

We can explicitly test this by mocking Post and counting queries, but that is a lot of extra work, and you have to remember to do it everywhere.

APIs are unit testable

In contrast to standard controller tests, the result of a JSON API call can be declaratively described. The difference between

{:posts => [:id => 4, :title => "Jerusalem", :author_id => 33]}

and

{:posts => [:id => 4, :title => "Jerusalem", :author => {:name => "Josephus, :id => 33}]}

Is pretty clear, and easy to assert on. If we TDD our controller methods this way it protects us from a lot of the n-query problems in views.

API work comes too late

I’ve seen serveral teams experience pain because Rails does not make it easy to develop a JSON API alongside your HTTP controllers. Rails 3 is better than Rails 2 on this, but you still have to handle the cases separately, particularly as things get more complex.

Twitter is a good example of this. The HTML endpoint recently started consuming its own JSON API over HTTP for generating pages:

One of the most important architectural changes is that Twitter.com is now a client of our own API. It fetches data from the same endpoints that the mobile site, our apps for iPhone, iPad, Android, and every third-party application use. This shift allowed us to allocate more resources to the API team, generating over 40 patches. In the initial page load and every call from the client, all data is now fetched from a highly optimized JSON fragment cache.

This is a very cool pattern in general for a web application. Consuming your own API keeps your business logic in one place, separate from your display logic. And caching on JSON API calls can be easier than caching after building HTML. Obviously you don’t want to run two processes when you’re getting started, but we can take inspiration from this as we organize our code.

The Presenter pattern

A Presenter is a controller that delivers data rather than objects. People like Martin Fowler have been talking about this for a while, and Twitter recently announced a major technical change along these lines.

From the Wikipedia article on “Presenter First”, which is mostly aimed at desktop design, but is applicable to the web, too:

When used in GUI applications, this approach allows the presentation logic and business logic of the application to be developed in a test first manner decoupled from on-screen widgets. Thus, the vast majority of the application programming can be tested via unit tests in an automated test suite. In so doing, the reliance on GUI testing tools to perform extensive system testing can be reduced to verifying basic GUI operation or eliminated entirely.

Unlike standard Rails TDD, if you construct your page data as an API call, your controller tests can now assert directly on that data, instead of the objects that your templates can query with. Testing on a data dictionary is more declarative, and it helps you write thinner views.

Make your view a logical client of your API

We can use the Presenter pattern in Rails by having our controllers call the methods that generate our JSON API at the moment right before serialization to a string. That way there is only one code path that does business logic, which solves the divergence problem.

Vanna

Vanna is an exploration of how MVP can work in Rails. It’s not fully functional (in particular it needs a way to do non-200 returns), but it is enough to illustrate the point. Mixing Vanna into ActionController::Metal restructures controller flow to mimic JSON API calls. Your controllers wind up looking like this:

class VillainsController < ApplicationController
  def index
    {:main => {"villains" => Villain.all}}
  end

  def show(opts = params)
    villain = Villain.named(opts["villain"]).first
    sidebar = catchphrases("villains" => villain["partners"])
    {:main => {:villain =>villain, :sidebar => sidebar}}
  end

  def catchphrases(opts=params)
    names = opts["villains"]
    Villain.named(names).map{|p| p["catchphrase"]}
  end
end

How does this differ from a standard Rails controller? For one thing there’s a silly “catchphrases” method, which is a placeholder for an API method that gives a set of fields for a set of records:

def catchphrases(opts=params)
  names = opts["villains"]
  Villain.named(names).map{|p| p["catchphrase"]}
end

This is structured like an API call. It expects to take in a set of keys, and return some subset of fields on the objects with those keys. Exactly the sort of thing an API does, but not something that you usually see in web apps in their early stages. There’s also an explicit (opts=params) in the signature. Let’s take a look at the show method to see why that is there:

def show(opts = params)
  villain = Villain.named(opts["villain"]).first
  sidebar = catchphrases("villains" => villain["partners"])
  {:villain =>villain, :sidebar => sidebar}
end

On the second line of the method we are explicitly calling one method in a controller from another, passing a params hash (suggested by Richard Crowley). We construct a data dictionary for these smaller calls and pass it through to our template.

Here’s what the view looks like:

<div id=main style="width:70%;float:left;border:black 5px solid;">
  <ul>
    <li><%=villain[:name]%></li>
    <li><%=villain[:catchphrase]%></li>
  </ul>
</div>
<div id=sidebar style="width:20%;float:right;border:black 5px solid;">
  <% sidebar.each do |catchphrase| %>
    <%=catchphrase%> <br/>
  <% end %>
</div>

This looks almost exactly like a standard Rails ERB template, except that the top level objects available to you are accessed as locals, not with an @.

API for free

Vanna makes all your controller methods available as API calls automatically. Because you are explicitly returning a data dictionary, there’s no need to have a different code path for your HTML and JSON. The only difference is whether you go to ERB, or call to_json on the dictionary. So now we can do our controller tests as API tests (the tests in Vanna itself are a little different):

 def test_catchphrases
   header "Accept", 'application/json'
   get "/villains/catchphrases?villains=luis"
   assert{ JSON(last_response.body) == ["Hungry like the volcano!"] }
 end

So OK, we can call our catchphrases method as a JSON call.

def test_show_has_catchphrases
  header "Accept", 'application/json'
  get "/villains/show?villain=luis"
  assert{ JSON(last_response.body)["main"]["sidebar"] == ["You're gonna get punted!"] }
end

And the same data is available inside the larger call which includes it.

def test_html_has_sidebar
  get "/villains/show?villain=luis"
  assert{last_response.body.match(/div id=sidebar/) != nil }
end

And the template actually renders out the sidebar.

Now we have a single data retrieval code path, that breaks off into HTML right before we actually render HTML.

The code for Vanna is on Github, but here’s the meat of it:

module Vanna
  def self.included(klass)
    raise "#{klass.name} does not inherit from  ActionController::Metal" unless  klass.ancestors.include?(ActionController::Metal)
    klass.send(:include,  AbstractController::Layouts)
    klass.send(:include,  AbstractController::Callbacks)
    klass.append_view_path "app/views"
    klass.class_eval("def logger; ActionController::Base.logger; end;")
  end

  def process_action(method_name, *args)
    run_callbacks(:process_action, method_name) do    dictionary = send_action(method_name, *args)    dictionary = @layout_pieces.merge(dictionary) if @layout_pieces  && dictionary.is_a?(Hash)    self.response_body = request.format.symbol == :json ?
 dictionary.to_json : html_render(dictionary)
    end
  end
  def html_render(dictionary)
    render(nil, :locals => dictionary)
  end
end

It’s an example more that a real tool – it’s a demonstration of a valuable way to think about building web applications.

Try it out

It’s extremely rough still, but it serves pages. Here’s the setup:

Gemfile

gem 'vanna', :git => 'git://github.com/MikeSofaer/vanna.git'

ApplicationController

Change ‘ActionController::Base’ to ‘Vanna::Base’

layouts/application.html.erb

Remove the javascript_include line. (and if you know what to do so this isn’t necessary, tell me.)

Write Once, Run Anywhere

Will Read

Monday, May 3, 2010

In response to some recent web browser related debates:
http://sachin.posterous.com/the-web-sucks
http://techcrunch.com/2010/04/30/joe-hewitt-web-development/
http://yehudakatz.com/2010/04/30/the-web-doesnt-suck-browsers-are-innovating/

The web has long since tried to help developers realize the [perhaps misguided] promise that you can write software once and run it anywhere without modification. This might have even seemed feasible when the only real consumer web-enabled device was a desktop or laptop PC with a fairly standard monitor resolution, keyboard, mouse – predictable. You could even bank on IE probably being the browser of choice at one point, like Netscape before it.

But then we started getting choices. Your screen might be 17″, it might be 27″. You might be using a keyboard and tabbing around, or you might only have a finger for input. Both hardware and software have been bringing their A games. Meanwhile web developers cried out for standards adherence in an attempt to create the One Ring to rule them all (but with less fiery eyeballs and dead hobbits).

Then phones made us wake up. The iPhone made the first strong compelling argument to target your web app for a specific resolution, input mechanism, and browser. Now we had two views for our website, the iPhone, and the ‘everything else bucket’.

The ‘everything else bucket’ is probably as good for the web as the No Child Left Behind program is for public education. Sure it has merits, like getting stuff out the door, but for tailored user experience, you’d need a lot more teachers and class rooms, or in our case developers.

Subsequently, yeah iPhone, iPad, Android, WebOS apps rock for UX when you do ‘em up right, but only because a business has placed value in that UX. Conversely, they’ve said that the UX between Firefox and Safari on a desktop is “pretty much the same” and there’s not a lot of value in differentiation.

What I would be curious to see is how much MVC the Rails Way has paid off in this time of many platforms. My guess is that it hasn’t been nearly as helpful as having an easily consumable API. Yes, there’s overlap in those two areas, but picture an API written without knowledge of the MVC pattern and a well structured MVC web site where you now want to add specific view code for PlatformX. I think nine times out of ten you’ll see the API gets more mileage because the views aren’t one-for-one on different platforms.

What might be a separate menu on a phone, could also be part of a side be on a desktop-like presentation. Now you find yourself writing new controllers, so all you’ve really saved on is the model code, which Rails writes most of that for you – you are only slightly better off than if you decided to write the thing from scratch.

It’s easy to argue for this implementation, or that, or ‘…if you rearrange things, then you can…’, but the problem is that there is a cost associated with catering to any specific platform. Developers have to know the nuances of those systems and be able to test in that specific environment. Then you have to balance that cost with the user base you can expect to gain. Until it becomes so cheap to know all the differences and simulate the all combinations of environments, targeted UX will continue to be a low priority for businesses.

Since I know we aren’t all holding our breath for that day, let me pose what may sound like an even more ridiculous solution: What if you could only get to certain parts of the web using a certain browser? Just like some software isn’t written for the Mac, or Windows, or Linux today, what if you could change the expectation that I can surf to any web site with any browser? What if we went to google.com and it said “Sorry, please use Chrome” would that be so awful? Or I went to cnn.com and it said “This site requires Firefox.” Would you do it if the user experience was amazing?

The two expectations, ‘I can go anywhere’, and ‘I can run my software anywhere’ are what cause us developers to beat our heads against the wall when a new browser comes out, or one deviates from the standard. I think if you can find a way to manage those expectations you can get down to brass tax with your users.

API Cake

Will Read

Monday, February 15, 2010

A made up conversation with myself:

Let’s say I have a cool service. Now I want someone to take my cool service and develop against it. So I make an API. I’m using RESTful methods in RoR so it’s pretty straight forward to expose the CRUD actions. Done!

I release it in to the wild. People are jazzed, and they start to make clones and clients that do the same thing as my service. That’s cool, it’s driving an extra .5% of the traffic to my site. But I’m waiting for someone to innovate and blow my mind. With what? I don’t know!

Well, I gave them the ingredients, and the instructions, and they took my API and made an API cake, just like I showed them to make. They can read the product catalog, or they can post messages, and they do.

The cool part for developers is the “intellectual property”, the numbers in most cases. Let me get at how people are using my cool new service so I can start to make conjectures like “if A happens in the context of B, then we usually see C”. Like what if I had the number of facebook status messages about a new comedy show premier, and I search those for the word “funny” and plotted them over time? I could paint you a picture of what parts of the show were entertaining. But I can’t get at that kind of data.

Similarly, what if Amazon told me how many self help books were sold in New Jersey over time? Then I could make a map of where a psychologist should open up a practice.

Some of this information deals with privacy of the users. Some of the information can be anonymized, but is still too close to something the company views as a core process which is currently giving them a competitive edge. That’s exactly why it is interesting, why those developers are hungry for it, why you should release it, and give your users a tremendous experience. If you set loose the data, your developers will be able to creat more than just a copy of existing functionality, they can really innovate and tell you things about your business that your customers didn’t even realize.

Pivotal Tracker API – new version (V3) to be released on Jan. 23

Dan Podsedly

Sunday, January 17, 2010

We’re planning a Pivotal Tracker upgrade on Jan 23. As part of this release, we will be introducing a new API version (V3), which will make it easier to follow project activity, allow you to add file attachments, move (re-prioritize) stories, associate source commits with stories, and more.

The current API version (V2) will not change, but V1 will no longer work. If you’re still using V1, you will need to change your client code to use V2 or V3.

To find out what’s changing in V3, continue reading.

How to tell what version of the API you’re using?

The API version identifier is part of the request URLs. For example, this is a V2 request:


http://www.pivotaltracker.com/services/v2/projects

What’s New or Changed in V3

Activity

The response for the activity queries will change significantly. It will include a version # (to allow you to keep track of unique events and their order), event type, when the activity occurred (with time zone), and a nested element with all story attributes that changed as part of the activity. Example:

<activities type="array">
  <activity>
    <id type="integer">1031</id>
    <version type="integer">175</version>
    <event_type>story_update</event_type>
    <occurred_at type="datetime">2009/12/14 14:12:09 PST</occurred_at>
    <author>James Kirk</author>
    <project_id type="integer">26</project_id>
    <description>James Kirk accepted &quot;More power to shields&quot;</description>
    <stories>
      <story>
        <id type="integer">109</id>
        <url>https://www.pivotaltracker.com/services/v3/projects/26/stories/109</url>
        <accepted_at type="datetime">2009/12/14 22:12:09 UTC</accepted_at>
        <current_state>accepted</current_state>
      </story>
    </stories>
  </activity>
</activities>

You’ll also be able to query for all activity since a particular date or version #, and you limit how many entries to return (up to 100):

curl -H "X-TrackerToken: TOKEN" -X GET http://www.pivotaltracker.com/services/v3/activities?newer_than_version=13
curl -H "X-TrackerToken: TOKEN" -X GET http://www.pivotaltracker.com/services/v3/activities?occurred_since_date=2009/12/14&limit=50

Activity Web Hook

This will allow you to specify a URL per project (in project settings), which Tracker will post story activity to, in the same XML format as above. You’ll be able to “pull” story activity out of Tracker via normal API GET requests, or have it POSTed to your client as it occurs via the activity web hook.

Projects

The project XML response will include the current and initial velocity, last activity date, and a list of all labels in the project.

Example:

<project>
  <id>1</id>
  <name>Sample Project</name>
  <iteration_length type="integer">2</iteration_length>
  <week_start_day>Monday</week_start_day>
  <point_scale>0,1,2,3</point_scale>
  <account>James Kirks Account</account>
  <velocity_scheme>Average of 4 iterations</velocity_scheme>
  <initial_velocity>10</initial_velocity>
  <last_activity_at type="datetime">2010/01/16 17:39:10 CST</last_activity_at>
  <number_of_done_iterations_to_show>12</number_of_done_iterations_to_show>
  <labels>shields,transporter</labels>
  <allow_attachments>true</allow_attachments>
  <public>false</public>
  <use_https>true</use_https>
  <bugs_and_chores_are_estimatable>false</bugs_and_chores_are_estimatable>
  <commit_mode>false</commit_mode>
  <memberships>
    <membership>
      <id>1006</id>
      <person>
        <email>kirkybaby@earth.ufp</email>
        <name>James T. Kirk</name>
        <initials>JTK</initials>
      </person>
      <role>Owner</role>
    </membership>
  </memberships>
</project>

When creating a project via the API, the user represented by the API token will be made an owner of that project by default. To leave the new project without an owner (because your client is acting on behalf of a different user, for example), you’ll need to include <no_owner type="boolean">true</no_owner> in the post data.

Stories

You’ll be able to move (re-prioritize) stories via the API. To move a story to after another story:

curl -H "X-TrackerToken: TOKEN"
    -X POST http://www.pivotaltracker.com/services/v3/projects/PROJECT_ID/stories/STORY_ID/moves?move[move]=after&move[target]=TARGET_STORY_ID"

Or, move it before a story:

curl -H "X-TrackerToken: TOKEN"
    -X POST http://www.pivotaltracker.com/services/v3/projects/PROJECT_ID/stories/STORY_ID/moves?move[move]=before&move[target]=TARGET_STORY_ID

As part of the new integrations feature (watch for more on that later), you’ll be able to associate a story with a ticket or issue in an external system, such as Lighthouse or JIRA. You’ll need to specify a ticket/issue ID, and optionally which specific integration to use (a project may be set up with multiple):

curl -H "X-TrackerToken: TOKEN" -H "Content-type: application/xml"
    -d "<story><lighthouse_id>54</lighthouse_id></story>"
    -X PUT http://www.pivotaltracker.com/services/v3/projects/PROJECT_ID/stories/STORY_ID"

Stories that are linked to a ticket or issue in an external system (for example JIRA or Lighthouse) will include the external ID as an attribute, as well as the URL to the linked ticket/issue:

<story>
  <id type="integer">STORY_ID</id>
  <story_type>feature</story_type>
  <url>http://www.pivotaltracker.com/story/show/STORY_ID</url>
  <estimate type="integer">1</estimate>
  <current_state>unstarted</current_state>
  <lighthouse_id>43</lighthouse_id>
  <lighthouse_url>http://mylighthouseapp.com/projects/100/tickets/43</lighthouse_url>
  <name>More power to shields</name>
  <requested_by>James Kirk</requested_by>
  <created_at type="datetime">2008/12/10 00:00:00 UTC</created_at>
</story>

File attachments

Here’s how you’ll be able to upload a file attachment to a story:

curl -H "X-TrackerToken: TOKEN" -X POST -F Filedata=@/path/to/file http://www.pivotaltracker.com/services/v3/projects/PROJECT_ID/stories/STORY_ID/attachments

The story response will include information about file attachments in this nested XML element:

<attachments type="array">
  <attachment>
    <id type="integer">17</id>
    <filename>Picture_36.png</filename>
    <description></description>
    <uploaded_by>Rob</uploaded_by>
    <uploaded_at type="datetime">2010/01/17 14:57:57 CST</uploaded_at>
  </attachment>
</attachments>

Note: Attachments in the story response XML will most likely not include a URL to the actual AWS S3 file, since these URLs are only valid temporarily. You’ll need to make a separate API call (details TBD) to get the S3 URL for a given story file attachment.

Source Control Post Commit Hooks

This will allow you to set up post-commit hooks in git, github, subversion, etc., to link commits to stories (and optionally mark them as finished) based on this message syntax:

“Torpedoes now sufficiently powered [fixes #123456]“.

Curl example, of what you might do in a custom post-commit hook script for subversion or git:

curl -H "X-TrackerToken: TOKEN" -H "Content-type: application/xml"
    -d "<source_commit><message>$MESSAGE</message><author>$AUTHOR</author><commit_id>$REV</commit_id><url>http://trac.yourcompany.com/browser/?rev=$REV</url></source_commit>"
    -X POST http://www.pivotaltracker.com/services/v3/source_commits

Stories will show associated source commits as comments, with a link to the commit if you include a URL in the post body:

Github Support

The V3 version of the API will also support native Github post-commit hooks, allowing you to configure your Github repo to send commit information directly to Tracker, along with the [fixes #12345] message syntax.

The format of the Github post-commit request will be:


https://www.pivotaltracker.com/services/v3/github_commits?token=API_TOKEN_OF_THE_USER_TO_ATTRIBUTE_ACTIONS_TO