I want to focus this post on a quote from the book:

The definition of “ready for development” can be achieved only if all specialties work together to estimate features, break them into small enough deliverables without losing too much customer value, and to agree on acceptance tests.

Ready for development is an interesting concept — it implies that every business stakeholder that has input knows what the product owner needs to ensure a developer can deliver an acceptable story.  In my experience, this is usually not the case once a product is out in the marketplace and people from non-product teams (marketing, support, executives) have specific product needs.  This could look like promo code tracking to support a marketing campaign, or a data export to support your “data science” team.  Without a good process for getting these stories into the backlog, the result is an unhealthy IPM where the PM and developers have to divine what exactly the intent of a story is.

At one of my last companies, we had this problem in spades.  Even after introducing a Pre-IPM meeting, we still felt like the quality of the stories we presented to developers in IPM didn’t represent the time we had spent discussing them.  To add insult to injury, getting the story requester to accept a delivered story was like pulling teeth.  Once we did sit them down to walk through a story, rejection was often the result!

After reflection (and seeing this come up as a theme in several retros) we created a new weekly meeting called “Storyation Workshop.”  This session felt a lot like office hours, and we made clear the intent to only let high-quality stories into the backlog.  We (PM and Tech Lead) would do our best to interpret the needs for a feature and turn it into actionable stories.  We would also break out the necessary pieces from the nice-to-haves.  Make no mistake, though — the business owners were on the hook to bring justification for a feature and if it didn’t pass muster it lived in the Icebox for another sprint.

We were fortunate (as a company) to have support for keeping a healthy backlog and not jumping stories in the priority queue.  I saw many benefits to this new process:

• All the other teams — marketing, professional services, customer support, data science — paid closer attention in IPMs and made sure that their needs were well represented
• Acceptance happened more quickly after a story was delivered, and expectation alignment meant rejections went way down
• Everyone got better at story writing as they started to grok how we broke down features and the logic behind it
• More greenfield thinking popped up and made it into the backlog as actionable work

The first three were proof to me that we were missing a critical process in the arc of a story.  The last bullet (greenfield thinking) was a pleasant surprise.  It turned out that people were afraid of bringing a half-baked idea to IPM (rightfully so) but didn’t feel like they had the right forum to finish baking an idea.

Maybe this all sounds like common sense; if that’s the case congrats on having a healthy product development process!  For anyone struggling to deal with symptoms like these I suggest you try introducing a safe space for new ideas to come to light.  A Storyation Workshop could come in many forms, but you’ll know it’s working when you see more buy in from non-product team stakeholders and a faster cycle-team for their stories through the backlog.

As always, I’d love to hear if you’ve incorporated a similar technique into your agile product development process.

The post Storyation Workshop: How to get business stakeholders to create stories appeared first on Pivotal Labs.

Should a web and iOS project have one or two tracks/teams/IPMs?

I posted that question to our internal Q&A forum a few months ago.  We were kicking off a client on a large project, building multiple applications for web (incl. mobile) and iOS. We expected some disparities in functionality between the two but assumed the iOS client would be a subset of the website. The client dedicated a PM for each track and there were many shared resources across both projects (designers, UX, backend, other in-house integrations).  One interesting wrinkle that would be a driving factor: the web would be a backbone app driven by an API that another team was in the process of developing.

I was fortunate to get a variety of viewpoints from Pivots that had been through similar projects (hat tip to Onsi for his great answer that influenced this post!).  My hypothesis was that we should stick to one IPM, one standup, etc. and effectively stay as one team as long as possible.  Three months in, I wanted to share an experience report on what has worked and where we’re still improving.

Clear Roles

Having a PM for each platform meant we could effectively manage the user experience specific to the device.  Mark Pincus pushes for everyone to be the “CEO of Something” and this structure ensured that the PM was not a bottleneck as our development team ramped up to six pairs.  Having one designer meant our UI and UX felt consistent across platforms.  Do not take for granted that a user transitioning from one device to another will be disoriented by an inconsistent experience.  Be careful to watch out for communication issues between PMs, and make sure they’re both describing mutually compatible products. It’s possible to specify a feature in a way that’s subtly incompatible between web and iOS — try to catch these issues early!

Group Discussion

We started off with one IPM but this grew unwieldy (talking through, and pointing out, six pairs worth of work just takes time).  We decided to split into two IPMs but the anchor from iOS joined the Web IPM and vice versa.  We also ensured that these anchors were part of a pre-IPM process that delivered the broad themes and saved the larger IPM audience from unimportant decision points.

Own the API

As I mentioned, we started off with the expectation that we’d develop our front ends while another team worked on an API to expose their backend.  It only took a few weeks to realize this would create a lot of churn and we absorbed a layer of API that let us iterate quickly and optimize endpoints for iOS.

The web is an agile-friendly platform where rapid deployment is possible. iOS, thanks to Apple’s review process, is not. Tying minor tweaks on the web client to a (potentially non-backward compatible) full deploy of the API will almost certainly land you into trouble. You have to think carefully about backward compatibility with the iOS app; imposing a clear separation between the web client and the Rails API really helps.  Have (API-level) integration tests in both your iOS suite and your web suite that hit the API app. These integration tests represent the contract that your API agrees to satisfy.

Parallelization is Hard

Understand that parallelizability will be very difficult at the beginning of the project and at major milestones of each sub-project (iOS and web). Breaking stories into tracks of work helps to highlight this.  One consequence of two backlogs is a tight coupling of stories across two Tracker projects.  iOS features that rely on additions to the API are necessarily blocked until the API team can deliver them, and the PMs must negotiate the priority of these stories in relation to web-centric features.  We opted to separate iOS and Web work, but perhaps a more fluid team could better navigate these dependencies.

Build Product One Screen At a Time

We’re actively trying to improve the story mapping/ideation process for new features.  It’s helpful to develop high-level features for all platforms at the same time, but it’s unusual that a web-focused feature can be copied verbatim to iOS.  Similarly, who breaks the tie when the web and iOS PMs have a different viewpoint on functionality or user experience?  Having one ultimate product owner could potentially address this point, yet it’s unclear how they wouldn’t be a giant bottleneck for both teams.

Closing Thoughts

Building this much (this fast) is fun!  We stood up complete experiences across three platforms while rapidly iterating on functionality.  With a team of this scale it’s critical that you’re re-Incepting your project and including the whole team in roadmap discussions to create a shared product vision.

What did I miss?  What are some best practices you’ve discovered managing products across web and mobile?

The post Feature hydra: how many heads does your product have? appeared first on Pivotal Labs.

In agile software development, an information radiator is a (normally large) physical display placed in a prominent location in an office, where passers-by can see it, and which presents an up-to-date summary of the status of a software product or products.[1]

The best way we’ve found to keep the team informed of the build status is to display it high on a wall near the team as a big red or green indicator. When there’s only a single project going on, we’ve found that a screen that’s simply all red or all green is effective. At Pivotal Labs, though, we have many projects going on at once. Rather than putting numerous TVs up on the wall, we’ve created an application that aggregates multiple projects onto one page.

Tddium

Edward announced this four years ago as http://ci.pivotallabs.com.  A year later, we open sourced this as Pivotal Pulse and then renamed it CIMonitor.  That lasted a few years, until we integration Pivotal Tracker details, which prompted a rename to Project Monitor last year.

Cruise Control

Fast forward to me joining Pivotal Labs in November.  Project Monitor was on the walls of our office, but the radiator had a leak.  Our internal Project Monitor instance was running on a server in our Director of IT’s office.  During Sandy, every monitor in SF timed out trying to connect to our nyc-based ci servers.  We had serious scalability issues that affected polling, the delayed job queue was unreliable, and our codebase needed a code debt payment plan.  There had been a few dedicated sprints to triage open wounds, but a long running branch for an eventmachine based polling refactor had lingered as a pull request from several months ago.

This is pretty common in older codebases, especially so for internal projects that don’t get any attention unless they’re on fire.  Even worse, the backlog had suffered from being on autopilot for too long.  I’ve seen this happen in many products that have been in the market for a while — they’ve checked the box on value proposition and moved onto “wouldn’t it be cool if” type functionality.  I was informed that I’d take the reigns as Product Owner and we had a mini-inception on my first day.

Team City

Beyond ramping myself up on a new codebase, I struggled to sort out who Project Monitor’s stakeholders were and what its remit truly was.  Because consumption is inherently passive, I couldn’t rely on metrics or many of the other tools in my belt.  It was working, but was it solving problems?

Luckily, two key events helped me crystallize a thesis for what PM should be.  First, during a trip to our SF office, I realized how broken support was for our newest CI, Travis.  The CloudFoundry team, which had just moved in-house, was growing frustrated with confusion over which of their (40+?) builds was actually broken vs. broken on a branch vs. just not broken.  With no beach in sight, I carved out some time to push a fix and heard a collective sigh of relief from the CF team.

Shortly after, a hiring surge paired with a project ramp down created a large team of Pivots ready, willing, and able to tackle our backlog.  My colleague, Johnny Mukai, sent on a reflection on how things were going and his concerns for the product’s direction:

It’s felt great to dig into Project Monitor and take care of some traditionally thorny pieces of the app, merge in some long running branches, and revel in the camaraderie of being on a large team again.

As we plow through open issues and Project Monitor transitions from ‘please fix this’ to a ‘what new features shall we implement’ kind of project, I have some reservations.  In my experience, Pivots use Project Monitor for one thing and one thing only and they only want it to do one thing and it had better do that one thing well. (One. Thing.)

The build goes red, instantly I understand that something has run afoul. On a board of builds where only two things happen, it’s obvious that I need to immediately address this. On a busy board with a million things going on — some of them open to subjective interpretation —  it’s easy for me to just glaze over. If it were important, there would obviously be a giant red notice for me, somewhere.

Closer (by Travis) [2]

Click.  At that moment it was clear to me that Project Monitor is a dashboard for engineers and product owners building a product.  Not a devops board, not a metrics showcase, just an indicator that someone needs to act.  Not everyone agrees, but being opinionated is the only way to ensure that a product stays relevant.

We just spent time reimagining the Tracker portion of our status screen, and I’m proud of all the collaboration and feedback the company gave as we broke down the problem, released and refined.[3]  I saw Project Monitor running in a colleague’s office this weekend, and even fielded a support question from a Windows user in Finland!  Stepping into a new product as PM is hard; if you find yourself in that position I encourage you to work on a clear thesis that guides your efforts.

Project Monitor today.  See our repo for more information on getting started.

[1] http://en.wikipedia.org/wiki/Agile_software_development

[2] Travis – Closer (Spotify)

[3] Think, Make, Check!

The post Project Monitor, Product Management, Post Meridian, and Other PM Acronyms appeared first on Pivotal Labs.

I’ve sat in on enough IPMs to realize that they’re all unique snowflakes, but here’s what the agenda for an ideal IPM looks like to me.

PM as Facilitator

The Product Owner/Manager should run an IPM.  It’s his/her job to ensure that everyone involved with the product development process knows where we just came from and where we’re going.  To that end, I like to start by “walking the wall” to look through any stories still in progress from the last sprint, along with any pertinent information about features just finished.  This should be a quick process to jog everyone’s memories and ground them in the work that’s coming up.

Invest in Stories

Starting at the top of the backlog, step through each story; the INVEST mnemonic is useful in confirming that they’re shovel ready.  Talk through the user-facing value for a feature, ensure that any comps, wires, assets, flows, data, etc. are attached to the story.  Confirm that the requester marked is the person accepting the story and clarify any acceptance criteria.  The goal here is to be crystal clear on when a feature is “done done.”

Complexity Check

If it doesn’t already have an estimate, each developer that could work on a story should roll on the points to deliver.  If the implementation is not clear, they should have time to talk through approaches.  That said, their role is to nail down a level of complexity, not pin themselves to a specific technical implementation.  Based on the estimate size or developer feedback, stories can be nominated for merging or splitting up.  If that’s the case, capture the pieces of work as placeholders and update with details post-IPM.  The worst thing you can do in an IPM is not respect everyone’s time during this kind of housekeeping.

Paying Down Debt

A healthy development process will incorporate refactors and tackle technical debt in concert with new user value.  In addition to explicit tech debt chores, PMs and developers should look for opportunities to wrap this work into feature development.  For example, if a story calls for adding a field to an existing form you should consider also cleaning up the logic that delivers form validation errors.  [Giving canned examples of identifying technical debt is hard -- please forgive this one!]

Two Iterations Max

Tracker will chunk stories into iterations based on Velocity.  You should only step through stories until you’ve got two sprints worth of estimated work.  I like to keep the visibility to two weeks to cover for quicker than intended delivery of features, and to limit the IPM to a reasonable amount of upcoming work.  It’s taxing to keep the mental inventory of features in your head; sticking with the short term future focuses everyone on the team around tangible new features.

Block and Promote

If a story is blocked, mark it as such and add a comment with what will unblock it.  If a story won’t reasonably become unblocked during the sprint, I’ll move it to the Icebox to be sure the Backlog only reflects actionable work.  Similarly, this is a good time to call for nominations for bugs/chores/features that should be prioritized out of the Icebox.

Short and Sweet

Once you hit an hour of IPM, developers zone out and business owners get antsy about the emails they’re missing (or worse, they whip out their phones).  You can and should be able to limit sidebar discussions to stay focused on one story at a time.  When you have a large team, it’s especially important to play time cop.  If you don’t have a healthy Backlog and find yourself with a lack of new work, end the meeting.  It’s far better to hold an emergency IPM two days later than to suffer the pain of making up stories in real time!

These are not hard and fast rules, but chances are if you sit in one of my IPMs I’ll focus on these pieces.  What’d I miss?  What parts of an IPM are essential to a successful sprint?

Logistics

• Kill the Icebox and bump up the font size.

• Use the Tracker Header bookmarklet to get some more real estate

• Check team strength and set accordingly in Pivotal Tracker

• Use screen sharing (i.e., join.me) for any remote participants

• Turn off all screens unless they’re in support of the IPM (note taking, clarifying comments in Tracker)

[1] Tracker Header Toggle Bookmarklet

A Javascript Bookmarklet to toggle the Tracker header on and off, giving you more room to view stories. To use it, copy “javascript:["header","controlPanel"].each(Element.toggle);app.layout._resizePanels();”, paste into a new bookmarklet, go to your Tracker project, and toggle the bookmarklet to hide the header.

The post Running an IPM appeared first on Pivotal Labs.

You’ve probably read stories proclaiming how effective A/B testing is for Twitter, 37 Signals and even the Obama campaign.  There’s no shortage of third-party services that boast one-click setup via javascript snippet and claim to deliver a double digit boost to your bottom line.

I talked in my last post about the concept of opportunity cost and it’s with this lens that I view excessive testing and experimentation.  If you are still in growth mode, you’re still figuring out what A is.  There’s too much at stake (and too few developer cycles) to distract yourself with subtle experiments that are ripe for invalidation by small sample sizes, statistical insignificance, and indecision.

“One consequence of this data-driven revolution is that the whole attitude toward writing software, or even imagining it, becomes subtly constrained. A number of developers told me that A/B has probably reduced the number of big, dramatic changes to their products. They now think of wholesale revisions as simply too risky—instead, they want to break every idea up into smaller pieces, with each piece tested and then gradually, tentatively phased into the traffic.” — The A/B Test (Wired)

Sounds like the agile software development process, right?  The difference here is that you gain efficiency and transparency by splitting feature work into atomic units of customer value.  You risk building broken software when you split features into chores that aren’t customer-focused; similarly, you risk building a broken product when you try to subcompose the UX into lots of trivial tests.

I say all this because I’ve employed A/B testing in a couple of startups and we never got our bang for the buck.  At one, we used Optimizely but found the integration points to be lacking[1] when we wanted to focus on anything embedded into our app experience.  Landing pages were easy enough to test but acquisition is only one of your challenges.

We then moved to A/Bingo, a framework written by the amazing Patrick McKenzie [2].  This felt like a framework we could grow into, but we were also moving from server- to client-side functionality and we had to shoehorn the testing payloads into a homegrown api.  The result was way too much time invested into infrastructure and not enough time delivering more customer value.  It still kills me to think about the time we devoted to just getting a great new feature to the starting line.

I then joined a startup that had rolled their own A/B testing for life-cycle and transactional emails.  I didn’t even realize this was going on until we started adding KISSmetrics tracking to the emails.  What was the result of all of this wonderful testing?  It turned out that we weren’t storing any of the results, and had been sending only one variant for the last year.  Whoops!

We did have some success with a feature flipper powered by Rollout.  A feature flipper lets you enable functionality for specific customers or a controlled subset of your audience.  We weren’t using it ambitiously, but it was helpful to have the plumbing in place to deliver new features.  I was eager to give it a try, but any changes we wanted to deploy were largely tested and validated before we started building.  Perhaps we should’ve tried turning off features that we suspected weren’t valuable, but we never got around to it (limited cycles and all).

I look forward to the day I can enthusiastically get behind A/B testing, but until that day I will encourage anyone that asks what else they could do with their time.

Is A/B testing worth it for you?  Do you have any horror stories to share?

[1] Caveat emptor: I haven’t used Optimizely in a few years so I’m not an expert on their current functionality

[2] A/B testing is definitely worth the time to Patrick (because he has found his product/market fit).  I encourage you to read through everything he’s written if you’re building a SaaS app.

The post [Metrics] A/B Testing, Feature Flipping and going too far appeared first on Pivotal Labs.

So let’s say your marketing team advocates implementing auto-login for customers by following links in their email.  Sounds useful, but certainly has some security and privacy concerns.  Surely someone’s built this already?  One quick Google search later reveals a promising start.  You start writing the story and prioritize it in the backlog.

But wait — do you need that feature?  Product management by “wouldn’t it be cool if…” happens, but I wouldn’t recommend it.  Instead, channel your inner-lean and ask how you could validate that this idea has merit.  I.e., what hypothesis would this feature prove, and is there a way to test without building it?

In this case, I’d create an assumption like: “Customers aren’t re-engaging with our site from emails because they can’t remember their sign-in credentials.”  That leads to a hypothesis along the lines: “Customer engagement should increase by including automatic login in all emailed links to our site.”

This thought exercise points us at a conversion to track — how often do customers follow an emailed link and log in?  Luckily we’ve already implemented transactional email funnel metrics, which look something like this:

Hmm.  I see that we’re getting a 33% open rate on transactional emails; potentially good given the type of product.  Of those that view, 26% clicked to follow a link.  30% of those that followed a link weren’t part of the last step (visited site) because our tracking cookie didn’t recognize them; they didn’t log in and hence didn’t stitch their session into this funnel.  These numbers don’t scream for an auto-login feature, do they?  Sure, we’re churning 42 customers between click and login, but that only represents 2.6% of the customers that received emails.

Given that we’ve fit auto-login into the universe of engagement, I would look to prioritize any feature work could deliver a larger lift to this KPI.  Perhaps you could build more targeted emails that drive more customers to open their email?  A 16% bump in opens will have a larger effect than a near-impossible bump on click-to-visit conversions of 30%.

While far from perfect, data-driven product management will force you to evaluate any new feature from the context of increasing customer value on some axis.  In this example we were fortunate to draw upon historical email funnel metrics.  This won’t always be the case, but I encourage you to find cheaper, quicker approaches to [in]validating the merit of an idea before it becomes a feature story.

The post [Metrics] Do you need that feature? appeared first on Pivotal Labs.

The idea behind this email is to give churned customers a quick way to summarize why they’re no longer using your product. After you haven’t seen them for some period of time (a week? two weeks?), you send them an email that looks something like this:

We haven’t seen you at Wibbles in a while, and we figured you’ve moved on. Before you go, do you have a minute to tell us what went wrong?

It didn’t work
It was confusing…
I don’t need it
Something else

Chances are some of these customers won’t even remember why they signed up for Wibbles, but most will. You might even get direct responses asking for help. For those that have a spare 30 seconds, clicking on a link will take them to a “Thanks” landing page (bonus points for a discount or re-engagement tactic). You can build a funnel in KISSmetrics to measure how effective these emails are, and you’ll get something that looks like this:

Here’s where it gets interesting: if you use KISSmetrics’ URL API, you can record all of their answers in the same system that tracked their behavior in your app:

http://wibbles.com/landing?kme=Responded+to+email+survey&kmi=bob%40example.com&km_survey_choice=It+didn’t+work

Now you can revisit some of the core flows in your app, but filter to the lens of “confusing.” Did those customers all get stuck in the same place?  Was there a key step in the process they missed?  Perhaps they all used a certain browser like IE that didn’t render a key call to action button.

For this particular example, I did some homework and realized most people that called the app confusing came from a direct link and never saw our intro/landing page. D’oh! I offered them another free month and a live screen sharing session to show them how things worked. The response was positive, and we salvaged a bunch of customers that would have never converted without this abandonment email.

Perhaps this example makes more sense in the context of a free trial or SaaS app, but hopefully I’ve planted the seed for what you can build as re-engagement and learning tools.  Do you have a tried and true email you use to capture churned users?

The post [Metrics] It’s so hard to say goodbye appeared first on Pivotal Labs.

Today I want to focus on the funnel, a strange analogy for describing websites and conversions.  If you bought a funnel that only delivered 10% of the water you poured in, you’d ask for a refund.  Yet, that’s how we describe the attrition that occurs as users/customers engage with your product.  Funnels are important for two reasons: they force you to think about which interactions are most meaningful, and they help you understand where your assumptions aren’t valid.

At Profitably, we built a financial dashboard for small businesses.  Our historical analysis offering was off the mark, so when the time came to tackle forecasting we wanted to build something more engaging, but also build in a way that allowed us to iterate quickly.

We took inspiration from the wizards in TurboTax to collect the information we needed from a business owner to deliver a useful customer forecast.  Even though we were worried about the amount of information we [thought we] needed, we were encouraged by Twitter’s dramatic results from the introduction of gradual engagement for sign up (+29% conversion).  This flow also enabled us to create a rather linear process from start to finish: no registration step, just enter the name of your company and click Get Started.  Someone that successfully completed the wizard would see a detailed forecast for customer growth…end of the funnel.

We had talked with lots of small businesses before building to understand what they had been doing instead, but you can only get so far with conversations and mockups.  Beyond these qualitative tactics, behavioral analytics gave us the ability to dive into this funnel and come out with hard numbers.  Here’s what the initial KISSmetrics funnel looked like early in the release process:

Man, that is sobering.  I had reached out to every person that went through to get a sense of their business and what problems they were hoping we could solve.  But, to be honest, there was a lot of work staring us in the face just from looking at that [looooong] graph.  Here are my thoughts on each step, and how we reacted:

1. We bounced almost half of the visitors that landed on the first page of our app.  We initially assumed people would see our brochure site first, so the getting started call to action was prominently featured over the value prop and greater app context.  Action: We added more context and treated this page like a potential first landing point for new customers.
2. We lost 25% on revenue streams.  Most people started with one revenue stream their first time through, but didn’t understand they could just hit next to default to that.  Action: We pre-populated that default and made the skip process easier to understand.
3. Another 11%; turns out people don’t know the difference between customer segments  and revenue streams.  Action: We removed this from the default flow, but power users could still get to it.
4. 6% bailed on channels; they knew what this was but got stuck, again on default/skip ambiguity.  Action: Make one channel the default for new customers.
5. Everyone got through the next two steps.  Great!  Action: Ignored any nagging problems with these pages (for the time being).
6. Nearly two-thirds of visitors churned at Topline growth.  Woof.  Action:  Spent tons of time talking to and watching people interact with this page.  Ripped out this page’s innards and made it a lot faster.  Ultimately, this page (in this carnation) didn’t survive.
7. Another 17% couldn’t wait to get to the finish and stopped at conversion rate.  Action: Simplified this to a default, and applied it across all months.  Power users could tweak later if they were so inclined.
8. Bonus Action: Introduced an “Easy Mode” that offered default templates for different businesses, and short-circuited the whole process.

Those are just the micro/optimization-focused thoughts and actions.  After implementing some tactical fixes, we saw the funnel conversion go from 13% to something in the 65% range (5X).  I’ll leave how we improved that funnel on a macro scale and took the product to a much better place for a different post.  For now, I’ll leave you with something I try to remember when thinking about making a product better:

“Product development is about figuring out the single most important problem that exists right now and doing that and only that” (Jason Freedman).

By forcing yourself to identify the funnels that are most important to your product, you’ll force yourself to explain why people are falling out.  You’ll also give yourself a quantitative feedback loop to confirm that new features actually keep people in.

The post [Metrics] Plug your Funnel appeared first on Pivotal Labs.