A user metric is a terrible thing to waste

Monday, April 29, 2013

When it comes to user metrics, what should you log when you don’t know what to log?

Everything.

When you’re first getting an MVP site off the ground, you don’t know quite what you might want to track. And might not have time to think about it.

If you log everything your users do, however, then weeks or months of data will be there for you when you need it most, to answer your questions right away. That’s why I’m excited about companies like Heap are doing, logging every single user click.

For those without Heap invites, or if you’d prefer to use Kissmetrics, I wrote a simple gem to do the same with KissMetrics, for all your user clicks that hit your web server: km_everything. It logs all controller actions by default, but you can set up a whitelist to give them more meaningful and product-manager friendly names, or a ‘blacklist’ to prevent certain actions from being logged that aren’t meaningful and would use too much of your KissMetrics event quota.

Also, from what I’ve observed on KissMetrics, server-side metrics have proven more reliable. Metrics recorded via Javascript can be off if the user closes their browser too quickly after taking an action. Logging the metrics server-side also won’t take up any of the user’s processing power or show signs that remind them they’re being tracked.

Excited about our brave new world of meaningful ad hoc analytics with reams of user data, powered by ubiquitous metrics. This data will then show us how to make our sites even better for our users.

[Metrics] A/B Testing, Feature Flipping and going too far

Graham Siener

Sunday, March 17, 2013

A/B testing is probably not worth your time. When you start hooking metrics up to your product, the feedback is addictive. All of a sudden you’ve got lots of actionable data and you’re tacking validation goals onto feature stories. This is great, but I implore you to not take it too far.

You’ve probably read stories proclaiming how effective A/B testing is for Twitter, 37 Signals and even the Obama campaign. There’s no shortage of third-party services that boast one-click setup via javascript snippet and claim to deliver a double digit boost to your bottom line.

I talked in my last post about the concept of opportunity cost and it’s with this lens that I view excessive testing and experimentation. If you are still in growth mode, you’re still figuring out what A is. There’s too much at stake (and too few developer cycles) to distract yourself with subtle experiments that are ripe for invalidation by small sample sizes, statistical insignificance, and indecision.

“One consequence of this data-driven revolution is that the whole attitude toward writing software, or even imagining it, becomes subtly constrained. A number of developers told me that A/B has probably reduced the number of big, dramatic changes to their products. They now think of wholesale revisions as simply too risky—instead, they want to break every idea up into smaller pieces, with each piece tested and then gradually, tentatively phased into the traffic.” — The A/B Test (Wired)

Sounds like the agile software development process, right? The difference here is that you gain efficiency and transparency by splitting feature work into atomic units of customer value. You risk building broken software when you split features into chores that aren’t customer-focused; similarly, you risk building a broken product when you try to subcompose the UX into lots of trivial tests.

I say all this because I’ve employed A/B testing in a couple of startups and we never got our bang for the buck. At one, we used Optimizely but found the integration points to be lacking[1] when we wanted to focus on anything embedded into our app experience. Landing pages were easy enough to test but acquisition is only one of your challenges.

We then moved to A/Bingo, a framework written by the amazing Patrick McKenzie [2]. This felt like a framework we could grow into, but we were also moving from server- to client-side functionality and we had to shoehorn the testing payloads into a homegrown api. The result was way too much time invested into infrastructure and not enough time delivering more customer value. It still kills me to think about the time we devoted to just getting a great new feature to the starting line.

I then joined a startup that had rolled their own A/B testing for life-cycle and transactional emails. I didn’t even realize this was going on until we started adding KISSmetrics tracking to the emails. What was the result of all of this wonderful testing? It turned out that we weren’t storing any of the results, and had been sending only one variant for the last year. Whoops!

We did have some success with a feature flipper powered by Rollout. A feature flipper lets you enable functionality for specific customers or a controlled subset of your audience. We weren’t using it ambitiously, but it was helpful to have the plumbing in place to deliver new features. I was eager to give it a try, but any changes we wanted to deploy were largely tested and validated before we started building. Perhaps we should’ve tried turning off features that we suspected weren’t valuable, but we never got around to it (limited cycles and all).

I look forward to the day I can enthusiastically get behind A/B testing, but until that day I will encourage anyone that asks what else they could do with their time.

Is A/B testing worth it for you? Do you have any horror stories to share?

[1] Caveat emptor: I haven’t used Optimizely in a few years so I’m not an expert on their current functionality

[2] A/B testing is definitely worth the time to Patrick (because he has found his product/market fit). I encourage you to read through everything he’s written if you’re building a SaaS app.

[Metrics] Do you need that feature?

Graham Siener

Monday, February 25, 2013

The hardest part of product management is deciding which features NOT to build, especially when they seem like great ideas. When you have a product in the market (or are getting it out there), you want to spend your engineering team’s time delivering value to your customers. With this logic in mind, building anything has a cost associated with it (and more importantly, an opportunity cost).

So let’s say your marketing team advocates implementing auto-login for customers by following links in their email. Sounds useful, but certainly has some security and privacy concerns. Surely someone’s built this already? One quick Google search later reveals a promising start. You start writing the story and prioritize it in the backlog.

But wait — do you need that feature? Product management by “wouldn’t it be cool if…” happens, but I wouldn’t recommend it. Instead, channel your inner-lean and ask how you could validate that this idea has merit. I.e., what hypothesis would this feature prove, and is there a way to test without building it?

In this case, I’d create an assumption like: “Customers aren’t re-engaging with our site from emails because they can’t remember their sign-in credentials.” That leads to a hypothesis along the lines: “Customer engagement should increase by including automatic login in all emailed links to our site.”

This thought exercise points us at a conversion to track — how often do customers follow an emailed link and log in? Luckily we’ve already implemented transactional email funnel metrics, which look something like this:

Hmm. I see that we’re getting a 33% open rate on transactional emails; potentially good given the type of product. Of those that view, 26% clicked to follow a link. 30% of those that followed a link weren’t part of the last step (visited site) because our tracking cookie didn’t recognize them; they didn’t log in and hence didn’t stitch their session into this funnel. These numbers don’t scream for an auto-login feature, do they? Sure, we’re churning 42 customers between click and login, but that only represents 2.6% of the customers that received emails.

Given that we’ve fit auto-login into the universe of engagement, I would look to prioritize any feature work could deliver a larger lift to this KPI. Perhaps you could build more targeted emails that drive more customers to open their email? A 16% bump in opens will have a larger effect than a near-impossible bump on click-to-visit conversions of 30%.

While far from perfect, data-driven product management will force you to evaluate any new feature from the context of increasing customer value on some axis. In this example we were fortunate to draw upon historical email funnel metrics. This won’t always be the case, but I encourage you to find cheaper, quicker approaches to [in]validating the merit of an idea before it becomes a feature story.

Always be validating

Robbie Clutton

Monday, February 18, 2013

Big companies with well established products and business practices often differ from smaller, younger, start-up companies with the cash flow available to prove business ideas. The startup companies will often have a “runway” or “burn-rate”, meaning they have a limited amount of cash to keep the lights on. Big companies have budgets, committees and risk departments. Those companies may take a risk on a new product, but if it fails the lights will stay on and people will more than likely still have a job. Established companies will also invest in existing products to strengthen their market position which may result in growth targets rather than a focusing on validating those targets.

Too big to fail

New products should always look for market validation regardless of the size of the company. Just look at how long the HP Touchpad (48 days) and the Microsoft Kin phone (49 days) lasted in the marketplace before being withdrawn. It’s reported that HP would go onto lose up to $300m after cutting the Touchpad and associated webOS product line along with job cuts. So much for market validation and their risk department. Another huge loss was RIMs Playbook, reported to have lost the company $1.5bn in a period when the company really needed a boost in the marketplace amid falling Blackberry sales.

Not since the dot com boom have startup companies had anywhere near that sort of money to take a product to market and while these examples are hardware based incurring the associated research and development costs there is a lesson about validation in there regardless of the size of the company. Software companies seem to fail in three categories: massive dot com blowouts (e.g. WebVan and pets.com); being squashed by a competitor (Friendster to MySpace to Facebook); and the ones you’ve never heard of as they failed before they made it.

That’s not to say that the companies that are using metrics and data to help make their decisions don’t push the process too far. The example of testing 40 shades of blue at Google is probably the best known. However, a little information can go a long way.

Agile and counting

How should a company with a limited cash flow proceed? That company needs to know that every feature built is going towards a larger goal, is required and ultimated validated by the users. How does a company know when a feature has been validated?

Each new feature should have some validation criteria, e.g. “Removing password confirmation should result in decreased number of users getting stuck and leaving at the registration stage”. This then needs to be measured, which means we need to have that measurement before the feature is written ideally.

There are a number of tools available to start gather metrics, Google Analytics is good for general visitor information but tools like KissMetrics have gathered support for more of data driven demands. There are also open source alternatives such as statsd and Cube if the project has the time to invest in their own tools. If the hypotheses are more visually driven, tools such as CrazyEgg have proved useful for validating calls to action and discoverability within an application.

As an example, something as simple as CrazyEgg can show where users of an application get confused. On a recent project we deployed CrazyEgg on the homepage and almost immediately we could see that users were clicking on headings like they were links. You could almost sense their frustration through the heatmap when a click didn’t change the page.

On another project we could see we were dropping a high number of users through a certain part of the application funnel. At this point we did some user testing and tried to focus on this area where users were dropping off. We found that the application, which had a “real-time” price for the service the user was getting a quote built in some assumptions for earlier selected defaults but without telling the user that had happened the users were getting put off by the high quote. We could hypothesise on what might increase the throughput of the funnel, make those changes and see which one worked best.

Tools and talent

There are some tools to help follow a feature through to validation. Tools such as Lean Launch Lab or more general tools such as Trello can help monitor the success of features and their validation.

So you have the tools, you have the talent. Now what? You need to know where the application currently stands with the metrics which drive the business. When a new feature is written it should include a hypothesis about what metrics will change when the metrics goes live. The feature gets built and goes live. How do you follow what happens next? Most tools and processes stop here, features have been accepted and is live, what more do we need? Those features should be monitored while the product team measures the effectiveness of the change. Did that feature make the hypothesised impact? If not, should the feature be re-evaluated or thrown out?

Step one to a successful business: know the product, understand the users and always be validating.

[Metrics] It’s so hard to say goodbye

Graham Siener

Monday, February 11, 2013

When you release early and often, you’re bound to churn through new customers. I showed you in my last post how to measure the funnel of trial to activation to engagement, but what should you do with all the people that didn’t make it? Today I’ll share a technique that’s a nice complement to a more traditional exit interview — I call it the Abandonment Email.

The idea behind this email is to give churned customers a quick way to summarize why they’re no longer using your product. After you haven’t seen them for some period of time (a week? two weeks?), you send them an email that looks something like this:

We haven’t seen you at Wibbles in a while, and we figured you’ve moved on. Before you go, do you have a minute to tell us what went wrong?
It didn’t work
It was confusing…
I don’t need it
Something else

Chances are some of these customers won’t even remember why they signed up for Wibbles, but most will. You might even get direct responses asking for help. For those that have a spare 30 seconds, clicking on a link will take them to a “Thanks” landing page (bonus points for a discount or re-engagement tactic). You can build a funnel in KISSmetrics to measure how effective these emails are, and you’ll get something that looks like this:

Here’s where it gets interesting: if you use KISSmetrics’ URL API, you can record all of their answers in the same system that tracked their behavior in your app:

http://wibbles.com/landing?kme=Responded+to+email+survey&kmi=bob%40example.com&km_survey_choice=It+didn’t+work

Now you can revisit some of the core flows in your app, but filter to the lens of “confusing.” Did those customers all get stuck in the same place? Was there a key step in the process they missed? Perhaps they all used a certain browser like IE that didn’t render a key call to action button.

For this particular example, I did some homework and realized most people that called the app confusing came from a direct link and never saw our intro/landing page. D’oh! I offered them another free month and a live screen sharing session to show them how things worked. The response was positive, and we salvaged a bunch of customers that would have never converted without this abandonment email.

Perhaps this example makes more sense in the context of a free trial or SaaS app, but hopefully I’ve planted the seed for what you can build as re-engagement and learning tools. Do you have a tried and true email you use to capture churned users?

[Metrics] Plug your Funnel

Graham Siener

Monday, January 28, 2013

Once you’ve launched a product, “metrics” can be a powerful tool in measuring engagement and product issues. From the engineering side, devs have a hard time believing if a metric is valid or misleading. From the biz dev/marketing side, it’s hard to understand the technology enough to know what kinds of questions can be asked. I’ve gone up the learning curve on implementing, understanding, and actioning metrics and have noticed that people have a hard time getting started. This post is the first in a series on metrics, and I hope they give you the kick you need to get started.

Today I want to focus on the funnel, a strange analogy for describing websites and conversions. If you bought a funnel that only delivered 10% of the water you poured in, you’d ask for a refund. Yet, that’s how we describe the attrition that occurs as users/customers engage with your product. Funnels are important for two reasons: they force you to think about which interactions are most meaningful, and they help you understand where your assumptions aren’t valid.

At Profitably, we built a financial dashboard for small businesses. Our historical analysis offering was off the mark, so when the time came to tackle forecasting we wanted to build something more engaging, but also build in a way that allowed us to iterate quickly.

We took inspiration from the wizards in TurboTax to collect the information we needed from a business owner to deliver a useful customer forecast. Even though we were worried about the amount of information we [thought we] needed, we were encouraged by Twitter’s dramatic results from the introduction of gradual engagement for sign up (+29% conversion). This flow also enabled us to create a rather linear process from start to finish: no registration step, just enter the name of your company and click Get Started. Someone that successfully completed the wizard would see a detailed forecast for customer growth…end of the funnel.

We had talked with lots of small businesses before building to understand what they had been doing instead, but you can only get so far with conversations and mockups. Beyond these qualitative tactics, behavioral analytics gave us the ability to dive into this funnel and come out with hard numbers. Here’s what the initial KISSmetrics funnel looked like early in the release process:

Man, that is sobering. I had reached out to every person that went through to get a sense of their business and what problems they were hoping we could solve. But, to be honest, there was a lot of work staring us in the face just from looking at that [looooong] graph. Here are my thoughts on each step, and how we reacted:

We bounced almost half of the visitors that landed on the first page of our app. We initially assumed people would see our brochure site first, so the getting started call to action was prominently featured over the value prop and greater app context. Action: We added more context and treated this page like a potential first landing point for new customers.
We lost 25% on revenue streams. Most people started with one revenue stream their first time through, but didn’t understand they could just hit next to default to that. Action: We pre-populated that default and made the skip process easier to understand.
Another 11%; turns out people don’t know the difference between customer segments and revenue streams. Action: We removed this from the default flow, but power users could still get to it.
6% bailed on channels; they knew what this was but got stuck, again on default/skip ambiguity. Action: Make one channel the default for new customers.
Everyone got through the next two steps. Great! Action: Ignored any nagging problems with these pages (for the time being).
Nearly two-thirds of visitors churned at Topline growth. Woof. Action: Spent tons of time talking to and watching people interact with this page. Ripped out this page’s innards and made it a lot faster. Ultimately, this page (in this carnation) didn’t survive.
Another 17% couldn’t wait to get to the finish and stopped at conversion rate. Action: Simplified this to a default, and applied it across all months. Power users could tweak later if they were so inclined.
Bonus Action: Introduced an “Easy Mode” that offered default templates for different businesses, and short-circuited the whole process.

Those are just the micro/optimization-focused thoughts and actions. After implementing some tactical fixes, we saw the funnel conversion go from 13% to something in the 65% range (5X). I’ll leave how we improved that funnel on a macro scale and took the product to a much better place for a different post. For now, I’ll leave you with something I try to remember when thinking about making a product better:

“Product development is about figuring out the single most important problem that exists right now and doing that and only that” (Jason Freedman).

By forcing yourself to identify the funnels that are most important to your product, you’ll force yourself to explain why people are falling out. You’ll also give yourself a quantitative feedback loop to confirm that new features actually keep people in.