Chris Bank is the Growth Lead at UXPin, the UX Design App. For more tactics to preserve UX during product design, check out the Guide to UX Design & Process Documentation.
Aside from launching the product, the single most important process a product team uses to ensure success is measuring, learning, and tweaking the product.
Once your working product hits the market, its life becomes a sequence of iterations based on getting the right answers to the right questions. Do customers like my product? What could I add to make them happier to pay more? Are we even on track for long-term profitability?
Qualitative and quantitative data will become the fuel powering your product team as they seek perfection through tweaking. For the most helpful documentation, you’ll need to look at customer satisfaction, suggested feature ideas,bug reports as well as analyze performance metrics for the full sales funnel (everything from acquisition to lifetime value).
These two types of feedback loops combined will give you the most accurate picture of how well (or poorly) your new feature, sub-system or overall product is doing.
Design a customer feedback loop
Beta testing was finished up in the implementation phase. Your product is now in the wild for real testing. User insights brings purpose to the product, so make sure you categorize feedback depending on the stage of the sales cycle to get the fullest understanding.
Consulting firm Conversion XL believes that most company claims of being customer-centric don’t match reality. So, the solution to bridge that gap into actionable insights is to create segmented feedback loops.
The theory behind the feedback loop is that gathering feedback will help you improve the product, thereby reducing future feedback. We’ll explain a few tactics for capturing and documenting feedback from non-customers, leads, and customers.
1. Non-customers
If you’re launching an online product, you don’t want to miss out on visitor feedback. The tricky part is that you don’t have information about them (e.g. email address or phone number) so you’ll need to get the info on the spot.
Survey tools or live touch are great tactics because they guide visitors to the right resources and help you learn what needs improvement. If you’re looking for a more engaging method, remember that it’s never a wrong time to test and that in-person testing with your target demographic can provide the highest quality feedback.
Email marketing software company Vero triggers an on-site survey after 30 seconds of idleness. If you click on the first option, you’ll then be prompted with the below callout to learn more.
This self-serve setup helps visitors with information and gives Vero data to iterate their product and messaging. For example, the second and third choice informs the product team on whether pricing needs to be lowered and what features to add.
Most on-site survey tools also can export data in various formats so this can be easily documented and analyzed at scale to inform product and marketing teams.
2. Leads
Building a feedback loop for your opt-in leads is all about mastering the art of follow up. You have their information, now you just need their thoughts.
As an example, Noah Kagan’s AppSumo released a new online product “How To Make a $1000 A Month Business” which did well during beta testing but only resulted in 30 sales out of an initial email campaign of 30,000.
In response, AppSumo sent the above survey with four simple questions to people who had opened the email, clicked through, but never bought. The company then sorted all the data from those who answered “Yes”, analyzed the reasons, and redesigned the landing page to answer the most pressing objections.
The new landing page iteration ended up earning much higher sales and conversions.
3. Customers
For companies with physical products, most of your feedback will be from customers. Surveys and support ticket tools are more lightweight tactics. More complex options could include conducting exploratory interviews and adding in-app feedback that prompts customers differently depending on their actions.
While these solutions might be more expensive, they let you proactively hear and see what customers are doing versus relying on them to initiate responses.
If you’re going lightweight, Jason Fried, Founder of Basecamp, recommends that the language you choose for surveys needs to be modest and to-the-point. The below survey that Audi sent Jason a few days after purchasing is the perfect reminder to be realistic with the responses.
Looking at a dealer’s inventory isn’t an experience that’s likely ever going to register as “Truly Exceptional”, so the copy could be revised accordingly. The tactic, however, is sound.
Sherif Mansour, Senior Product Manager at Atlassian, emphasizes that customer feedback loops must be frictionless. For a lightweight tactic like support tickets, this means automatically populating data where possible and requiring minimal fields.
To test this, he implemented the below “Got Feedback?” button into a version of Confluence. Upon clicking the button, users were prompted with a simple form that only required the Subject be filled out.
The result? After release, he received over 900+ feedback submissions, the most at the time for Atlassian. If you’re developing a software product, you can integrate bug tracking into JIRA so that customer tickets are sent from your software straight to customer support. From there, the bugs can be prioritized accordingly and handled by development, completing your feedback loop.
Focus on the right metrics for the right insights
As discussed in the Guide to Minimum Viable Products, metrics and customer feedback act as checks and balances for validating assumptions. The dynamic between the two can be quite complex — while you shouldn’t follow a linear path of just fixing what’s broken and adding what customers request, you also don’t want to miss out on “earning without learning” by looking only at the bottom line.
It’s not always as simple as “if I change feature X, then metric Y will improve”, so it’s important to follow a process informed by data rather than obsessed with it.
Below, we’ll explain how setting and measuring your success metrics, prioritizing iterations based on ROI, and striving for your local product maximum can all help prevent you from wasting time and money on the wrong tweaks.
1. Determine your customer and business success metrics
When thinking about product metrics, it can be tempting to turn to numbers related to bugs resolved, total site visits, or uptick in brand mentions following launch. These are, after all, easily measured and useful to a certain extent. But they shouldn’t be the main measure of success — success needs to be viewed from a much more actionable perspective.
All companies, regardless of industry, need to focus on success metrics comprised of customer satisfaction and business sustainability.
As you’ll see summarized below, customer success metrics tell you if users get value out of your product while business success metrics look at your own business outcomes.
Some customer success metrics include:
- Activation percentage — How many customers completed a key action during the conversion process? This would include account, email, and blog signups.
- Retention percentage — Are enough customers coming back to get value from your product? This is based on your assumption of how often a satisfied customer uses your product.
- Net promoter score — Tying back to customer feedback, the NPS is a universal method of quantifying customer satisfaction. A score of at least 50 is a good sign of delighting early adopters.
Some business success metrics include:
- Customer Acquisition Cost — What does it cost to secure a paying customer?
- Percent Conversion to Paying Customers — Is your current price enticing customers?
- Lifetime Value — How much will a customer pay over time?
Ryan Hoover, Founder of Product Hunt, describes that his success metrics shortlist consists of revenue, retention, engagement, virality, and operating efficiency. Revenue is measured by lifetime value.
Retention is calculated by dividing monthly active users by daily active users. Engagement equates to actions divided by pageviews while virality equals the K-factor. Finally, he also verifies product quality by measuring operational efficiency (support emails per week).
Customer and business success metrics are especially relevant to the highly saturated mobile app market. Mashable believes there is no one-size-fits-all perfect KPI but there are a set of metrics to paint the picture of reality.
In addition to the previously mentioned metrics, Mashable also lists measuring app launch and load time as an indication of customer success since long load times lead to poor retention. To quantitatively measure customer satisfaction, you could use in-app messaging to push surveys that gauge feedback on a scale of 1 to 10. Then, you can aggregate the data to calculate the NPS.
Regardless of your business, the goal is to build a dashboard that reflects both customer and business success metrics.
For example, this comprehensive KPI spreadsheet for SaaS companies looks at everything from activation percentage to lifetime value. Once you have the right metrics, you can start iterating intelligently.
2. Prioritize iterations based on ROI
With a dashboard in place, you can start testing the business effects of your iterations.
For example, you may have added a social login button to your app based on customer feedback requesting better accessibility. After a few weeks, however, you see no improvement in time spent in the app or user retention — a signal that perhaps it’s a more complex problem requiring more than just one iteration.
Therefore, the beauty of setting and tracking metrics is to better validate your reaction to customer feedback.
But just racking up a product “To-Do” list, isn’t enough. Dan Olsen, former Group Product Manager at Intuit, believes you need to prioritize your iterations based on metrics you want to drive versus required resources.
Decide what features to change by weighing out current user satisfaction, required resources, and the potential benefits. Then you can plot your features in a prioritization matrix below.
At Pandora, the product team acquired more than 70 million active monthly users with just 40 engineers by sticking to a strict process of generating, culling, and prioritizing feature ideas. Team members would generate 60 to 80 ideas and then defend each idea by explaining the specific success metrics and time required.
Scope was then expressed in dollar amounts. Features requiring one engineer one month to complete were worth $5 and increased based on monthly time. The team was then given a budget of roughly $150, which usually resulted in only the most crucial 20 features being built every 90 days.
Once you’ve prioritized your iterations, you can track them with a cumulative flow diagram. By using a cumulative flow spreadsheet to track features that were being developed, tested, and released, Siemens Health Services actually reduced its median iteration cycle by 21 percent.
3. Get to the local maximum but keep a higher peak in sight
The end goal of data-driven iteration is to get your product to a state of diminishing returns. At this point, the product is already great and the cost versus benefits of new features becomes less and less attractive.
The local maximum is the point at which the product team will need to determine if they’re satisfied or if a higher peak of quality is in sight. If the product prioritization matrix justifies going after a higher peak, then the team returns to the Product Definition Stage.
For example, Spotify’s website was iterated for four years before returning to the Definition stage for a complete revamping in summer of 2012. Afterwards, the new website reflected the Spotify vision in a dramatically more effective way.
Rian van der Merwe, former Senior Product Manager at eBay, explains the key to getting to this higher peak is understanding the difference between variation and iteration. When you can’t move any ideas forward, you’ve hit the local maximum and it’s time to switch gears from iteration to variation.
Go back to the drawing board and consider all possible ideas again. If the local maximum is the best house in the neighborhood, then the global maximum is the best neighborhood on the planet.
The product circle of life never ends
The goal of listening to customers and measuring your success (or failures) is so that your product can one day achieve its global maximum — a rare state in which your customers are thrilled and there’s no improvements possible.
The process of getting there, however, can be long and full of bumps. That’s why iteration is the heart of the product design process— it evolves your product based on the latest market signals. So while iteration can feel like an eternity, it’s still the shortest path to success. Keep going, and when you’re ready to replicate your successes, don’t be afraid to start back at square one.
For more advice on the process before and after product launch, check out the Guide to UX Design Process & Documentation.
Read next: Product research 101: How to do your research before diving into design
Top image credit: Business on a Shoestring
Get the TNW newsletter
Get the most important tech news in your inbox each week.