Effect of Product Changes

There has been an explosion in the development of new products in recent years with the rise of experimental product iteration. When a product changes, a reaction almost always follows causing a shift in the key metrics. Therefore, it is critical to understand the impact of shipping a product and ensure that the right outcome is achieved.

For example, in July 2014, when Facebook began notifying users that they would no longer have the option of messaging within its core mobile app, nearly 20 million iOS users in the United States downloaded the separate Messenger app within a one-month period (see Figure 1).

As another example, consider the metrics that might be influenced by changes to a product’s notification algorithm: the number of people who see notifications, the percentage who act on notifications, the percentage of those who end up on the landing page, and the percentage who then actively interact with the product. All of these metrics and more may be affected when a product changes.

Of course, product changes aren’t always intentional. Bugs, such as those that prevent users from downloading your app, updating it, sending messages or receiving notifications, can significantly influence key metrics. Therefore, carefully tracking your key metrics — particularly after a major product change or app store update — can help you detect, understand and limit the damage bugs cause.

The best way to evaluate the impact of a change to your product is A/B testing. In A/B testing, two or more variations are shown at random to users and statistical analysis is used to determine which variation performs better against a given goal. It is especially important to use A/B testing when a change is relatively small, which is most often the case: A large change, such as the Facebook example above, can sometimes be ‘eyeballed’. Usually, though, it is nearly impossible to effectively detect and measure impact without an A/B test.

A product change — whether a new product rollout, a changing notification strategy or an unintended bug — will invariably result in a shift in metrics.

This work is a product of Sequoia Capital’s Data Science team. Jamie Cuffe, Avanika Narayan, Chandra Narayanan, Hem Wadhar and Jenny Wang contributed to this post. Please email data-science@sequoiacap.com with questions, comments and other feedback.

Related posts:

More and more often I stumble on the Internet on the popular strategy of “pivot point” from the site investing. I do not know why this strategy is so popular, but since it does, I decided to try to…

HPB is a new blockchain plateform, that is very easy to use. High Performance Blockchain is aiming to extend the performance of distributed applications to meet real top world business requirements…