Measuring the efficacy of product roadmap

Product roadmap is one of the most critical elements that determine the long term success of the product and if it cannot be measured, it cannot be improved. To measure anything, we need to set goals. I have already outlined that product roadmap is a plan of execution to accomplish product objectives. There would be a timeframe to accomplish the product objectives and Product Manager has to periodically assess how product roadmap is contributing to the realization of product objectives. Product objectives could be targets related to either revenue, growth etc.

Understanding customer needs and addressing the needs that are valued most by customers is a means to achieving product objectives but it is not the only means. I always maintained that merit of the product contributes utmost 50% (or little higher but definitely not 100%) to a successful sale. In any selling process, customer evaluates the product against certain set of attributes. The attributes include both product and non-product; both of them jointly influence the selling process. For instance, in case of restaurant where food is considered a product, the ambience and the location of the restaurant is as important as the taste of the food. In case of B2C, marketing play a critical role in communicating the value of the product and establishing an emotional connect. In case of B2B, the existence of reliable support system, brand, and distribution network are other important factors that complement the core product. So establishing a direct causal and effect relation between product roadmap and product objectives is not foolproof. Therefore in addition to using product objectives as a means to measure efficacy of product roadmap, I am also advocating the use of feature usage metric.

During prioritization of product requirements, I indicated about the need to pick right product attributes that are valued most by customer and assign appropriate weights. While measuring the efficacy of product roadmap, the single most important criteria is to evaluate whether Product Manager has picked the right product attributes that will influence the selling process. Two methodologies indicated earlier will be used to measure the efficacy of product roadmap and to provide reinforcing feedback back into product requirements prioritization process, so right product attributes could be used to prioritize requirements that are valued most by customers.

Product objectives (aka goal) metrics

The purpose is not to validate whether we have accomplished the objectives, but to ascertain the progress and identify whether the product is on the right track to accomplish the objectives. If the product is indeed on the right track to accomplishing the objectives, Product Manager can presume that roadmap has played it role because existence of both product and non-product attributes is mandatory to a successful sale. Even so, Product Manager has to understand whether customers are buying the product for the same reasons that Product Manager has envisaged. If customers are buying the product for its reliability and price while Product Manager assuming that customers are valuing the product for its design and usability might not be a nice situation because any further prioritization decisions will be focused more on design and usability instead of reliability. Each customer will have their own reasons and no two customers are alike. Yet, Product Manager has to wisely choose a sample of customers and understand the common patterns that will influence their buying behavior.

Similar exercise needs to be done if the product is far from realizing the product objectives as well. Product Manager should always identify the reasons behind success and failure. Product Manager has to analyze the losses to determine whether there is lack of proper understanding of the product attributes that are valued most by customers and reinforce the feedback back into the product prioritization process to evolve the product with right set of requirements.

Measuring product objectives

Feature usage metrics

The idea of feature usage metrics is to understand whether top prioritized features are most used by customers. If not, something might be terribly going wrong with the product prioritization process. The primary premise of the exercise is that customers use features that are really useful for them. They do not use features just because they are available on the product. Not every product would have a direct mechanism to understand on how many customers are using each feature, in such scenario I normally take the help of support system. Please take a look at my earlier blog post to understand how support data can be gleaned to derive feature usage metrics. Feature usage metrics will reflect the efficiency of product requirements prioritization process. Product Manager can use those metrics to revisit the prioritization process whether (s)he is choosing right attributes (product attributes are already defined in ‘Prioritizing product requirements’ blog) to identify requirements that are valued most by customers. In this methodology, the underlying assumption is that the customer is aware of all the features available in the product and lack of usage of any features is not due to unawareness.

Gauge the mood – Measure the perception

I would not call it as a methodology as it is more subjective and in general it does not indicate any thing specifically about the prioritization process and effectiveness of the chosen attributes. This methodology is to measure the perception of everyone (primarily customers) especially when the business model supports sharing of product roadmap. When Product Manager shares the roadmap with customers and elaborates about what will be made available in the product over the next couple of years, Product Manager really needs to understand the reaction of customer. Do customers get excited and enquire eagerly of what is being made available or just don’t react or express anguish over the inability of product roadmap to match their business requirements. As I have been stating all along, no two customers are alike, so look for generalized reaction of broader set of customers.

Perception of customers reflects in their actions and therefore measuring the perception is critical to make customers invest in the product. Perception sometimes contradicts the ground reality and if it does so in a negative way, nothing could be more disastrous. The ground reality might be that the product is at the fore-front of innovation and perfectly evolving to meet the business challenges of customers. If customer(s) perception about the product is contradictory and it would obviously reflects in their decision to invest on the product. Customer visits, escalations are quite a few ways to measure the perception and same could be achieved through product roadmap presentations as well.

Promising product roadmap creates positive vibes among internal stakeholders too (more importantly engineering and sales team). Present the roadmap to them as well. I have earlier talked about leveraging the expertise and experience of sales and engineering team to discover customer needs, so it is only appropriate to share the roadmap throwing signals that preparing roadmap is an inclusive approach. Take their feedback and gauge their perception as well, product roadmap should motivate engineering team to build better product and sales team to sell more. Internal stakeholders should be excited about the product roadmap and should be convinced about the direction in which product is heading.