Cross pollination of agile and waterfall

Withstanding all the hype surrounding agile, we did follow waterfall model while we built the new product. Agile does not work for all kinds of products even though waterfall model had its own flaws of not allowing the possibility to modify the requirements after entering development phase. So in order to introduce agility within waterfall model, we integrated both agile and waterfall. This blog talks about our experiences of experimenting with a new methodology combining both agility and waterfall. The below excerpts are from my eBook – ‘Building New Product – Experiences of a Product Manager

Our traditional approach to product development has always been the waterfall model. Nevertheless, while we embarked on our journey to build the new product we built the product upon the foundation of waterfall methodology while introducing agility within it. The ability or willingness to experiment with processes or methodologies should be fundamental for any new product development. The experimentation of processes or methodologies can occur when we know why we are doing what we are doing. There should always be a fit between the product and its development methodology, alike product-market fit. If we follow herd mentality and religiously follow a methodology without realizing why we are doing what we are doing, there will be hardly any scope for candid introspection of whether the product and its development methodology have the right fit.

Agile - WaterfallWe never consciously discussed altering waterfall and introducing agility within it. Program Manager naturally adapted the waterfall model to accommodate agility while we had several assumptions to validate, unknowns to eliminate and risks to mitigate. During the course of product development phase. There were never any serious discussions to brainstorm how to integrate both agile and waterfall development methodologies. The objectives and constraints of product development phase naturally triggered our Program Manager to combine them. Real kudos to our Program Manager to introduce agility within waterfall model. It is easy to follow stage gate process to ensure that we complete one stage and move to other. However, while we can only partially complete a stage and move to next stage. Later reverting to earlier stages based on the outcome of a current stage, it is extremely difficult to track dependencies across stages. Let me brief what we did.

For an enterprise product, the architecture of the new product is crucial. I can split the product into two blocks. Architecture block and product functionality block. Redesigning the architecture of the product is always costly and it is as good as developing a new product. Further, the choice of an HW is heavily dependent on the architecture and choice of product architecture is dependent on the HW. Both are mutually dependent. There is a necessity to ensure that HW and architecture can allow the software (i.e. functionality block) to scale as needs of customers evolve with little or no change to both the elements. Why are we putting so much emphasis on product architecture? I once came across a tweet that states every scale has an expiry date. True to those words, software has to scale in future as customer needs evolve and product architecture should support such scale requirements in future with little or no changes to product architecture. Certain elements of product architecture are rigid and it is not possible to modify it as new needs arise. Therefore, it becomes crucial to building a product architecture with a hindsight of how customer needs evolve in future. I did vehemently establish the fact several times in this eBook that the new product should cater to needs of tomorrow and not just for needs of today. Building MVP is not viable to identify needs of tomorrow while it is a viable option for identifying needs of today. Developing customer insights along with identification of factors that can influence the evolution of technology, the evolution of the market, the evolution of customer needs are crucial for anticipating probable needs of tomorrow. Clearly, for enterprise products where architecture can play a big role in determining the success of the new product and where the design and development of architecture happen upfront, agile is counterproductive. Incrementally building product architecture is a grave mistake. However, Program Manager introduced agility in building functionality blocks that address actual customer needs.

During requirements phase, we did outline all assumptions, unknowns, and dependencies across requirements of both product architecture and product functionality. The first priority was to eliminate all unknowns and validate all assumptions surrounding the design of product architecture. The design was mere theoretical until we start building it. Therefore, there will be many assumptions primarily around the ability of product architecture to meet scale requirements. To validate all those assumptions and eliminate unknowns, we constructed hypotheses. On those lines, we also constructed hypotheses surrounding product functional requirements. The product development happens in an interval of 6-8 weeks called DTHOs. Our 1st priority was to eliminate unknowns and validate all assumptions related to product architecture in earlier DTHOs. Dependent products requirements were refined based on the outcomes of those DTHOs.

 

Hypotheses of Product Architecture
Hypotheses Associated Requirements Dependent Requirements
H1 R1 R11
H2 R2 R12
H3 R3 R13
H4 R4 NA
H5 R5 NA

 

Hypotheses of Product Functionality
Hypotheses Associated Requirements Dependent Requirements
H6 R6 R14
H7 R7 NA
H8 R8 R15
H9 R9 R16
H10 R10 NA

The list of hypotheses surrounding both product architecture and product functionality has provided an indication to Development Manager and Program Manager on two aspects. First, the set of functionalities or requirements (R1…R10) that engineering team should implement during the initial DTHOs for validating hypotheses. Second, the set of functionalities or requirements (R11…R16) that engineering team should delay until validation of hypotheses (H1…H10). Delayed functionalities or requirement are later implemented depending upon pivot or preserve scenario in accordance with the outcome of validating each hypothesis.  PRD has clearly articulated the list of incomplete requirements. As initial DTHOs validated hypotheses (H1…H10) through implementing requirements (R1…R10), I refined all dependent requirements (R11…R16) implemented in subsequent DTHOs. The dependent requirements are mostly functional requirements that are refined after validation of hypotheses related to both the requirements of product functionality and product architecture.

One example of a hypothesis is – Ability of the product to meet throughput scale requirements (Hypotheses – H1). The PRD had a clear requirement for the product to deliver 60 Gbps at an average packet size of 512 bytes while maintaining low latency. The conventional wisdom is to measure and report maximum throughput at an average packet size of 512 bytes. Engineering team did validate the hypothesis by building the entire product but performing sizing through simulation. We identified during hypothesis validation that 60 Gbps is achievable albeit with a higher packet size. As I had indicated earlier, marketing team later did lots of analysis, found out the average packet sizes in customer networks is much higher, and we could conveniently defy the conventional wisdom. We never froze entire requirements. Depending on the hypotheses validation, dependent requirements were refined during the course of the new product development introducing agility within waterfall model.

The biggest criticism of waterfall model is that it always heads in one direction. However, we did introduce agility to shift our thinking back and forth between development, design, and requirements. Requirements (R11…R16) were kept open and refined after corresponding hypotheses are validated. We clearly introduced a loop containing product requirements, product design, and product development until we validated all the hypotheses. However, following a strict waterfall model, we had to build a foundation for product architecture upfront unlike in agile before going in a loop containing requirements, design, and development of product functionality. Such model is required for enterprise software products until we find out a methodology for modularly building product architectures for complex systems. Modular architecture should eliminate rigidness facilitating incremental additions to product architecture without the need for redesigning the entire system.

I really feel bad that some of the Product Managers hide their inefficiencies under the veil of methodologies such as Agile, MVP etc. Those are excellent methodologies drafted with good intentions. Nevertheless, those methodologies are more abused than used. MVP formulated by Eric Ries is an excellent way to incrementally build the product through an incremental process of build, measure and learn. Does it mean that a Product Manager should learn every aspect of product requirements through MVP? When it gets difficult for a Product Manager to comprehend customer requirements, the easy way is to put emphasis on MVP. If we start validating every requirement of the product, it will unnecessarily delay product development. There is always a necessity to strike a balance which I find missing. Further, in the case of enterprise products that have a direct effect on customer’s business, the customers will not be ready to validate it thoroughly. While building complex products for enterprise customers, it is always crucial to do extensive customer research and draft those details in PRD to get the bigger picture of the overall product. User stories do not communicate bigger picture. I can definitely bet if I ask for a PRD, some might even perceive me as a person from the dinosaur age. I am not showing my indignation on everyone. Probably, just a handful of them. My request is to embrace methodologies and process, comprehend them thoroughly and adapt them to suit your new product development. Always find the right fit between the product and its corresponding methodology.

Leave a Reply