Measuring radical innovation project success: typical metrics don’t work

Jimmi Normann Kristiansen (Department of Business and Management, Aalborg University, Aalborg, Denmark)
Paavo Ritala (School of Business and Management, Lappeenranta University of Technology, Lappeenranta, Finland)

Journal of Business Strategy

ISSN: 0275-6668

Article publication date: 27 July 2018

Issue publication date: 8 August 2018

9923

Abstract

Purpose

Firms frequently struggle with measuring the performance of their radical innovation activities. Due to the uncertainty and ambiguity involved, key performance indicators (KPIs) used for incremental innovation projects are often not useful in this context. The purpose of this paper is to explore suitable KPIs particularly useful for radical innovation projects.

Design/methodology/approach

This study first reviews commonly used measures for innovation projects, which is then followed by case-study evidence from three industry-leading international firms. This study includes 13 in-depth interviews with innovation managers and directors in these firms, providing insights on how they measure the progress and performance of radical innovation projects.

Findings

KPIs used commonly in incremental innovation showed lackluster results in the case firms and were problematic for radical innovation context. A key finding was that radical innovation project performance should be evaluated based on the process rather than on the expected outcome. Concurrently, based on the literature review and the cases, three sets of KPIs with 13 specific KPIs useful for radical innovation projects are proposed.

Originality/value

The paper addresses a core challenge in using established KPIs in a radical innovation context. The paper gathers and synthesizes a range of measurement points suitable for radical innovation projects and provides specific suggestions for appropriate metrics that innovation managers can use.

Keywords

Citation

Kristiansen, J.N. and Ritala, P. (2018), "Measuring radical innovation project success: typical metrics don’t work", Journal of Business Strategy, Vol. 39 No. 4, pp. 34-41. https://doi.org/10.1108/JBS-09-2017-0137

Publisher

:

Emerald Publishing Limited

Copyright © 2018, Jimmi Normann Kristiansen and Paavo Ritala.

License

Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode.


An important part of large firms’ radical innovation competency is tied to an appropriate managerial mind-set and a system that facilitates experimentation. Accordingly, organizational setup, culture, processes, launch strategies and top management involvement have a major effect on the success of radical innovation. From a management viewpoint, radical innovation projects are characterized by higher uncertainty and absorption of new knowledge for the firm, as well as exploration of new markets, technologies and/or business models. Therefore, suitable managerial practices to support these projects vary substantially from those supporting incremental innovation projects.

Throughout the radical innovation project life cycle of Discovery (exploration), Incubation (experimentation) and Acceleration (development) (O‘Connor et al., 2008), suitable key performance indicators (KPIs) (metrics) must be applied. During the Acceleration phase, radical innovation projects are matured to a point, where they should be measured in line with established metrics for incremental innovation within firms. However, in the innovation front-end (Discovery and Incubation), both the process and expected output have a much lower degree of predictability than in incremental innovation projects. Because radical innovation projects have distinct features differing from incremental innovation projects, commonly used metrics such as time to market and net present value provide little use and may even be harmful for project progress.

Illustrating the above, O‘Connor et al. (2008) discuss IBM’s project for silicon-germanium alloy for integrated circuits (from the 1990s). Here, the radical innovation project led by Bernie Meyerson had to be bootlegged and protected from the rest of the R&D organization to survive. The initial business model was misaligned, and subsequent sales growth and market expectations showed lower than expected fiscal returns. This nearly killed the project. A modification of evaluation metrics had to be made to represent the nature of the project. Silicon-germanium, a highly efficient semiconductor alloy, is still among the best-performing technology platforms for computer chips today. It currently competes with, and possibly outperforms, Intel and their pure silicon-based chips in the computer-chip efficiency race (Armasu, 2015).

Radical innovation projects are uncertain, long-term investments that often target new business areas for the distant future (5+ years). Therefore, metrics should be adjusted to meet demands and criteria of success suitable for this type of project, as commonly used metrics fit poorly if firms are to move beyond an incremental innovation strategy (Christensen et al., 2008).

Previous research has emphasized the importance of using suitable metrics for incremental and radical innovation (Henttonen et al., 2016; Joh and Mayfield, 2009; Paulson et al., 2007; Griffin and Page, 1996). There is, however, a need for a clearer understanding of the damaging effects incurred by using “traditional” product development metrics in the early life cycle of radical innovation projects. Moreover, a further discussion of suitable metrics for the early stages of radical innovation projects’ life cycle is called for. In this paper, frequently used metrics in innovation management are examined and discussed in terms of their usability with respect to the contrast between incremental and radical innovation projects. This is followed by a presentation of challenges with innovation project measurement in three industry-leading global firms. This leads to a discussion of suitable metrics for radical innovation activities.

1. Frequently used metrics for innovation projects

The term “key performance indicator” has been widely used in the management literature and refers to identifying key activities of a value-creation process in the firm and generating a way of measuring those activities (Kaplan and Norton, 1996). For the innovation management literature, key performance indicators have been researched mostly in relation with non-radical innovation projects. The popular measures used to gauge success are either firm- or product-related, which often target markets (e.g. market size and time to market) and finances (net present value or similar) and whether the pre-established plan is followed (Blindenbach-Driessen et al., 2010; Henttonen et al., 2016).

These measures very rarely target specifically radical innovation projects. One implication is that evaluations will be largely focused on fiscal output and expected market performance, rather than the process itself. Frequently used metrics for innovation projects, their applicability and their implications for radical innovation projects are given in Table I.

The metrics mentioned in Table I, covering both pre- and post-launch, either will constitute a poor fit if used or simply will not be applicable because of the different time horizons in project life cycle periods in incremental innovation projects compared to radical innovation projects.

For radical innovation projects, the technological route or market feasibility assessment may deviate from the preexisting assumptions, and new opportunities may occur during the course of five or more years of development. A key challenge in measuring progress in radical innovation projects is, therefore, firms’ inability to follow pre-set goals and measure project performance according to these goals during the project period. Challenges may also arise post-market launch, as new market learning may be needed for successful adoption and impact (Feiereisen et al., 2013).

To further investigate the challenge of finding appropriate metrics for radical innovation, the authors conducted case studies in three large, international firms that are global leaders in their respective fields. All three firms employ more than 5,000 people and have an annual turnover exceeding US$2bn. The firms spend between 6 and 13 per cent annually on R&D, and all of them have a proven track record in incremental innovation (two of the firms spend three times the industry average on R&D, and the third firm is on par with the industry). Pseudonyms of GreenCO, HeavyCO and MasterCO are used. A total of 13 in-depth semi-structured interviews with managers and directors of innovation were conducted at the firms. All interviews were recorded, transcribed and coded. This was further supplemented with documentation from project tools, strategy workshops, seminar work and publicly available information.

2. Understanding the measurement challenge: case evidence from three large firms

As radical innovation projects have different characteristics from incremental innovation projects, and frequently used metrics, such as net present value, introduced challenges once applied in a radical innovation context. According to their innovation director, HeavyCO was experiencing signs of low performance when using their established metrics:

It is flagged as a low net present value project. Senior management can deliberately say, this is a strategic project. But at every point, they will be told, this has no money, this has no money.

Even though firms are following the suggested management practices of radical innovation, metrics stemming from incremental innovation activities provided substantial challenges for HeavyCO. Another example was given from MasterCO:

What really will resonate in 90 per cent of the rest of the organization is: “How much have you sold? What is the bottom line?” “Yes, but we have a strong portfolio…”, or, “Sure, but how much have you sold?” And that is the name of the game.

Applying commonly used metrics for radical innovation projects has created challenges in the case firms especially because the initial focus of radical innovation project management is to conceptualize and experiment and, through this, gradually reduce the uncertainty affiliated with the project. Both the expected financial outcomes and projected time-to-market provide challenges for radical innovation project activities when compared to incremental innovation activities. The time-horizon also created issues in GreenCO, as the cycles of how often projects were measured using the commonly used evaluation criteria were too short:

If I have to choose a radical or an incremental project, it depends on what you gain and what you lose. But the difficulty is, we do these annual evaluations based on results. And if you do something really well, it may take a longer time to see that.

To resolve measurement challenges, the MasterCO respondent argued that they would prefer to use forward-looking rather than backward-looking metrics. MasterCO consolidated the information on what the organization had learned throughout their work with radical innovation. This enabled project managers to improve legitimacy toward sponsors in senior management. A concluding remark from a manager in MasterCO was related to the need for metrics for radical innovation projects:

In the beginning, we said, “Well, we shouldn’t have key performance indicators; we shouldn’t be measured. We cannot create a budget because there is too much uncertainty”. And I think that is nonsense. I think that it is an entirely different perspective than if you look into a factory making 10 million units per year with some well-defined variants. In reality, we need just as much structure to be able to attract resources. We have to find other ways of getting structure out of chaos to be able to communicate with our surroundings.

The illustrative examples above are a part of a larger case study with these three firms. As it is arduous to use commonly used metrics for radical innovation projects, there is a crucial exercise in developing replacement metrics for these projects.

3. Metrics for radical innovation projects

Based on a review of the literature and case study insights, we suggest three applicable sets of KPIs for assessing radical innovation project performance.

3.1 Market orientation

A commonly discussed feature of radical innovation is technological novelty. Certainly, the technological progression radical innovations can introduce may have a breakthrough nature. However, many technologically novel products fail the actual market test, and firms should therefore pay attention to how they are oriented toward the market for their radical innovation activities (Table II).

3.2 Learning and future opportunities

Radical innovation projects are often a part of a larger platform investment in firms. Here, firms have the possibility to cross-fertilize learning and investments across opportunities and business segments. Firms should therefore encourage opportunities that ensure positive conditions for new opportunities to emerge and even to foster growth in existing business segments.

Another aspect will be to go/terminate decisions. In case of project termination, firms should make high-quality termination reports to de-brief key learning from the process.

Finally, firms should also be able to assess how they use and grow core competencies, including the increase in the relevant knowledge base of the firm. Examples of firms that have been actively working with innovation as a holistic innovation system to boost the knowledge base are 3M[3], General Electric and Coca-Cola (Alsever, 2015) (Table III).

3.3 Resource dedication

Radical innovation activities may need to compete for resources from the firm, and it is therefore crucial to have a project overview with a description of resources required to support projects. HeavyCO had been focusing on keeping the proportion of financial resources allocated to their radical innovation projects low. However, these projects involved substantial intangible resources in the form of highly skilled, cross-functional teams. In addition, these employees had contact with the existing resource base of the firm, enabling them to efficiently and quickly gain access to key stakeholders of the firm.

When radical innovation projects mature in the pipeline, they will eventually require an increased allocation of financial resources to get to market. Firms should ex ante decide whether the resources will be available when they are needed. For areas of higher uncertainty, resources can be shared among firms. This includes sharing of the knowledge base but could also represent a pool of test equipment. For example, GreenCO had been sharing resources with external partners on test equipment for a project for a substantial period. Access to these external resources enabled GreenCO to vastly accelerate planned pilot tests, not to mention saving the firm millions of dollars, as they did not have to build a pilot plant (Table IV).

4. Conclusion: measuring value creation rather than value created

Established companies across industries have developed comprehensive toolsets for managing innovation projects and portfolios. Most of these tools and approaches are well suited for incremental innovation, but not for radical innovation. Highly uncertain radical innovation projects demand toolsets that are unlike those that perform well in the realm of incremental innovation.

Building on the radical innovation literature and the case studies of three global firms, three sets of metrics for radical innovation are proposed: market orientation, learning and future opportunities and resource dedication.

KPIs for radical innovation projects deviate from the traditional R&D project measures and should be adapted to fit an uncertain environment. The proposed metrics are targeted for the front-end, i.e. before the commercialization phase. In the commercialization phase, commonly used metrics have better usability. Therefore, innovation managers should pay close attention to the overall portfolio and adopt an appropriate set of measures for different projects depending on their maturity and type.

The findings are based on the literature and a case study on three firms that have been actively working with radical innovation for several years. It is not expected that the metrics presented here are fully exhaustive or provide an immediate “silver bullet” for radical innovation project success. The benefits of adoption of any type of measure depend on who is using the measures. Different kinds of managerial biases (e.g. group thinking, pet projects and confirmation bias) can hamper the potentially useful information available. Nevertheless, the study does discuss immediate and pertaining issues with using established metrics for radical innovation. The study provides useful metrics that can be part of a more holistic and effective assessment of radical innovation projects.

Commonly used metrics for innovation projects

Key performance indicator Key applicability Challenges for radical innovation evaluation
Net present value Assesses (pre-launch) the difference between future cash inflows and outflows and discounts it to the value represented today Uncertain and fluctuating net present value as future revenue is arduous to predict with accuracy (especially in early stages)
Return on investment Gives (post-launch) feedback on the net income from launched projects. Compares gains versus costs of investments Investments in radical innovation are broader than “single projects”. Return on investment does not provide a valuation of new competency and spillover effects built through radical innovation projects
Percentage of profits from products less than n years old Provides information on how new projects contribute to the firm’s turnover and the firm’s competitive position Analyses will often show that most profit comes collectively from incremental innovation projects, except for periods with radical innovation market breakthroughs
Total patents filed/pending/awarded Explains how firms are able to secure patent rights, giving an idea of future licensing potential, etc. Time required to patent is often longer for radical innovation projects. It may also be an unfamiliar patent landscape. A lot of experimentation is involved before product/technology descriptions are made
Time-to-market Describes the speed from innovation project investment to the first customer Expected conceptualization and experimentation for 2+ years before commercialization path is laid. Often a 5+ year time horizon to market
Success/failure rate of projects Measures the degree to which new projects in the portfolio succeed/fail. Indicates our ability to select “the right” projects for the pipeline “Failure” rates will be higher for radical innovation projects. These projects target multiple applications, and an initial project “failure” may not portray overall success and new competency development

Market orientation metrics for radical innovation projects

Key performance indicator Measurement Description
Potential of market 1-5 Likert scale, text description The potential of the market should be described in terms of attractiveness, growth and, if possible, the size. Latent markets are more difficult to assess, as these are mostly be based on “trends” rather than existing market needs
Existing links to markets Text: contacts, former market penetration Does the firm have contacts that help in gaining access to markets? Has the firm previously navigated in markets similar to the one pursued? Are there complementary players in the firm’s ecosystem who can be helpful? Gives an understanding of the market uncertainty related to the project, and whether this uncertainty can be embraced through networking
Finding new market needs Text and 1-5 Likert scale : degree of novelty, existing needs or future needs Are needs already saturated by other products or services? Are we targeting existing or latent needs? The firm has to understand the implications of either entering a market with a substituting solution, or whether new market needs can be created and tapped. Can also include a text scenario description of the use of new business models
Deliverance of real benefits to the customer 1-5 Likert scale, text description Is it something that can be clearly attributed as delivering a significant value to the customer? The uncertainty related to radical innovation projects is significant; the potential benefits that the customer should receive should be at least equally significant 

Source: Based on literature review and the case studies in this paper

Learning and future opportunity metrics for radical innovation projects

Key performance indicator Measurement Description
Leads to other opportunities 1-5 Likert scale, text description Projects that can be used in many different settings are better than projects that can be launched only for single products. Allows for other opportunities to branch out, which mitigates the risk of dead ends 
Learning per dollar spent # of opp. explored vis-à-vis $ Managers of radical innovation projects should consider how feasibility and experimentation studies are carried out and what resources the studies require. Enables firms to understand the best way in which resources are utilized in terms of learning
Potential for growth in existing business segments 1-5 Likert scale, text description Does the opportunity give an opening for growth or synergy in an existing business segment? Studies have previously indicated that “product and service bundling” with existing solutions increases success rates for radical innovation projects 
Termination report quality Text and # of parameters filed, quality of information This reflects on discontinued projects. Provides a precise description of why the project was shut down. Also explains the parameters that were fulfilled as part of the project, whom to contact, and under what conditions the project could be reactivated
Ability to use and grow core competencies 1-5 Likert scale, text Is the project related to the core business activities? Is the firm able to benefit from core competencies and resources? Where are the synergies located? Enables increased legitimacy for projects and may increase the likelihood of embedding the radical innovation project in the core of the firm in the future

Source: Based on literature review and the case studies in this paper

Resource dedication metrics for radical innovation projects

Key performance indicator Measurement Description
Active projects supported Table overview, text Gives an overview of projects supported as a part of the current pipeline ($ spent, $ on budget) with a headcount indication per project (i.e. conversion of full-time employee per project). Gives management an indication of how resources are distributed across the pipeline
Intangible resources 1-5 Likert scale, market/tech., degree to which internal resources can fulfill task Does the firm have an “in-house” market and technology know-how to fulfill the task? Does the firm’s knowledge base have the right competencies and absorptive capacity to fulfill the task?
Tangible resources Text: financial, equipment, etc. Does the firm possess, and is it able and willing to allocate, adequate financial resources and equipment for the project? (Before any serious commitment is made to the projects, the firm has to decide whether it is willing to dedicate substantial investment for the new project in terms of tangible resources)
External resources # of partners, text: expertise of partners Do we have an accessible ecosystem of upstream and downstream complementors and partners, as well as the institutional support to pursue the project? Under which premises can we dedicate external partners for the project, and what are our requirements for the partners? External partners are often crucial for radical innovation project success. Clear orientations of dedication and tolerance for the project should be stated early on

Source: Based on literature review and the case studies in this paper

References

Alsever, J. (2015), “Startups … inside giant companies”, available at: http://fortune.com/2015/04/26/startups-inside-giant-companies/ (accessed 30 August 2017).

Armasu, L. (2015), “IBM beats Intel to 7nm process thanks to silicon-germanium transistors, EUV lithography”, available at: www.tomshardware.co.uk/ibm-7nm-silicon-germanium-transistors,news-50713.html (accessed 30 August, 2017).

Blindenbach-Driessen, F., van Dalen, J. and van den Ende, J. (2010), “Subjective performance assessment of innovation projects”, Journal of Product Innovation Management, Vol. 27 No. 4, pp. 572-592.

Bremser, W.G. and Barsky, N.P. (2004), “Utilizing the balanced scorecard for R&D performance measurement”, R&D Management, Vol. 34 No. 3, pp. 229-238.

Christensen, C.M., Kaufman, S.P. and Shih, W.C. (2008), “Innovation killers: how financial tools destroy your capacity to do new things”, Harvard Business Review, Vol. 86 No. 1, pp. 98-105.

Feiereisen, S., Wong, V. and Broderick, A.J. (2013), “Is a picture always worth a thousand words? The impact of presentation formats in consumers‘early evaluations of really new products (RNPs)”, Journal of Product Innovation Management, Vol. 30 No. 1, pp. 159-173.

Griffin, A. and Page, A.L. (1993), “An interim report on measuring product development success and failure”, Journal of Product Innovation Management, Vol. 10 No. 4, pp. 291-308.

Griffin, A. and Page, A.L. (1996), “PDMA success measurement project: recommended measures for product development success and failure”, Journal of Product Innovation Management, Vol. 13 No. 6, pp. 478-496.

Henttonen, K., Ojanen, V. and Puumalainen, K. (2016), “Searching for appropriate performance measures for innovation and development projects”, R&D Management, Vol. 46 No. 5, pp. 914-927.

Joh, J.M. and Mayfield, M. (2009), “The discipline of product discovery: identifying breakthrough business opportunities”, Journal of Business Strategy, Vol. 30 Nos 2/3, pp. 70-77.

Kaplan, R. and Norton, D. (1996), The Balanced Scorecard, Harvard Business School Press, Boston, MA.

Kirsner, S. (2015), “What Big companies get wrong about innovation metrics”, available at: https://hbr.org/2015/05/what-big-companies-get-wrong-about-innovation-metrics (accessed 30, August 2017).

O‘Connor, G.C., Leifer, R., Paulson, A.S. and Peters, L.S. (2008), Grabbing Lightning: Building A Capability for Breakthrough Innovation, Wiley & Sons, San Francisco, CA.

Paulson, A.S., O’Connor, G.C. and Robeson, D. (2007), “Evaluating radical innovation portfolios”, Research Technology Management, Vol. 50 No. 5, pp. 17-29.

Further reading

Cooper, R.G. (2013), “Where are all the breakthrough new products? Using portfolio management to boost innovation”, Research Technology Management, Vol. 56 No. 5, pp. 25-33.

Winterhalter, S., Weiblen, T., Wecht, C.H. and Gasmann, O. (2017), “Business model innovation processes in large corporations: insights from BASF”, Journal of Business Strategy, Vol. 38 No. 2, pp. 62-75.

Corresponding author

Jimmi Normann Kristiansen can be contacted at: jimmi@business.aau.dk

About the authors

Jimmi Normann Kristiansen is Assistant Professor at Department of Business and Management, Aalborg University, Aalborg, Denmark.

Paavo Ritala is Professor at the School of Business and Management, Lappeenranta University of Technology, Lappeenranta, Finland.

Related articles